Open obriensystems opened 12 months ago
Procedure:
Create projects - GCP
export PROJECT_ID=vpn-aws-obs
gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default
gcloud config set project "${PROJECT_ID}"
export BILLING_ID=01D...0154
gcloud beta billing projects link ${PROJECT_ID} --billing-account ${BILLING_ID}
Set iAM permissions
export USER_EMAIL=michael@obr...tware
export ORG_ID=$(gcloud projects get-ancestors $PROJECT_ID --format='get(id)' | tail -1)
gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$USER_EMAIL --role=roles/compute.securityAdmin --quiet
gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$USER_EMAIL --role=roles/compute.networkeAdmin --quiet
gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$USER_EMAIL --role=roles/compute.admin --quiet
bug: routing mode should be regional bug: enable compute API on the project first before creating a VPC https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-the-ha-vpn-gateway-and-cloud-router-on-google-cloud
gcloud services enable compute.googleapis.com
create network
export NETWORK=vpn-aws-obs-vpc
export SUBNET_MODE=custom
export BGP_ROUTING_MODE=regional
gcloud compute networks create $NETWORK --subnet-mode $SUBNET_MODE --bgp-routing-mode $BGP_ROUTING_MODE
export SUBNET_NAME=$NETWORK-sn
export IP_ADDRESS_RANGE=10.101.0.0/24
export REGION=northamerica-northeast1
gcloud compute networks subnets create $SUBNET_NAME --network $NETWORK --region $REGION --range $IP_ADDRESS_RANGE
create HA VPN Gateway
export HA_VPN_GATEWAY_NAME=$PROJECT_ID-vpn
gcloud compute vpn-gateways create $HA_VPN_GATEWAY_NAME --network $NETWORK --region $REGION
see
INTERFACE0: 34.124.12.157
INTERFACE1: 34.104.78.53
export ROUTER_NAME=$PROJECT_ID-router
export GOOGLE_ASN=65534
gcloud compute routers create $ROUTER_NAME --region $REGION --network $NETWORK --asn $GOOGLE_ASN --advertisement-mode custom --set-advertisement-groups all_subnets
Create 2 customer gateways
export INTERFACE_0_IP_ADDRESS=34.124.12.157
export INTERFACE_1_IP_ADDRESS=34.104.78.53
export GOOGLE_ASN=65534
aws ec2 create-customer-gateway --type ipsec.1 --public-ip $INTERFACE_0_IP_ADDRESS --bgp-asn $GOOGLE_ASN
{
"CustomerGateway": {
"BgpAsn": "65534",
"CustomerGatewayId": "cgw-0219d0daa1e0f6f02",
"IpAddress": "34.124.12.157",
"State": "available",
"Type": "ipsec.1",
}
}
export CUSTOMER_GATEWAY_1=cgw-0219d0daa1e0f6f02
aws ec2 create-customer-gateway --type ipsec.1 --public-ip $INTERFACE_1_IP_ADDRESS --bgp-asn $GOOGLE_ASN
{
"CustomerGateway": {
"BgpAsn": "65534",
"CustomerGatewayId": "cgw-066a98dc85373f724",
"IpAddress": "34.104.78.53",
"State": "available",
"Type": "ipsec.1"
}
}
export CUSTOMER_GATEWAY_2=cgw-066a98dc85373f724
Create VPC
Bug: https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create_gateways_and_vpn_connections_on_aws missing VPC creation on AWS including private subnet , ingress firewall rules https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli
Create AWS VPC https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli
export AWS_IP_ADDRESS_RANGE=10.51.0.0/24
export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/26
aws ec2 create-vpc --cidr-block $AWS_IP_ADDRESS_RANGE --query Vpc.VpcId --output text
export VPC_ID=vpc-0c95bd8820efebea4
BUG: VPC already has a route table with a default route after vpc creation
BUG: switch subnet to us-east-1a (to match VPC) - or VPC missing region
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id vpc-1a2b3c4d5e6f1a2b3 --cidr-block 10.0.1.0/20 --availability-zone us-east-2a --query Subnet.SubnetId --output text
An error occurred (InvalidParameterValue) when calling the CreateSubnet operation: Value (us-east-2a) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1e, us-east-1f.
BUG: VPC must be /16 not /24 - in order to use subnets of /20
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id vpc-1a2b3c4d5e6f1a2b3 --cidr-block 10.0.1.0/20 --availability-zone us-east-1a --query Subnet.SubnetId --output text
An error occurred (InvalidVpcID.NotFound) when calling the CreateSubnet operation: The vpc ID 'vpc-1a2b3c4d5e6f1a2b3' does not exist
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
An error occurred (InvalidSubnet.Range) when calling the CreateSubnet operation: The CIDR '10.51.1.0/20' is invalid.
[cloudshell-user@ip-10-6-18-196 ~]$ export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/26
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
An error occurred (InvalidSubnet.Range) when calling the CreateSubnet operation: The CIDR '10.51.1.0/26' is invalid.
create subnet
aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
export SUBNET_ID=
Recreate a new VPC
export AWS_IP_ADDRESS_RANGE=10.51.0.0/16
export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/24
aws ec2 create-vpc --cidr-block $AWS_IP_ADDRESS_RANGE --query Vpc.VpcId --output text
export VPC_ID=vpc-0f504a74c4317f091
Add subnet
aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
export SUBNET_ID=subnet-0cdb542888b890f9f
No AWS IGW or NGW - but put a NGW for the private subnet - step 6
# allocate EIP
aws ec2 allocate-address --domain vpc --query AllocationId --output text
export ALLOCATION_ID=eipalloc-05613c22cf3f36803
# create a NAT GW
aws ec2 create-nat-gateway --subnet-id $SUBNET_ID --allocation-id $ALLOCATION_ID
{
"ClientToken": "6d1b56aa-2d25-4cae-bdd4-a94a593d37a6",
"NatGateway": {
"CreateTime": "2023-08-27T03:35:47+00:00",
"NatGatewayAddresses": [
{
"AllocationId": "eipalloc-05613c22cf3f36803",
"IsPrimary": true,
"Status": "associating"
}
],
"NatGatewayId": "nat-0c4da22f458fe7cbb",
"State": "pending",
"SubnetId": "subnet-0cdb542888b890f9f",
"VpcId": "vpc-0f504a74c4317f091",
"ConnectivityType": "public"
}
}
export NAT_ID=nat-0c4da22f458fe7cbb
# create route table
aws ec2 create-route-table --vpc-id $VPC_ID --query RouteTable.RouteTableId --output text
export ROUTETABLE_ID=rtb-0ff23a1fb426e1301
# create route to Nat GW
aws ec2 create-route --route-table-id $ROUTETABLE_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $NAT_ID
# associate route table with private subnet
aws ec2 associate-route-table --route-table-id $ROUTETABLE_ID --subnet-id $SUBNET_ID
{
"AssociationId": "rtbassoc-0a155c6354b7008c4",
"AssociationState": {
"State": "associated"
}
}
Create VM
# create VM
# test external internet connectivity
finished with https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli
Attach VPG to VPC return to step 2 of https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create_gateways_and_vpn_connections_on_aws
export VPC_ID=vpc-0f504a74c4317f091
export AWS_SIDE_ASN=65501
aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn $AWS_SIDE_ASN
{
"VpnGateway": {
"State": "available",
"Type": "ipsec.1",
"VpcAttachments": [],
"VpnGatewayId": "vgw-0bd4928d84cd855cf",
"AmazonSideAsn": 65501
}
}
export VPN_GATEWAY_ID=vgw-0bd4928d84cd855cf
aws ec2 attach-vpn-gateway --vpn-gateway-id $VPN_GATEWAY_ID --vpc-id $VPC_ID
{
"VpcAttachment": {
"State": "attaching",
"VpcId": "vpc-0f504a74c4317f091"
}
}
follow for options https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html#vpn-create-vpn-connection https://docs.aws.amazon.com/cli/latest/reference/ec2/create-vpn-connection.html https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNTunnels.html
generate preshared keys https://cloud.google.com/network-connectivity/docs/vpn/how-to/generating-pre-shared-key or use
openssl rand -base64 24
export CUSTOMER_GATEWAY_1=cgw-0219d0daa1e0f6f02
export CUSTOMER_GATEWAY_2=cgw-066a98dc85373f724
export AWS_T1_IP=169.254.51.0/30
export AWS_T2_IP=169.254.52.0/30
export SHARED_SECRET_1=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G
export SHARED_SECRET_2=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0
#2030827: edit - was duplicated 52 and 53 but didn't use these anyway below - so no T2/T3 overalp
export AWS_T3_IP=169.254.53.0/30
export AWS_T4_IP=169.254.54.0/30
export SHARED_SECRET_3=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI
export SHARED_SECRET_4=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=$AWS_T1_IP,PreSharedKey=$SHARED_SECRET_1},{TunnelInsideCidr=$AWS_T2_IP,PreSharedKey=$SHARED_SECRET_2}]'
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=$AWS_T3_IP,PreSharedKey=$SHARED_SECRET_3},{TunnelInsideCidr=$AWS_T4_IP,PreSharedKey=$SHARED_SECRET_4}]'
BUG:
BUG: shared key site generates / and + (invalid chars) - convert to . https://cloud.google.com/network-connectivity/docs/vpn/how-to/generating-pre-shared-key
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk+MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0+chcFdgh1UN0}]'
An error occurred (InvalidParameterValue) when calling the CreateVpnConnection operation: Value for parameter PreSharedKey is invalid.
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0}]'
{
"VpnConnection": {
"CustomerGatewayConfiguration": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<vpn_connection id=\"vpn-011bfe843f9f75e55\">\n <customer_gateway_id>cgw-0219d0daa1e0f6f02</customer_gateway_id>\n <vpn_gateway_id>vgw-0bd4928d84cd855cf</vpn_gateway_id>\n <vpn_connection_type>ipsec.1</vpn_connection_type>\n <ipsec_tunnel>\n <customer_gateway>\n <tunnel_outside_address>\n <ip_address>34.124.12.157</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.51.2</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65534</asn>\n <hold_time>30</hold_time>\n </bgp>\n </customer_gateway>\n <vpn_gateway>\n <tunnel_outside_address>\n <ip_address>34.232.42.137</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.51.1</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65501</asn>\n <hold_time>30</hold_time>\n </bgp>\n </vpn_gateway>\n <ike>\n <authentication_protocol>sha1</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>28800</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>main</mode>\n <pre_shared_key>4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G</pre_shared_key>\n </ike>\n <ipsec>\n <protocol>esp</protocol>\n <authentication_protocol>hmac-sha1-96</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>3600</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>tunnel</mode>\n <clear_df_bit>true</clear_df_bit>\n <fragmentation_before_encryption>true</fragmentation_before_encryption>\n <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n <dead_peer_detection>\n <interval>10</interval>\n <retries>3</retries>\n </dead_peer_detection>\n </ipsec>\n </ipsec_tunnel>\n <ipsec_tunnel>\n <customer_gateway>\n <tunnel_outside_address>\n <ip_address>34.124.12.157</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.52.2</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65534</asn>\n <hold_time>30</hold_time>\n </bgp>\n </customer_gateway>\n <vpn_gateway>\n <tunnel_outside_address>\n <ip_address>54.159.162.176</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.52.1</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65501</asn>\n <hold_time>30</hold_time>\n </bgp>\n </vpn_gateway>\n <ike>\n <authentication_protocol>sha1</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>28800</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>main</mode>\n <pre_shared_key>SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0</pre_shared_key>\n </ike>\n <ipsec>\n <protocol>esp</protocol>\n <authentication_protocol>hmac-sha1-96</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>3600</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>tunnel</mode>\n <clear_df_bit>true</clear_df_bit>\n <fragmentation_before_encryption>true</fragmentation_before_encryption>\n <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n <dead_peer_detection>\n <interval>10</interval>\n <retries>3</retries>\n </dead_peer_detection>\n </ipsec>\n </ipsec_tunnel>\n</vpn_connection>",
"CustomerGatewayId": "cgw-0219d0daa1e0f6f02",
"Category": "VPN",
"State": "pending",
"VpnConnectionId": "vpn-011bfe843f9f75e55",
"VpnGatewayId": "vgw-0bd4928d84cd855cf",
"GatewayAssociationState": "associated",
"Options": {
"EnableAcceleration": false,
"StaticRoutesOnly": false,
"LocalIpv4NetworkCidr": "0.0.0.0/0",
"RemoteIpv4NetworkCidr": "0.0.0.0/0",
"OutsideIpAddressType": "PublicIpv4",
"TunnelInsideIpVersion": "ipv4",
"TunnelOptions": [
{
"OutsideIpAddress": "34.232.42.137",
"TunnelInsideCidr": "169.254.51.0/30",
"PreSharedKey": "4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G",
"LogOptions": {
"CloudWatchLogOptions": {
"LogEnabled": false
}
}
},
{
"OutsideIpAddress": "54.159.162.176",
"TunnelInsideCidr": "169.254.52.0/30",
"PreSharedKey": "SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0",
"LogOptions": {
"CloudWatchLogOptions": {
"LogEnabled": false
}
}
}
]
},
"Routes": [],
"Tags": []
}
}
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0}]'
"VpnConnectionId": "vpn-011bfe843f9f75e55",
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.53.0/30,PreSharedKey=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI},{TunnelInsideCidr=169.254.54.0/30,PreSharedKey=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf}]'
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.53.0/30,PreSharedKey=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI},{TunnelInsideCidr=169.254.54.0/30,PreSharedKey=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf}]'
{
"VpnConnection": {
"CustomerGatewayConfiguration": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<vpn_connection id=\"vpn-09e6693b14fa31452\">\n <customer_gateway_id>cgw-066a98dc85373f724</customer_gateway_id>\n <vpn_gateway_id>vgw-0bd4928d84cd855cf</vpn_gateway_id>\n <vpn_connection_type>ipsec.1</vpn_connection_type>\n <ipsec_tunnel>\n <customer_gateway>\n <tunnel_outside_address>\n <ip_address>34.104.78.53</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.53.2</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65534</asn>\n <hold_time>30</hold_time>\n </bgp>\n </customer_gateway>\n <vpn_gateway>\n <tunnel_outside_address>\n <ip_address>44.215.227.192</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.53.1</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65501</asn>\n <hold_time>30</hold_time>\n </bgp>\n </vpn_gateway>\n <ike>\n <authentication_protocol>sha1</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>28800</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>main</mode>\n <pre_shared_key>ihwsnhelOYX8GAYQvxyszRDnKUigL7wI</pre_shared_key>\n </ike>\n <ipsec>\n <protocol>esp</protocol>\n <authentication_protocol>hmac-sha1-96</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>3600</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>tunnel</mode>\n <clear_df_bit>true</clear_df_bit>\n <fragmentation_before_encryption>true</fragmentation_before_encryption>\n <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n <dead_peer_detection>\n <interval>10</interval>\n <retries>3</retries>\n </dead_peer_detection>\n </ipsec>\n </ipsec_tunnel>\n <ipsec_tunnel>\n <customer_gateway>\n <tunnel_outside_address>\n <ip_address>34.104.78.53</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.54.2</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65534</asn>\n <hold_time>30</hold_time>\n </bgp>\n </customer_gateway>\n <vpn_gateway>\n <tunnel_outside_address>\n <ip_address>52.2.98.253</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.54.1</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65501</asn>\n <hold_time>30</hold_time>\n </bgp>\n </vpn_gateway>\n <ike>\n <authentication_protocol>sha1</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>28800</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>main</mode>\n <pre_shared_key>7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf</pre_shared_key>\n </ike>\n <ipsec>\n <protocol>esp</protocol>\n <authentication_protocol>hmac-sha1-96</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>3600</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>tunnel</mode>\n <clear_df_bit>true</clear_df_bit>\n <fragmentation_before_encryption>true</fragmentation_before_encryption>\n <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n <dead_peer_detection>\n <interval>10</interval>\n <retries>3</retries>\n </dead_peer_detection>\n </ipsec>\n </ipsec_tunnel>\n</vpn_connection>",
"CustomerGatewayId": "cgw-066a98dc85373f724",
"Category": "VPN",
"State": "pending",
"VpnConnectionId": "vpn-09e6693b14fa31452",
"VpnGatewayId": "vgw-0bd4928d84cd855cf",
"GatewayAssociationState": "associated",
"Options": {
"EnableAcceleration": false,
"StaticRoutesOnly": false,
"LocalIpv4NetworkCidr": "0.0.0.0/0",
"RemoteIpv4NetworkCidr": "0.0.0.0/0",
"OutsideIpAddressType": "PublicIpv4",
"TunnelInsideIpVersion": "ipv4",
"TunnelOptions": [
{
"OutsideIpAddress": "44.215.227.192",
"TunnelInsideCidr": "169.254.53.0/30",
"PreSharedKey": "ihwsnhelOYX8GAYQvxyszRDnKUigL7wI",
"LogOptions": {
"CloudWatchLogOptions": {
"LogEnabled": false
}
}
},
{
"OutsideIpAddress": "52.2.98.253",
"TunnelInsideCidr": "169.254.54.0/30",
"PreSharedKey": "7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf",
"LogOptions": {
"CloudWatchLogOptions": {
"LogEnabled": false
}
}
}
]
},
"Routes": [],
"Tags": []
}
}
VPNs take a couple min to transition from pending
step GCP 2 - VPN Tunnels https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-vpn-tunnels-and-cloud-router-interfaces-on-google-cloud
export AWS_GW_IP_1=34.232.42.137
export AWS_GW_IP_2=54.159.162.176
export AWS_GW_IP_3=44.215.227.192
export AWS_GW_IP_4=52.2.98.253
export PEER_GATEWAY_NAME=aws-peer-vpn-gw
gcloud compute external-vpn-gateways create $PEER_GATEWAY_NAME --interfaces 0=$AWS_GW_IP_1,1=$AWS_GW_IP_2,2=$AWS_GW_IP_3,3=$AWS_GW_IP_4
Creating external VPN gateway...done.
NAME: aws-peer-vpn-gw
REDUNDANCY_TYPE: FOUR_IPS_REDUNDANCY
4 tunnels
use ike-version=2 https://cloud.google.com/sdk/gcloud/reference/compute/vpn-tunnels/create
export REGION=northamerica-northeast1
export SHARED_SECRET_1=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G
export SHARED_SECRET_2=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0
export SHARED_SECRET_3=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI
export SHARED_SECRET_4=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf
export PROJECT_ID=vpn-aws-obs
export ROUTER_NAME=$PROJECT_ID-router
export HA_VPN_GATEWAY_NAME=$PROJECT_ID-vpn
export IKE_VERSION=2
gcloud compute vpn-tunnels create tunnel-1 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 0 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_1 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 0
Creating VPN tunnel...done.
NAME: tunnel-1
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 0
PEER_ADDRESS: 34.232.42.137
20230826 3 more
gcp side
aws side - IP_SEC is up only so far until we setup BGP
20230827 Add 3 more tunnel https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-vpn-tunnels-and-cloud-router-interfaces-on-google-cloud
gcloud compute vpn-tunnels create tunnel-2 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 1 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_2 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 0
Creating VPN tunnel...done.
NAME: tunnel-2
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 0
PEER_ADDRESS: 54.159.162.176
gcloud compute vpn-tunnels create tunnel-3 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 2 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_3 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 1
Creating VPN tunnel...done.
NAME: tunnel-3
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 1
PEER_ADDRESS: 44.215.227.192
gcloud compute vpn-tunnels create tunnel-4 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 3 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_4 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 1
Creating VPN tunnel...done.
NAME: tunnel-4
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 1
PEER_ADDRESS: 52.2.98.253
GCP side
AWS side
takes about 4 min for all 4
4 router interfaces
Get IPs from AWS VPN config (generic IKE2) - the customer gateway address
Inside IP Addresses
#export AWS_T1_IP=169.254.51.0/30
#export AWS_T2_IP=169.254.52.0/30
#export AWS_T3_IP=169.254.53.0/30
#export AWS_T4_IP=169.254.54.0/30
export PROJECT_ID=vpn-aws-obs
export ROUTER_NAME=$PROJECT_ID-router
export REGION=northamerica-northeast1
# take off the /30 - convert to non-CIDR
export GOOGLE_BGP_IP_TUNNEL_1=169.254.51.2
export GOOGLE_BGP_IP_TUNNEL_2=169.254.52.2
export GOOGLE_BGP_IP_TUNNEL_3=169.254.53.2
export GOOGLE_BGP_IP_TUNNEL_4=169.254.54.2
# inside addresses on gcp side match inside from aws above
gcloud compute routers add-interface $ROUTER_NAME --interface-name int-1 --vpn-tunnel tunnel-1 --ip-address $GOOGLE_BGP_IP_TUNNEL_1 --mask-length 30 --region $REGION
Updated [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/routers/vpn-aws-obs-router].
gcloud compute routers add-interface $ROUTER_NAME --interface-name int-2 --vpn-tunnel tunnel-2 --ip-address $GOOGLE_BGP_IP_TUNNEL_2 --mask-length 30 --region $REGION
gcloud compute routers add-interface $ROUTER_NAME --interface-name int-3 --vpn-tunnel tunnel-3 --ip-address $GOOGLE_BGP_IP_TUNNEL_3 --mask-length 30 --region $REGION
gcloud compute routers add-interface $ROUTER_NAME --interface-name int-4 --vpn-tunnel tunnel-4 --ip-address $GOOGLE_BGP_IP_TUNNEL_4 --mask-length 30 --region $REGION
mistake found - t2 and t3 CIDRS overlap - but I used direct strings and not the VARs above - so we are good
#export AWS_T2_IP=169.254.52.0/30
#export AWS_T3_IP=169.254.52.0/30
4 add BGP peers
--peer-ip-address: invalid ipv4 value: '169.254.51.0/30' The link-local address of the peer router. Must be a link-local IPv4 address belonging to the range 169.254.0.0/16. not in https://rajathithanrajasekar.medium.com/create-ha-vpn-in-google-cloud-using-gcloud-commands-66d691a8d8be
bug: CIDR not recognized - need IP like 169.254.51.1 from the vpn config - the BGP neighbor IP
--peer-ip-address AWS_T3_IP
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $AWS_T1_IP --region $REGION
ERROR: (gcloud.compute.routers.add-bgp-peer) argument --peer-ip-address: invalid ipv4 value: '169.254.51.0/30'
Usage: gcloud compute routers add-bgp-peer NAME --interface=INTERFACE --peer-asn=PEER_ASN --peer-name=PEER_NAME [optional flags]
is it the VPG
Inside IP Addresses
- Customer Gateway : 169.254.54.2/30
- Virtual Private Gateway : 169.254.54.1/30
use as example
in another working VPN of my we use 169.254.0.2 - this is the GOOGLE_BGP_IP_TUNNEL_1 address
working with BGP neighbor IP
export AWS_SIDE_ASN=65501
export PEER_ASN=$AWS_SIDE_ASN
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $AWS_T1_IP --region $REGION
replace
AWS_T1_IP with GOOGLE_BGP_IP_TUNNEL_1
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $GOOGLE_BGP_IP_TUNNEL_1 --region $REGION
ERROR: (gcloud.compute.routers.add-bgp-peer) HTTPError 400: Invalid value for field 'resource.bgpPeers[0].peerIpAddress': '169.254.51.2'. In BGP Peer "aws-conn1-tunn1", peer IP address must be different from IP address 169.254.51.2.
use the VPG neighbor IP
Inside IP Addresses
- Customer Gateway : 169.254.54.2/30
- Virtual Private Gateway : 169.254.54.1/30
-
BGP Configuration Options:
- Customer Gateway ASN : 65534
- Virtual Private Gateway ASN : 65501
- Neighbor IP Address : 169.254.54.1
- Neighbor Hold Time : 30
working
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address 169.254.51.1 --region $REGION
Creating peer [aws-conn1-tunn1] in router [vpn-aws-obs-router]...done.
gcp side
aws side
Add 3 remaining BGP sessions
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn2 --peer-asn $PEER_ASN --interface int-2 --peer-ip-address 169.254.52.1 --region $REGION
NOTICE 2023-08-27T14:06:52.119687391Z BGP Event: Successfully added configuration for peers: 169.254.51.1, 169.254.52.1
INFO 2023-08-27T14:06:52.128174992Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
NOTICE 2023-08-27T14:06:53.069505725Z BGP Event: BGP peering with 169.254.52.1 came up 0 seconds ago, hold time is 60 seconds, graceful restart timer is 60 seconds
INFO 2023-08-27T14:06:53.203166433Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
NOTICE 2023-08-27T14:06:54.232914Z [protoPayload.serviceName: compute.googleapis.com] [protoPayload.methodName: v1.compute.routers.patch] [protoPayload.resourceName: projects/vpn-aws-obs/regions/northamerica-northeast1/routers/vpn-aws-obs-router] [protoPayload.authenticationInfo.principalEmail: michael@obrien.software] audit_log, method: "v1.compute.routers.patch", principal_email: "michael@obrien.software"
INFO 2023-08-27T14:07:50.515738103Z Prefix 10.51.0.0/16 NextHops 169.254.51.1,169.254.52.1 priority 200 received by cloud router
INFO 2023-08-27T14:07:54.116909103Z Prefix 10.51.0.0/16 NextHops 169.254.52.1 priority 100 received by cloud router
1007-1011 - 4 min for AWS side
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn2-tunn1 --peer-asn $PEER_ASN --interface int-3 --peer-ip-address 169.254.53.1 --region $REGION
NOTICE 2023-08-27T14:16:48.364249433Z BGP Event: BGP peering with 169.254.54.1 came up 0 seconds ago, hold time is 60 seconds, graceful restart timer is 60 seconds
INFO 2023-08-27T14:16:48.390937235Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
INFO 2023-08-27T14:16:48.506182843Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
last of 4 bgp tunnels
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn2-tunn2 --peer-asn $PEER_ASN --interface int-4 --peer-ip-address 169.254.54.1 --region $REGION
All 4 tunnels up on both sides
GCP
AWS
check routes
use a bastion https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/
Verify BGP dynamic routes on both GCP and AWS sides
from AWS 51 subnet (101 is gcp)
INFO 2023-08-27T14:17:48.626770110Z Prefix 10.51.0.0/16 NextHops 169.254.53.1,169.254.54.1 priority 200 received by cloud router
INFO 2023-08-27T14:17:49.427057114Z Prefix 10.51.0.0/16 NextHops 169.254.54.1 priority 100 received by cloud router
gcloud compute routers get-status $ROUTER_NAME --region $REGION --format='flattened(result.bgpPeerStatus[].name, result.bgpPeerStatus[].ipAddress, result.bgpPeerStatus[].peerIpAddress)'
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers get-status $ROUTER_NAME --region $REGION --format='flattened(result.bgpPeerStatus[].name, result.bgpPeerStatus[].ipAddress, result.bgpPeerStatus[].peerIpAddress)'
result.bgpPeerStatus[0].ipAddress: 169.254.51.2
result.bgpPeerStatus[0].name: aws-conn1-tunn1
result.bgpPeerStatus[0].peerIpAddress: 169.254.51.1
result.bgpPeerStatus[1].ipAddress: 169.254.52.2
result.bgpPeerStatus[1].name: aws-conn1-tunn2
result.bgpPeerStatus[1].peerIpAddress: 169.254.52.1
result.bgpPeerStatus[2].ipAddress: 169.254.53.2
result.bgpPeerStatus[2].name: aws-conn2-tunn1
result.bgpPeerStatus[2].peerIpAddress: 169.254.53.1
result.bgpPeerStatus[3].ipAddress: 169.254.54.2
result.bgpPeerStatus[3].name: aws-conn2-tunn2
result.bgpPeerStatus[3].peerIpAddress: 169.254.54.1
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute vpn-tunnels list
NAME: tunnel-1
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 34.232.42.137
NAME: tunnel-2
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 54.159.162.176
NAME: tunnel-3
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 44.215.227.192
NAME: tunnel-4
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 52.2.98.253
gcloud compute vpn-tunnels describe tunnel-1 --region $REGION --format='flattened(status,detailedStatus)'
detailed_status: Tunnel is up and running.
status: ESTABLISHED
list dynamic routes
gcloud compute routers get-status $ROUTER_NAME --region $REGION --format="flattened(result.bestRoutes)"
result.bestRoutes[0].asPaths[0].asLists[0]: 65501
result.bestRoutes[0].asPaths[0].pathSegmentType: AS_SEQUENCE
result.bestRoutes[0].creationTimestamp: 2023-08-27T07:07:54.116-07:00
result.bestRoutes[0].destRange: 10.51.0.0/16
result.bestRoutes[0].kind: compute#route
result.bestRoutes[0].network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
result.bestRoutes[0].nextHopIp: 169.254.52.1
result.bestRoutes[0].nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
result.bestRoutes[0].priority: 100
result.bestRoutes[0].routeType: BGP
result.bestRoutes[1].asPaths[0].asLists[0]: 65501
result.bestRoutes[1].asPaths[0].pathSegmentType: AS_SEQUENCE
result.bestRoutes[1].creationTimestamp: 2023-08-27T07:17:49.427-07:00
result.bestRoutes[1].destRange: 10.51.0.0/16
result.bestRoutes[1].kind: compute#route
result.bestRoutes[1].network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
result.bestRoutes[1].nextHopIp: 169.254.54.1
result.bestRoutes[1].nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
result.bestRoutes[1].priority: 100
result.bestRoutes[1].routeType: BGP
Add public subnet and bastion for VM access on AWS
create internet gateway
attach IGW to VPC
aws ec2 attach-internet-gateway --vpc-id "vpc-0f504a74c4317f091" --internet-gateway-id "igw-0dec90e7440b2163c" --region us-east-1
remove NATGW from private subnet - not needed - move to public subnet - remove black hole
create new NATGW with existing EIP
add route to IGW from public subnet
Create bastion
Been a while trying also https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html
IGW and NAT dont' have route table entries - adding 0.0.0.0/0 to IGW
fixed 1 of 2
regular ssh working now
michaelobrien@mbp7 pubsec-declarative-toolkit % ssh ubuntu@jump.obrienlabs.cloud
ssh: connect to host jump.obrienlabs.cloud port 22: Operation timed out
michaelobrien@mbp7 pubsec-declarative-toolkit % ssh ubuntu@jump.obrienlabs.cloud
The authenticity of host 'jump.obrienlabs.cloud (35.153.35.64)' can't be established.
ED25519 key fingerprint is SHA256:gXubQVYDc2kGW4PV1yJIn7x8DVWouqr1HgHKnfSAg04.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'jump.obrienlabs.cloud' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Sun Aug 27 15:20:44 UTC 2023
System load: 0.0 Processes: 95
Usage of /: 20.6% of 7.57GB Users logged in: 0
Memory usage: 23% IPv4 address for eth0: 10.51.2.201
Swap usage: 0%
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@ip-10-51-2-201:~$
ubuntu@ip-10-51-2-201:~$ dig www.google.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3028
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.google.com. IN A
;; ANSWER SECTION:
www.google.com. 139 IN A 172.253.122.104
www.google.com. 139 IN A 172.253.122.105
www.google.com. 139 IN A 172.253.122.106
www.google.com. 139 IN A 172.253.122.147
www.google.com. 139 IN A 172.253.122.99
www.google.com. 139 IN A 172.253.122.103
;; Query time: 4 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sun Aug 27 15:21:33 UTC 2023
;; MSG SIZE rcvd: 139
ubuntu@ip-10-51-2-201:~$ wget http://www.google.com
--2023-08-27 15:21:44-- http://www.google.com/
Resolving www.google.com (www.google.com)... 172.253.122.105, 172.253.122.106, 172.253.122.99, ...
Connecting to www.google.com (www.google.com)|172.253.122.105|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’
index.html [ <=> ] 18.17K --.-KB/s in 0.001s
2023-08-27 15:21:44 (34.8 MB/s) - ‘index.html’ saved [18611]
I see the private VM as expected on the private 1 subnet
ubuntu@ip-10-51-2-201:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=64 time=0.868 ms
on a mac
michaelobrien@mbp7 pubsec-declarative-toolkit % ssh -A ubuntu@jump.obrienlabs.cloud
on the bastion
ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.164
On the private VM
ubuntu@ip-10-51-1-164:~$
ubuntu@ip-10-51-1-164:~$ dig private.googleapis.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> private.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60142
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;private.googleapis.com. IN A
;; ANSWER SECTION:
private.googleapis.com. 300 IN A 199.36.153.8
private.googleapis.com. 300 IN A 199.36.153.9
private.googleapis.com. 300 IN A 199.36.153.10
private.googleapis.com. 300 IN A 199.36.153.11
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sun Aug 27 20:27:09 UTC 2023
;; MSG SIZE rcvd: 115
Fix nat on private subnet on aws - prior to viewing bgp dynamic routes from gcp
ping to google working on private subnet now
ubuntu@ip-10-51-1-164:~$ ping www.google.com
PING www.google.com (142.251.16.104) 56(84) bytes of data.
64 bytes from bl-in-f104.1e100.net (142.251.16.104): icmp_seq=1 ttl=51 time=3.15 ms
Add PGA route in AWS on private subnet not by pointing to the IGW but my advertising a custom route to 199.36.153.8/30 that is picked up by the VPN on the AWS side's BGP router
don't do below
Check Routes
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers get-status $ROUTER_NAME --region $REGION
kind: compute#routerStatusResponse
result:
bestRoutes:
- asPaths:
- asLists:
- 65501
pathSegmentType: AS_SEQUENCE
creationTimestamp: '2023-08-27T07:07:54.116-07:00'
destRange: 10.51.0.0/16
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.52.1
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
priority: 100
routeType: BGP
- asPaths:
- asLists:
- 65501
pathSegmentType: AS_SEQUENCE
creationTimestamp: '2023-08-27T07:17:49.427-07:00'
destRange: 10.51.0.0/16
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.1
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
bestRoutesForRouter:
- asPaths:
- asLists:
- 65501
pathSegmentType: AS_SEQUENCE
creationTimestamp: '2023-08-27T07:07:54.116-07:00'
destRange: 10.51.0.0/16
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.52.1
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
priority: 100
routeStatus: ACTIVE
routeType: BGP
- asPaths:
- asLists:
- 65501
pathSegmentType: AS_SEQUENCE
creationTimestamp: '2023-08-27T07:17:49.427-07:00'
destRange: 10.51.0.0/16
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.1
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeStatus: ACTIVE
routeType: BGP
bgpPeerStatus:
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.51.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.51.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
priority: 100
routeType: BGP
enableIpv6: false
ipAddress: 169.254.51.2
linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
md5AuthEnabled: false
name: aws-conn1-tunn1
numLearnedRoutes: 1
peerIpAddress: 169.254.51.1
state: Established
status: UP
uptime: 6 hours, 50 minutes, 8 seconds
uptimeSeconds: '24608'
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.52.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.52.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
priority: 100
routeType: BGP
enableIpv6: false
ipAddress: 169.254.52.2
linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
md5AuthEnabled: false
name: aws-conn1-tunn2
numLearnedRoutes: 1
peerIpAddress: 169.254.52.1
state: Established
status: UP
uptime: 6 hours, 44 minutes, 5 seconds
uptimeSeconds: '24245'
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.53.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.53.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
priority: 100
routeType: BGP
enableIpv6: false
ipAddress: 169.254.53.2
linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
md5AuthEnabled: false
name: aws-conn2-tunn1
numLearnedRoutes: 1
peerIpAddress: 169.254.53.1
state: Established
status: UP
uptime: 6 hours, 38 minutes, 10 seconds
uptimeSeconds: '23890'
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
enableIpv6: false
ipAddress: 169.254.54.2
linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
md5AuthEnabled: false
name: aws-conn2-tunn2
numLearnedRoutes: 1
peerIpAddress: 169.254.54.1
state: Established
status: UP
uptime: 6 hours, 34 minutes, 9 seconds
uptimeSeconds: '23649'
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
michael@cloudshell:~ (vpn-aws-obs)$
We can see the private google access CIDR in the first advertised route along with the private subnet in GCP VPC
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
We should not need to - as it does not affect the on-prem network - but turn on PGA for the GCP private subnet
gcloud compute instances create vpn-jump \
--project=vpn-aws-obs \
--zone=northamerica-northeast1-a \
--machine-type=e2-small \
--network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=vpn-aws-obs-vpc-sn \
--maintenance-policy=MIGRATE \
--provisioning-model=STANDARD \
--service-account=1027024291560-compute@developer.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--tags=http-server,https-server \
--create-disk=auto-delete=yes,boot=yes,device-name=vpn-jump,image=projects/debian-cloud/global/images/debian-11-bullseye-v20230814,mode=rw,size=10,type=projects/vpn-aws-obs/zones/us-central1-a/diskTypes/pd-balanced \
--no-shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring \
--labels=goog-ec-src=vm_add-gcloud \
--reservation-affinity=any
on ssh
Please consider adding a firewall rule to allow ingress from the Cloud IAP for TCP forwarding netblock (from range 35.235.240.0/20) to the SSH port of your machine to start using Cloud IAP for TCP forwarding for better performance.
delete the default VPC
open the firewall
gcloud compute --project=vpn-aws-obs firewall-rules create allopen-ingress --direction=INGRESS --priority=1000 --network=vpn-aws-obs-vpc --action=ALLOW --rules=all --source-ranges=0.0.0.0/0
ssh to gcp vm
michael@vpn-jump:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
^C
--- 10.51.1.164 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2031ms
create a private instance - post PGA without public IP
gcloud compute instances create vpn-jump-private2 \
--project=vpn-aws-obs \
--zone=northamerica-northeast1-a \
--machine-type=e2-small \
--network-interface=stack-type=IPV4_ONLY,subnet=vpn-aws-obs-vpc-sn,no-address \
--maintenance-policy=MIGRATE \
--provisioning-model=STANDARD \
--service-account=1027024291560-compute@developer.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--tags=http-server,https-server \
--create-disk=auto-delete=yes,boot=yes,device-name=vpn-jump-private2,image=projects/debian-cloud/global/images/debian-11-bullseye-v20230814,mode=rw,size=10,type=projects/vpn-aws-obs/zones/us-central1-a/diskTypes/pd-balanced \
--no-shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring \
--labels=goog-ec-src=vm_add-gcloud \
--reservation-affinity=any
however we see AWS routes in GCP - but only the VPC - which may be subnet/vpc routing on the AWS side
vpn-aws-obs-router-aws-conn1-tunn2-1123490692 | Dynamic | IPv4 | 10.51.0.0/16 | 100 | None | VPN tunnel tunnel-2 |
-- | -- | -- | -- | -- | -- | -- | --
vpn-aws-obs-router-aws-conn2-tunn2-1369310470 | Dynamic | IPv4 | 10.51.0.0/16 | 100 | None | VPN tunnel tunnel-4
vpn-aws-obs-router-aws-conn1-tunn2-1123490692 Dynamic IPv4 10.51.0.0/16 100 None VPN tunnel [tunnel-2](https://console.cloud.google.com/hybrid/vpn/tunnels/details/northamerica-northeast1/tunnel-2?project=vpn-aws-obs)
vpn-aws-obs-router-aws-conn2-tunn2-1369310470 Dynamic IPv4 10.51.0.0/16 100 None VPN tunnel [tunnel-4](https://console.cloud.google.com/hybrid/vpn/tunnels/details/northamerica-northeast1/tunnel-4?project=vpn-aws-obs)
check netstat from both vms
AWS
ubuntu@ip-10-51-1-164:~$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.51.1.1 0.0.0.0 UG 0 0 0 eth0
10.51.0.2 10.51.1.1 255.255.255.255 UGH 0 0 0 eth0
10.51.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.51.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
GCP
michael@vpn-jump-private2:~$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.101.0.1 0.0.0.0 UG 0 0 0 ens4
10.101.0.1 0.0.0.0 255.255.255.255 UH 0 0 0 ens4
sudo apt install net-tools
first verify connectivity between VMs in both CSPs - spin up 2 more VMs
we are good on GCP
michael@vpn-jump-private2:~$ ping 10.101.0.5
PING 10.101.0.5 (10.101.0.5) 56(84) bytes of data.
64 bytes from 10.101.0.5: icmp_seq=1 ttl=64 time=1.56 ms
AWS good
ubuntu@ip-10-51-1-164:~$ ping 10.51.1.185
PING 10.51.1.185 (10.51.1.185) 56(84) bytes of data.
64 bytes from 10.51.1.185: icmp_seq=1 ttl=64 time=1.15 ms
ubuntu@ip-10-51-1-185:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=64 time=0.627 ms
Checking AWS propagation on the routes - off - this may be the issue
editing route propagation to route through the VPG
after
Routing working now from GCP to AWS
michael@vpn-jump-private2:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=63 time=17.7 ms
we can ping from GCP to AWS (prem) 10.101.0.0/16 to 10.51.0.0/16
and we can ping the reverse AWS(prem) to GCP 10.51.0.0/16 to 10.101.0.0/16
ubuntu@ip-10-51-1-185:~$ ping 10.101.0.4
PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.
64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=16.7 ms
the other VM
Last login: Sun Aug 27 20:25:15 2023 from 10.51.2.201
ubuntu@ip-10-51-1-164:~$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.51.1.1 0.0.0.0 UG 0 0 0 eth0
10.51.0.2 10.51.1.1 255.255.255.255 UGH 0 0 0 eth0
10.51.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.51.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
ubuntu@ip-10-51-1-164:~$ ping 10.101.0.4
PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.
64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=17.0 ms
baseline IP from AWS to GCP
ubuntu@ip-10-51-1-164:~$ curl https://199.36.153.8:443
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
ubuntu@ip-10-51-1-164:~$ curl http://199.36.153.8
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 401 (Unauthorized)!!1</title>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>401.</b> <ins>That’s an error.</ins>
<p>Your client does not have permission to the requested URL <code>/</code>. <ins>That’s all we know.</ins>
https://us-east-1.console.aws.amazon.com/route53/v2/hostedzones?region=us-east-1#CreateHostedZone
Plural A records use CR/LF separators
private.googleapis.com = 199.36.153.8,199.36.153.9,199.36.153.10,199.36.153.11
For each VPC that you associate with a private hosted zone, you must set the Amazon VPC settings enableDnsHostnames and enableDnsSupport to true.
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-updating
The "enable dns hostnames" was not set - we don't want it set - just "resolution" should be set
Do a dig on the private VM inside AWS
ubuntu@ip-10-51-1-164:~$ dig private.googleapis.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> private.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40574
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;private.googleapis.com. IN A
;; ANSWER SECTION:
private.googleapis.com. 300 IN A 199.36.153.10
private.googleapis.com. 300 IN A 199.36.153.11
private.googleapis.com. 300 IN A 199.36.153.8
private.googleapis.com. 300 IN A 199.36.153.9
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Aug 28 02:24:41 UTC 2023
;; MSG SIZE rcvd: 115
check an record not already in the private zone before we add it
ubuntu@ip-10-51-1-164:~$ dig storage.googleapis.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> storage.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42780
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;storage.googleapis.com. IN A
;; ANSWER SECTION:
storage.googleapis.com. 167 IN A 142.251.167.128
storage.googleapis.com. 167 IN A 172.253.62.128
storage.googleapis.com. 167 IN A 172.253.63.128
storage.googleapis.com. 167 IN A 172.253.115.128
storage.googleapis.com. 167 IN A 172.253.122.128
storage.googleapis.com. 167 IN A 142.250.31.128
storage.googleapis.com. 167 IN A 142.251.16.128
storage.googleapis.com. 167 IN A 142.251.111.128
storage.googleapis.com. 167 IN A 142.251.163.128
on my macbook
michaelobrien@mbp7 pubsec-declarative-toolkit % dig storage.googleapis.com
; <<>> DiG 9.10.6 <<>> storage.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31183
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;storage.googleapis.com. IN A
;; ANSWER SECTION:
storage.googleapis.com. 292 IN A 172.217.13.208
storage.googleapis.com. 292 IN A 172.217.13.112
storage.googleapis.com. 292 IN A 172.217.13.144
storage.googleapis.com. 292 IN A 172.217.13.176
private domain not having effect on the VPC - checking dhcp options
Turn on VPC flow logs
Checking traceroute
ubuntu@ip-10-51-1-185:~$ traceroute storage.googleapis.com
traceroute to storage.googleapis.com (142.251.163.128), 64 hops max
1 10.51.2.64 0.214ms 0.193ms 0.179ms
2 216.182.230.245 2.080ms 216.182.239.185 1.653ms 9.689ms
3 100.65.96.144 8.767ms 100.65.50.16 7.394ms 8.256ms
4 100.66.36.92 63.251ms 100.66.25.194 19.483ms 12.593ms
5 241.0.11.192 0.994ms 100.66.26.212 1.404ms 1.496ms
6 241.0.11.216 0.783ms 0.329ms 0.344ms
7 240.1.208.12 16.359ms 15.870ms 15.808ms
8 100.65.91.4 45.599ms 240.1.208.14 15.816ms 15.595ms
9 99.83.71.68 15.827ms 15.361ms 15.359ms
10 142.251.55.118 17.134ms 216.239.43.24 17.432ms 17.353ms
11 241.0.11.195 0.970ms * *
12 240.1.208.12 17.303ms 108.170.249.162 17.040ms 22.631ms
13 108.170.249.67 15.965ms 240.1.208.14 15.805ms 15.751ms
14 172.253.68.51 17.195ms 16.947ms 17.158ms
15 * * *
16 * * *
17 108.170.249.108 17.892ms 17.381ms 17.306ms
18 172.253.65.78 15.935ms 15.328ms 15.245ms
19 216.239.40.133 18.348ms 18.743ms 19.057ms
20 142.250.209.70 18.043ms * *
21 * * *
22 * * *
23 209.85.252.200 16.217ms 15.756ms 15.844ms
24 * * *
25 * * *
26 * * *
27 * * *
ubuntu@ip-10-51-1-185:~$ traceroute private.googleapis.com
traceroute to private.googleapis.com (199.36.153.9), 64 hops max
1 * *
check /etc/hosts.txt override
199.36.153.14 storage.googleapis.com
break it on purpose first to verify we don't get through a public secondary DNS
ubuntu@ip-10-51-1-185:~$ sudo vi /etc/hosts
ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
INFO 0828 03:48:20.911258 retry_util.py] Retrying request, attempt #1...
^CCaught CTRL-C (signal 2) - exiting
ubuntu@ip-10-51-1-185:~$ ping storage.googleapis.com
PING storage.googleapis.com (199.36.153.28) 56(84) bytes of data.
then set to a valid ip
ubuntu@ip-10-51-1-185:~$ sudo vi /etc/hosts
ubuntu@ip-10-51-1-185:~$ ping storage.googleapis.com
PING storage.googleapis.com (199.36.153.8) 56(84) bytes of data.
^C
--- storage.googleapis.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1029ms
ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.png
ubuntu@ip-10-51-1-185:~$
working only through the private IP as 199.36.153.8 is not routable on the internet
Install gcloud cli https://cloud.google.com/sdk/docs/install#linux
ubuntu@ip-10-51-1-185:~$ sudo apt-get update
Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
ubuntu@ip-10-51-1-185:~$ curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-444.0.0-linux-x86_64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 178M 100 178M 0 0 7618k 0 0:00:24 0:00:24 --:--:-- 1582k
ubuntu@ip-10-51-1-185:~$ tar -xf google-cloud-cli-444.0.0-linux-x86_64.tar.gz
ubuntu@ip-10-51-1-185:~$ ./google-cloud-sdk/install.sh
Welcome to the Google Cloud CLI!
To help improve the quality of this product, we collect anonymized usage data
and anonymized stacktraces when crashes are encountered; additional information
is available at <https://cloud.google.com/sdk/usage-statistics>. This data is
handled in accordance with our privacy policy
<https://cloud.google.com/terms/cloud-privacy-notice>. You may choose to opt in this
collection now (by choosing 'Y' at the below prompt), or at any time in the
future by running the following command:
gcloud config set disable_usage_reporting false
Do you want to help improve the Google Cloud CLI (y/N)? y
Your current Google Cloud CLI version is: 444.0.0
The latest available version is: 444.0.0
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Components │
├───────────────┬──────────────────────────────────────────────────────┬──────────────────────────────┬──────────┤
│ Status │ Name │ ID │ Size │
├───────────────┼──────────────────────────────────────────────────────┼──────────────────────────────┼──────────┤
│ Not Installed │ App Engine Go Extensions │ app-engine-go │ 4.5 MiB │
│ Not Installed │ Appctl │ appctl │ 21.0 MiB │
│ Not Installed │ Artifact Registry Go Module Package Helper │ package-go-module │ < 1 MiB │
│ Not Installed │ Cloud Bigtable Command Line Tool │ cbt │ 11.4 MiB │
│ Not Installed │ Cloud Bigtable Emulator │ bigtable │ 7.0 MiB │
│ Not Installed │ Cloud Datastore Emulator │ cloud-datastore-emulator │ 36.2 MiB │
│ Not Installed │ Cloud Firestore Emulator │ cloud-firestore-emulator │ 42.5 MiB │
│ Not Installed │ Cloud Pub/Sub Emulator │ pubsub-emulator │ 61.2 MiB │
│ Not Installed │ Cloud Run Proxy │ cloud-run-proxy │ 13.4 MiB │
│ Not Installed │ Cloud SQL Proxy │ cloud_sql_proxy │ 7.8 MiB │
│ Not Installed │ Cloud Spanner Emulator │ cloud-spanner-emulator │ 31.4 MiB │
│ Not Installed │ Cloud Spanner Migration Tool │ harbourbridge │ 20.9 MiB │
│ Not Installed │ Google Container Registry's Docker credential helper │ docker-credential-gcr │ 1.8 MiB │
│ Not Installed │ Kustomize │ kustomize │ 4.3 MiB │
│ Not Installed │ Log Streaming │ log-streaming │ 13.9 MiB │
│ Not Installed │ Minikube │ minikube │ 34.6 MiB │
│ Not Installed │ Nomos CLI │ nomos │ 26.4 MiB │
│ Not Installed │ On-Demand Scanning API extraction helper │ local-extract │ 14.4 MiB │
│ Not Installed │ Skaffold │ skaffold │ 22.7 MiB │
│ Not Installed │ Spanner migration tool │ spanner-migration-tool │ 20.9 MiB │
│ Not Installed │ Terraform Tools │ terraform-tools │ 66.1 MiB │
│ Not Installed │ anthos-auth │ anthos-auth │ 20.4 MiB │
│ Not Installed │ config-connector │ config-connector │ 56.7 MiB │
│ Not Installed │ enterprise-certificate-proxy │ enterprise-certificate-proxy │ 8.2 MiB │
│ Not Installed │ gcloud Alpha Commands │ alpha │ < 1 MiB │
│ Not Installed │ gcloud Beta Commands │ beta │ < 1 MiB │
│ Not Installed │ gcloud app Java Extensions │ app-engine-java │ 65.1 MiB │
│ Not Installed │ gcloud app Python Extensions │ app-engine-python │ 8.5 MiB │
│ Not Installed │ gcloud app Python Extensions (Extra Libraries) │ app-engine-python-extras │ 27.3 MiB │
│ Not Installed │ gke-gcloud-auth-plugin │ gke-gcloud-auth-plugin │ 7.9 MiB │
│ Not Installed │ kpt │ kpt │ 14.5 MiB │
│ Not Installed │ kubectl │ kubectl │ < 1 MiB │
│ Not Installed │ kubectl-oidc │ kubectl-oidc │ 20.4 MiB │
│ Not Installed │ pkg │ pkg │ │
│ Installed │ BigQuery Command Line Tool │ bq │ 1.6 MiB │
│ Installed │ Bundled Python 3.9 │ bundled-python3-unix │ 63.6 MiB │
│ Installed │ Cloud Storage Command Line Tool │ gsutil │ 11.3 MiB │
│ Installed │ Google Cloud CLI Core Libraries │ core │ 21.6 MiB │
│ Installed │ Google Cloud CRC32C Hash Tool │ gcloud-crc32c │ 1.2 MiB │
└───────────────┴──────────────────────────────────────────────────────┴──────────────────────────────┴──────────┘
To install or remove components at your current SDK version [444.0.0], run:
$ gcloud components install COMPONENT_ID
$ gcloud components remove COMPONENT_ID
To update your SDK installation to the latest version [444.0.0], run:
$ gcloud components update
Modify profile to update your $PATH and enable shell command completion?
Last login: Mon Aug 28 03:02:10 2023 from 10.51.2.201
ubuntu@ip-10-51-1-185:~$ gcloud init
Welcome! This command will take you through the configuration of gcloud.
Your current configuration has been set to: [default]
You can skip diagnostics next time by using the following flag:
gcloud init --skip-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).
You must log in to continue. Would you like to log in (Y/n)?
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32555940559.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fauthcode.html&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=TNrMg266uDdOt4gnpc9Lg5yTRL5B9L&prompt=consent&access_type=offline&code_challenge=kKbf_2KEOisLiCLtxbK7toHy4M6iIHya_lUDJIc85qk&code_challenge_method=S256
You are logged in as: [michael@obrien.software].
Pick cloud project to use:
[1] all-services-obs
[2] bce-dev-so
[3] cs-host-6e8fdca189434184a30c0c
[4] cuda-obs
[5] dep1-team1-dev-os
[6] dep1-team1-nprod-os
[7] dns-sim-prem
[8] dns-sim-prod-host
[9] dns-sim-prod-service1
[10] logging-hh015-gz357
[11] manual-lz
[12] monitoring-dev-hh015-gz357
[13] monitoring-nonprod-hh015-gz357
[14] monitoring-prod-hh015-gz357
[15] vpc-host-nonprod-hh015-gz357
[16] vpc-host-prod-hh015-gz357
[17] vpn-aws-obs
[18] vpn-aws-svc-obs
[19] Enter a project ID
[20] Create a new project
Please enter numeric choice or text value (must exactly match list item): 17
Your current project has been set to: [vpn-aws-obs].
Do you want to configure a default Compute Region and Zone? (Y/n)? y
Which Google Compute Engine zone would you like to use as project default?
If you do not specify a zone via a command line flag while working with Compute Engine resources, the default is assumed.
[1] us-east1-b
... [115] us-west4-c
[116] Do not set default zone
Please enter numeric choice or text value (must exactly match list item): 92
Your project default Compute Engine zone has been set to [northamerica-northeast1-a].
You can change it by running [gcloud config set compute/zone NAME].
Your project default Compute Engine region has been set to [northamerica-northeast1].
You can change it by running [gcloud config set compute/region NAME].
Created a default .boto configuration file at [/home/ubuntu/.boto]. See this file and
[https://cloud.google.com/storage/docs/gsutil/commands/config] for more
information about configuring Google Cloud Storage.
Your Google Cloud SDK is configured and ready to use!
* Commands that require authentication will use michael@obrien.software by default
* Commands will reference project `vpn-aws-obs` by default
* Compute Engine commands will use region `northamerica-northeast1` by default
* Compute Engine commands will use zone `northamerica-northeast1-a` by default
Run `gcloud help config` to learn how to change individual settings
This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects.
Run `gcloud topic configurations` to learn more.
Some things to try next:
* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.
* Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting
* Run `gcloud cheat-sheet` to see a roster of go-to `gcloud` commands.
ubuntu@ip-10-51-1-185:~$ gcloud config set project vpn-aws-obs
Updated property [core/project].
How do I verify that googleapis.com traffic is not going through the NAT or IGW on the public subnet - check routes
Destination | Target | Status | Propagated |
---|---|---|---|
0.0.0.0/0 | nat-05be2cd683e8b153c | Active | No |
10.51.0.0/16 | local | Active | No |
10.101.0.0/24 | vgw-0bd4928d84cd855cf | Active | Yes |
199.36.153.8/30 | vgw-0bd4928d84cd855cf | Active | Yes |
Destination Target Status Propagated 0.0.0.0/0 nat-05be2cd683e8b153c Active No 10.51.0.0/16 local Active No 10.101.0.0/24 vgw-0bd4928d84cd855cf Active Yes 199.36.153.8/30 vgw-0bd4928d84cd855cf Active Yes
Add a GCS bucket to be able to list/update from AWS
from AWS
ubuntu@ip-10-51-1-185:~$ gsutil ls
gs://aws-transfer-obs/
ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.png
Update: PGA is working as expected from on-prem (simulated by an AWS VPC via VPN - which works well as a non-GCP on-prem/ground) following procedures in the github issue below. Essentially the main changes are what is detailed in the deck and docs. Verified 1 and 2
1 - Propagate routes on both sides but with an additional static route on the 199.36.153.8/30 CIDR from the GCP side so that googleapis.com calls can use that BGP route instead of the on prem IG public IPs (199.* is not publicly routable).
2 - Add a on-prem private zone (Route53 in AWS) to A record the 4 199.36.153.8/30 PGA IPs.
3 - optionally add forwarding private zone from prem to gcp
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494
on an AWS private VM (ping back to GCP VM, run a GCS ls) ubuntu@ip-10-51-1-185:~$ ping 10.101.0.4 PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=17.7 ms ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/ gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.png
Add PSC Private Service Connect capability through endpoints (for now focus on onprem to GCP)
Tasks for gcloud and awscli - (KRM/Terraform/CloudFormation later)
Check network intelligence
follow https://cloud.google.com/vpc/docs/configure-private-service-connect-apis existing services
michael@cloudshell:~ (vpn-aws-obs)$ gcloud services list
NAME: bigquery.googleapis.com
TITLE: BigQuery API
NAME: bigquerymigration.googleapis.com
TITLE: BigQuery Migration API
NAME: bigquerystorage.googleapis.com
TITLE: BigQuery Storage API
NAME: cloudapis.googleapis.com
TITLE: Google Cloud APIs
NAME: cloudtrace.googleapis.com
TITLE: Cloud Trace API
NAME: compute.googleapis.com
TITLE: Compute Engine API
NAME: datastore.googleapis.com
TITLE: Cloud Datastore API
NAME: logging.googleapis.com
TITLE: Cloud Logging API
NAME: monitoring.googleapis.com
TITLE: Cloud Monitoring API
NAME: networkmanagement.googleapis.com
TITLE: Network Management API
NAME: oslogin.googleapis.com
TITLE: Cloud OS Login API
NAME: servicemanagement.googleapis.com
TITLE: Service Management API
NAME: serviceusage.googleapis.com
TITLE: Service Usage API
NAME: sql-component.googleapis.com
TITLE: Cloud SQL
NAME: storage-api.googleapis.com
TITLE: Google Cloud Storage JSON API
NAME: storage-component.googleapis.com
TITLE: Cloud Storage
NAME: storage.googleapis.com
TITLE: Cloud Storage API
NAME: websecurityscanner.googleapis.com
TITLE: Web Security Scanner API
enable services
gcloud services enable networkmanagement.googleapis.com
on top of compute add
gcloud services enable servicedirectory.googleapis.com
gcloud services enable networkconnectivity.googleapis.com
gcloud services enable dns.googleapis.com
IP for endpoint - pick one from the VPC but not in a subnet
10.101.0.6
verify ingress 0.0.0.0/0 implies egress (stateful firewall)
Check existing private DNS zone p.googleapis.com none existing = ok
Verify PGA on the subnet
export REGION=northamerica-northeast1
export NETWORK=vpn-aws-obs-vpc
export SUBNET_NAME=$NETWORK-sn
gcloud compute networks subnets describe $SUBNET_NAME --region=$REGION --format="get(privateIpGoogleAccess)"
True
create the endpoint https://cloud.google.com/vpc/docs/configure-private-service-connect-apis#create-endpoint
reserve an alternate IP
export PSC_ENDPOINT_IP=10.101.2.7
gcloud compute addresses create psc-endpoint-ip --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=$PSC_ENDPOINT_IP --network=$NETWORK
Created [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/addresses/psc-endpoint-ip].
Create forwarding rule (no namespaces/namespace)
export PROJECT_ID=vpn-aws-obs
gcloud compute forwarding-rules create pscendpoint --global --network=$NETWORK --address=psc-endpoint-ip --target-google-apis-bundle=all-apis --service-directory-registration=projects/$PROJECT_ID/locations/$REGION
verify endpoint
export PSC_ENDPOINT_IP=10.101.2.7
curl -v $PSC_ENDPOINT_IP/generate_204
getting a timeout - placing Ip outside the VPC
michael@cloudshell:~ (vpn-aws-obs)$ curl -v $PSC_ENDPOINT_IP/generate_204
* Trying 10.101.2.7:80...
* connect to 10.101.2.7 port 80 failed: Connection timed out
* Failed to connect to 10.101.2.7 port 80: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to 10.101.2.7 port 80: Connection timed out
michael@cloudshell:~ (vpn-aws-obs)$ export PSC_ENDPOINT_IP=10.102.0.2
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute addresses create psc-endpoint-ip2 --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=$PSC_ENDPOINT_IP --network=$NETWORK
Created [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/addresses/psc-endpoint-ip2].
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute forwarding-rules create pscendpoint2 --global --network=$NETWORK --address=psc-endpoint-ip2 --target-google-apis-bundle=all-apis --service-directory-registration=projects/$PROJECT_ID/locations/$REGION
Created [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/forwardingRules/pscendpoint2].
michael@cloudshell:~ (vpn-aws-obs)$ curl -v $PSC_ENDPOINT_IP/generate_204
* Trying 10.102.0.2:80...
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute forwarding-rules list \
--filter target="(all-apis OR vpc-sc)" --global
NAME: pscendpoint
REGION:
IP_ADDRESS: 10.101.2.7
IP_PROTOCOL: TCP
TARGET: all-apis
NAME: pscendpoint2
REGION:
IP_ADDRESS: 10.102.0.2
IP_PROTOCOL: TCP
TARGET: all-apis
DNS zone was created by the endpoint
Service Directory Namespace
add route 35.199.192.0/19
AWS side has extra route
add private zone on gcp side - that will be advertised above
gcloud dns --project=vpn-aws-obs managed-zones create gcp-local --description="" --dns-name="gcp.local." --visibility="private" --networks="https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc"
add an A record to the private zone for the jump server on gcp - first reserve an internal IP (in case we reboot)
add compute.networkAdmin role
gcloud dns --project=vpn-aws-obs record-sets create jump.gcp.local. --zone="gcp-local" --type="A" --ttl="300" --rrdatas="10.101.0.4"
Add AWS route53 local private zone for gcp.local
Add custom route for PSC endpoint on the router for 10.101.2.7
check propagated routes
Add DNS ingress policy
gcloud dns --project=vpn-aws-obs policies create test-inbound --description="" --networks="" --enable-inbound-forwarding
gcloud dns --project=vpn-aws-obs policies update test-inbound --description="" --networks="vpn-aws-obs-vpc" --alternative-name-servers="" --private-alternative-name-servers="" --enable-inbound-forwarding --no-enable-logging
fix custom route to the PSC endpoint - use /32
testing the endpoint DNS name before we add a record to the PSC ip
ubuntu@ip-10-51-1-185:~$ nslookup bigquery-pscendpoint.p.googleapis.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: bigquery-pscendpoint.p.googleapis.com
Address: 172.253.122.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 142.250.31.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 142.251.16.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 142.251.111.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 142.251.163.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 142.251.167.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 172.253.62.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 172.253.63.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 172.253.115.95
Name: bigquery-pscendpoint.p.googleapis.com
Address: 2607:f8b0:4004:c1b::5f
Name: bigquery-pscendpoint.p.googleapis.com
Address: 2607:f8b0:4004:c1d::5f
Name: bigquery-pscendpoint.p.googleapis.com
Address: 2607:f8b0:4004:c07::5f
Name: bigquery-pscendpoint.p.googleapis.com
Address: 2607:f8b0:4004:c17::5f
Adding Route53 resolver to p.googleapis.com
In this case the client does not need pipeline access -manual after VPC/service-enablement provided https://registry.terraform.io/modules/terraform-google-modules/data-fusion/google/latest
For datafusion the SA will be developer and the wizard adds the runner role
https://console.cloud.google.com/data-fusion/locations/-/instances?project=vpn-aws-obs https://cloud.google.com/data-fusion/docs/how-to/granting-service-account-permission?_ga=2.94099933.-1155012423.1687982978
TODO TF add to
datafusion.googleapis.com
Verify bq enablement
look for Datafusion terraform https://registry.terraform.io/modules/terraform-google-modules/data-fusion/google/latest
See related forwarding rule issue https://github.com/GoogleCloudPlatform/magic-modules/pull/7480 https://github.com/hashicorp/terraform-provider-google/issues/9758
Testing the PSC private endpoint in VPC and on-prem
GCP based
internet based
ichael@cloudshell:~ (vpn-aws-obs)$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage.googleapis.com/storage/v1/b?project=$(gcloud config get-value project)"
Your active configuration is: [cloudshell-2578]
{
"kind": "storage#buckets",
"items": [....
private based (notice that the CNAME record is wildcarded - so any psc application name works
michael@cloudshell:~ (vpn-aws-obs)$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage-anyrandomstr.p.googleapis.com/storage/v1/b?project=$(gcloud config get-value project)"
Your active configuration is: [cloudshell-2578]
{
"kind": "storage#buckets",
"items": [
{
"kind": "storage#bucket",
"selfLink": "https://www.googleapis.com/storage/v1/b/aws-transfer-obs",
"id": "aws-transfer-obs",
"name": "aws-transfer-obs",
"projectNumber": "1027024291560",
"metageneration": "1",
"location": "NORTHAMERICA-NORTHEAST1",
"storageClass": "STANDARD",
"etag": "CAE=",
"timeCreated": "2023-08-28T03:24:27.491Z",
"updated": "2023-08-28T03:24:27.491Z",
"iamConfiguration": {
"bucketPolicyOnly": {
"enabled": true,
"lockedTime": "2023-11-26T03:24:27.491Z"
},
"uniformBucketLevelAccess": {
"enabled": true,
"lockedTime": "2023-11-26T03:24:27.491Z"
},
"publicAccessPrevention": "enforced"
},
"locationType": "region"
},
{
"kind": "storage#bucket",
"selfLink": "https://www.googleapis.com/storage/v1/b/df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
"id": "df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
"name": "df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
"projectNumber": "1027024291560",
"metageneration": "1",
"location": "NORTHAMERICA-NORTHEAST1",
"storageClass": "STANDARD",
"etag": "CAE=",
"timeCreated": "2023-08-29T14:52:20.813Z",
"updated": "2023-08-29T14:52:20.813Z",
"lifecycle": {
"rule": [
{
"action": {
"type": "Delete"
},
"condition": {
"daysSinceCustomTime": 20
}
}
]
},
"labels": {
"cdf_instance": "test"
},
"iamConfiguration": {
"bucketPolicyOnly": {
"enabled": false
},
"uniformBucketLevelAccess": {
"enabled": false
},
"publicAccessPrevention": "inherited"
},
"locationType": "region"
}
]
}
michael@cloudshell:~ (vpn-aws-obs)$
add CNAME record *.googleapis.com pointing to private.googleapis.com
add A record private.googleapis.com pointing to the IP for the PSC endpoint
delete the PGA 199.36.153.8/30 first
then add one to 10.102.0.2
Missing on manual PSC setup - the googleapis.com private zone with CNAME and A record to private.googleapis.com (see TF created PSC)
gcloud dns --project=vpn-aws-obs managed-zones create dz-apis --description="Private DNS zone to configure private.googleapis.com" --dns-name="googleapis.com." --visibility="private" --networks="https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc"
gcloud dns --project=vpn-aws-obs record-sets create *.googleapis.com. --zone="dz-apis" --type="CNAME" --ttl="300" --rrdatas="private.googleapis.com."
gcloud dns --project=vpn-aws-obs record-sets create private.googleapis.com. --zone="dz-apis" --type="A" --ttl="300" --rrdatas="10.102.0.2"
wait for DNS propagation
ubuntu@ip-10-51-1-185:~$ nslookup private.googleapis.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: private.googleapis.com
Address: 199.36.153.10
Name: private.googleapis.com
Address: 199.36.153.11
Name: private.googleapis.com
Address: 199.36.153.8
Name: private.googleapis.com
Address: 199.36.153.9
AWS based
michaelobrien@mbp7 pbmm-on-gcp-onboarding % ssh -A ubuntu@jump.obrienlabs.cloud
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)
*** System restart required ***
Last login: Tue Sep 5 13:21:59 2023 from 174.112.128.160
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.185
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)
*** System restart required ***
Last login: Tue Sep 5 13:22:16 2023 from 10.51.2.201
change route advertised for the PSC endpoint ip
on the private AWS vm - need a pre-generated token for the -H header
ubuntu@ip-10-51-1-185:~$ curl -X GET "https://10.102.0.2/storage/v1/b?project=vpn-aws-obs"
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Public/private bq rest calls public first
get auth token https://cloud.google.com/bigquery/docs/authorization https://cloud.google.com/docs/authentication/provide-credentials-adc#how-to https://github.com/GoogleCloudPlatform/shell-samples/blob/HEAD/bigquery/authorization/snippets.sh
gcloud auth application-default login
after we work with
gcloud auth application-default print-access-token
ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
create a default big query table - in this case one of my GPS data exports 201611185_gps_distinct_rollerblad.csv
biometric:benchmark michaelobrien$ export GOOGLE_CLOUD_PROJECT=bigquery-ol
biometric:benchmark michaelobrien$ curl -H "Authorization: Bearer $ACCESS_TOKEN" \
> "https://www.googleapis.com/bigquery/v2/projects/$GOOGLE_CLOUD_PROJECT/datasets"
{
"kind": "bigquery#datasetList",
"etag": "irRvAdMvQcDWywbnuMPyBg==",
"datasets": [
{
"kind": "bigquery#dataset",
"id": "bigquery-ol:rollerblade",
"datasetReference": {
"datasetId": "rollerblade",
"projectId": "bigquery-ol"
},
"location": "northamerica-northeast1"
}
]
}
curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade
{
"kind": "bigquery#dataset",
"etag": "YGr4j2cVE8o77F0DFWySFg==",
"id": "bigquery-ol:rollerblade",
"selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade",
"datasetReference": {
"datasetId": "rollerblade",
"projectId": "bigquery-ol"
},
...
}
biometric:benchmark michaelobrien$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"kind": "bigquery#tableDataList",
"etag": "wVKEep3xEdb51S3g1w7o3w==",
"totalRows": "4331",
"pageToken": "BHGUBPMJRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAWYTJM5YXKZLSPEWW63ARQX7PZB65AAAAAEQLOJXWY3DFOJRGYYLEMUNCINLEHE2GIYTEHEWWMMJUMUWTIM3GMQWTQZJQMYWTEMJVGFRWEOJTME3DMMQSBNZG63DMMVZGE3DBMRSRUJBYG5RTQNZUGM3S2YZZGBSC2NBWGVTC2OJYGA4C2ZJXGY4WGMBTGRSGEMLE",
"rows": [
{
"f": [
{
"v": "45.424712"
},
{
"v": "-75.698209"
}
]
},
{
"f": [
{
"v": "45.424712"
},
{
"v": "-75.6949"
}
]
}
]
}
On AWS private VM back to GCP via public
ubuntu@ip-10-51-1-185:~$ nslookup bigquery.googleapis.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
bigquery.googleapis.com canonical name = private.googleapis.com.
Name: private.googleapis.com
Address: 10.102.0.2
GCP: 10.102.0.2 is the PSC endpoint 2 on GCP
AWS: Because we have a CNAME from the wildcard *.googleapis.com to private.googleapis.com - the A record 10.102.0.2 in Route53
michaelobrien@mbp7 pbmm-on-gcp-onboarding % ssh -A ubuntu@jump.obrienlabs.cloud
ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.185
ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKENcurl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"kind": "bigquery#tableDataList",
"etag": "xTHaKWTlPPkd2/tRo1kgDQ==",
"totalRows": "4331",
"pageToken": "BHQG7VMKRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAW5TQNYWWC53TFVXWE4YR5ADGWH7PAAAAAEQLOJXWY3DFOJRGYYLEMUNCINZRGMYTONRXGYWTEZRSGIWTIMRUGYWTQZLEMYWTINDGGM4WKNDEGQ4WGYISBNZG63DMMVZGE3DBMRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
"rows": [
{
"f": [
{
"v": "45.424712"
},
{
"v": "-75.698209"
}
]
},
{
"f": [
{
"v": "45.424712"
},
{
"v": "-75.6949"
}
]
}
]
}
re-authenticate on each login
On AWS private VM back to GCP via private PSC check VPC flow logs https://cloud.google.com/vpc/docs/about-accessing-vpc-hosted-services-endpoints#logging https://console.cloud.google.com/logs/query;query=resource.type%3D%22gce_subnetwork%22%0Aresource.labels.subnetwork_id%3D%228559254801916337169%22%0Aresource.labels.subnetwork_name%3D%22vpn-aws-obs-vpc-sn%22;cursorTimestamp=2023-09-12T20:04:01.752941560Z;duration=PT5M?project=vpn-aws-obs&pli=1&rapt=AEjHL4Mo6P4M3IbIcrStuUx11l0uR6yW-FgSwD7Nj_Ww7a_v-8WxH9uu8N078EVWo10pRmlFecvDDxrjE93KQrJ6qYEuvlRFZw
from gcp jump only
{
"insertId": "1uvikfff5i6eii",
"jsonPayload": {
"connection": {
"dest_port": 443,
"dest_ip": "10.102.0.2",
"protocol": 6,
"src_ip": "10.101.0.4",
"src_port": 42072
},
"end_time": "2023-09-12T20:06:14.947442181Z",
"bytes_sent": "2804",
"start_time": "2023-09-12T20:06:14.924006768Z",
"packets_sent": "24",
"src_instance": {
"zone": "northamerica-northeast1-a",
"project_id": "vpn-aws-obs",
"region": "northamerica-northeast1",
"vm_name": "vpn-jump-private2"
},
"reporter": "SRC",
"src_vpc": {
"vpc_name": "vpn-aws-obs-vpc",
"subnetwork_name": "vpn-aws-obs-vpc-sn",
"project_id": "vpn-aws-obs"
}
},
"resource": {
"type": "gce_subnetwork",
"labels": {
"subnetwork_name": "vpn-aws-obs-vpc-sn",
"location": "northamerica-northeast1-a",
"subnetwork_id": "8559254801916337169",
"project_id": "vpn-aws-obs"
}
},
"timestamp": "2023-09-12T20:06:22.946732290Z",
"logName": "projects/vpn-aws-obs/logs/compute.googleapis.com%2Fvpc_flows",
"receiveTimestamp": "2023-09-12T20:06:22.946732290Z"
}
Add service account for bigquery
biqueryAdmin role
add serviceAccountTokenCreator role to admins
see https://cloud.google.com/bigquery/docs/use-service-accounts https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev
ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=76......0-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fapplicationdefaultauthcode.html&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login&state=a7fI6fW8EBiD3skC32FHcB6Qr5t46L&prompt=consent&access_type=offline&code_challenge=qN6I0UEBHl0SQ8dSQ7qbPO_y7pdxU54Zuant3PSf9xc&code_challenge_method=S256
Enter authorization code: 4/0Adeu5.........AS-6OWiDm4Nr36BJHlNhIiCFHVi1RALV0wS1XhcKdnkCcr7g
Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]
These credentials will be used by any library that requests Application Default Credentials (ADC).
ubuntu@ip-10-51-1-185:~$ cat /home/ubuntu/.config/gcloud/application_default_credentials.json
{
"delegates": [],
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/big....a@vpn-aws-obs.iam.gserviceaccount.com:generateAccessToken",
"source_credentials": {
"client_id": "76408.........6pt8ejuq83di341hur.apps.googleusercontent.com",
"client_secret": "d-FL.............HD0Ty",
"refresh_token": "1//0.....................AESNgF-L9IrXpW9FfxsR8BZmLIorBGxG5Bxf2HlTZW6Vpg_9muB8CSwLgy4U-pmR04Aje0ppUdUDg",
"type": "authorized_user"
},
"type": "impersonated_service_account"
ubuntu@ip-10-51-1-185:~$ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"kind": "bigquery#tableDataList",
"etag": "lj66QnFhOWfLoLt73uLkyw==",
"totalRows": "4331",
"pageToken": "BF6E6O4PRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAW5TQNYWWC53TFVXWE4YR5ADGWH7PAAAAAEQLOJXWY3DFOJRGYYLEMUNCINZRGMYTONRXGYWTEZRSGIWTIMRUGYWTQZLEMYWTINDGGM4WKNDEGQ4WGYISBNZG63DMMVZGE3DBMRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
"rows": [
{
"f": [
{
"v": "45.424712"
test access by temporarily deleting bigquery.admin role
Reauthenticate impersonation to check biqquery deny
ubuntu@ip-10-51-1-185:~$gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=76408....&state=OZUfD7okVYuyMyKGDuBCTEgDvNA3g4&prompt=consent&access_type=offline&code_challenge=ET3rQo9WuB86WSG7vwQkKXdwA0Tl2uce_B7FvBm59jg&code_challenge_method=S256
Enter authorization code: 4/0Adeu5BU1Gmjm...........
Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]
These credentials will be used by any library that requests Application Default Credentials (ADC).
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"error": {
"code": 403,
"message": "Access Denied: Table vpn-aws-obs:rollerblade.rollerblade: Permission bigquery.tables.getData denied on table vpn-aws-obs:rollerblade.rollerblade (or it may not exist).",
"errors": [
{
"message": "Access Denied: Table vpn-aws-obs:rollerblade.rollerblade: Permission bigquery.tables.getData denied on table vpn-aws-obs:rollerblade.rollerblade (or it may not exist).",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
Reinstate bigquery.admin
ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=764086.....in&state=VTg2YpRi4woOr5xsOPKCINRCQemoqU&prompt=consent&access_type=offline&code_challenge=9cDrYAw20PjQ087WE-vhkRTSTKTZIvxpQphne3AaPGk&code_challenge_method=S256
Enter authorization code: 4/0Ade...5DKUtVunUt6FiW8iPw
Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]
These credentials will be used by any library that requests Application Default Credentials (ADC).
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"kind": "bigquery#tableDataList",
"etag": "z+UPOJmTYBEcLen4sYqTAA==",
"totalRows": "4331",
"pageToken": "BFJDEQMPRIA...MRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
"rows": [
{
"f": [
{
"v": "45.424712"
},
{
"v": "-75.698209"
}
Going through the security controls distributed in the hub-env pkg around the peering between hub and spoke packages - ran into mention of the forwarding rule region issue for the gcloud workaround proposed (Nice idea btw) https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/305
Dave’s comment on mar 31st for the bug referenced in the meet is this one right?
https://github.com/GoogleCloudPlatform/magic-modules/pull/7480 Forgot I had a reference to it in the client PSC work. Your spoke-unclass-env package helps us out with porting over a better usable implementation of the IaaS fw use case - thank you
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/305
https://github.com/hashicorp/terraform-provider-google/issues/14049
See partner interconnect procedure (any BGP will do - vpn, partner interconnect, direct interconnect, peering)
Architecture
20230827:2300: pivot to PSC from PGA https://cloud.google.com/vpc/docs/about-accessing-vpc-hosted-services-endpoints
https://cloud.google.com/vpc/docs/configure-private-service-connect-apis
Document and simulate GCP + Customer procedure: Shadow https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/299 Follow https://cloud.google.com/vpc/docs/private-access-options
Architecture Discussion:
Asset Inventory - GCP side
Notes:
https://docs.google.com/presentation/d/13sjT2tJ4yLIYGRREE3wBrylB1OvcEMpKdquVuJB_nX4/edit?resourcekey=0-N3DruQaiutFvZ98HTT7-vQ#slide=id.g1154b3b950f_2_3458 slide 27 https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid https://cloud.google.com/vpc/docs/configure-private-service-connect-apis#on-premises
Customer environment consists of already created interconnect/VPN where there is a BGP route for the DNS proxy egress from onprem Use case is one where google APIs and googledomains.com queries into GCP both resolve and are kept private on the premium google network
https://cloud.google.com/vpc/docs/private-service-connect#:~:text=Similarly%2C%20a%20Private%20Service%20Connect,internal%20IP%20addresses%20for%20endpoints.
Reference: procedures
See https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/763