Closed fmichaelobrien closed 7 months ago
obriensystems
commented
1 year ago AWS (prem/ground) to GCP HA VPN up for private google access and DNS forwarding simulation in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494
obriensystems
commented
1 year ago Shadow
Procedure:
Create projects - GCP
export PROJECT_ID=vpn-aws-obs
gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default
gcloud config set project "${PROJECT_ID}"
export BILLING_ID=01D...0154
gcloud beta billing projects link ${PROJECT_ID} --billing-account ${BILLING_ID}Set iAM permissions
export USER_EMAIL=michael@obr...tware
export ORG_ID=$(gcloud projects get-ancestors $PROJECT_ID --format='get(id)' | tail -1)
gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$USER_EMAIL --role=roles/compute.securityAdmin --quiet
gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$USER_EMAIL --role=roles/compute.networkeAdmin --quiet
gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$USER_EMAIL --role=roles/compute.admin --quietbug: routing mode should be regional bug: enable compute API on the project first before creating a VPC https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-the-ha-vpn-gateway-and-cloud-router-on-google-cloud
gcloud services enable compute.googleapis.comcreate network
export NETWORK=vpn-aws-obs-vpc
export SUBNET_MODE=custom
export BGP_ROUTING_MODE=regional
gcloud compute networks create $NETWORK --subnet-mode $SUBNET_MODE --bgp-routing-mode $BGP_ROUTING_MODE
export SUBNET_NAME=$NETWORK-sn
export IP_ADDRESS_RANGE=10.101.0.0/24
export REGION=northamerica-northeast1
gcloud compute networks subnets create $SUBNET_NAME --network $NETWORK --region $REGION --range $IP_ADDRESS_RANGEcreate HA VPN Gateway
export HA_VPN_GATEWAY_NAME=$PROJECT_ID-vpn
gcloud compute vpn-gateways create $HA_VPN_GATEWAY_NAME --network $NETWORK --region $REGION
see
INTERFACE0: 34.124.12.157
INTERFACE1: 34.104.78.53
export ROUTER_NAME=$PROJECT_ID-router
export GOOGLE_ASN=65534
gcloud compute routers create $ROUTER_NAME --region $REGION --network $NETWORK --asn $GOOGLE_ASN --advertisement-mode custom --set-advertisement-groups all_subnetsCreate 2 customer gateways
export INTERFACE_0_IP_ADDRESS=34.124.12.157
export INTERFACE_1_IP_ADDRESS=34.104.78.53
export GOOGLE_ASN=65534
aws ec2 create-customer-gateway --type ipsec.1 --public-ip $INTERFACE_0_IP_ADDRESS --bgp-asn $GOOGLE_ASN
{
"CustomerGateway": {
"BgpAsn": "65534",
"CustomerGatewayId": "cgw-0219d0daa1e0f6f02",
"IpAddress": "34.124.12.157",
"State": "available",
"Type": "ipsec.1",
}
}
export CUSTOMER_GATEWAY_1=cgw-0219d0daa1e0f6f02
aws ec2 create-customer-gateway --type ipsec.1 --public-ip $INTERFACE_1_IP_ADDRESS --bgp-asn $GOOGLE_ASN
{
"CustomerGateway": {
"BgpAsn": "65534",
"CustomerGatewayId": "cgw-066a98dc85373f724",
"IpAddress": "34.104.78.53",
"State": "available",
"Type": "ipsec.1"
}
}
export CUSTOMER_GATEWAY_2=cgw-066a98dc85373f724Create VPC
Bug: https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create_gateways_and_vpn_connections_on_aws missing VPC creation on AWS including private subnet , ingress firewall rules https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli
Create AWS VPC https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli
export AWS_IP_ADDRESS_RANGE=10.51.0.0/24
export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/26
aws ec2 create-vpc --cidr-block $AWS_IP_ADDRESS_RANGE --query Vpc.VpcId --output text
export VPC_ID=vpc-0c95bd8820efebea4BUG: VPC already has a route table with a default route after vpc creation
BUG: switch subnet to us-east-1a (to match VPC) - or VPC missing region
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id vpc-1a2b3c4d5e6f1a2b3 --cidr-block 10.0.1.0/20 --availability-zone us-east-2a --query Subnet.SubnetId --output text
An error occurred (InvalidParameterValue) when calling the CreateSubnet operation: Value (us-east-2a) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1e, us-east-1f.BUG: VPC must be /16 not /24 - in order to use subnets of /20
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id vpc-1a2b3c4d5e6f1a2b3 --cidr-block 10.0.1.0/20 --availability-zone us-east-1a --query Subnet.SubnetId --output text
An error occurred (InvalidVpcID.NotFound) when calling the CreateSubnet operation: The vpc ID 'vpc-1a2b3c4d5e6f1a2b3' does not exist
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
An error occurred (InvalidSubnet.Range) when calling the CreateSubnet operation: The CIDR '10.51.1.0/20' is invalid.
[cloudshell-user@ip-10-6-18-196 ~]$ export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/26
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
An error occurred (InvalidSubnet.Range) when calling the CreateSubnet operation: The CIDR '10.51.1.0/26' is invalid.create subnet
aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
export SUBNET_ID=Recreate a new VPC
export AWS_IP_ADDRESS_RANGE=10.51.0.0/16
export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/24
aws ec2 create-vpc --cidr-block $AWS_IP_ADDRESS_RANGE --query Vpc.VpcId --output text
export VPC_ID=vpc-0f504a74c4317f091Add subnet
aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
export SUBNET_ID=subnet-0cdb542888b890f9fNo AWS IGW or NGW - but put a NGW for the private subnet - step 6
# allocate EIP
aws ec2 allocate-address --domain vpc --query AllocationId --output text
export ALLOCATION_ID=eipalloc-05613c22cf3f36803
# create a NAT GW
aws ec2 create-nat-gateway --subnet-id $SUBNET_ID --allocation-id $ALLOCATION_ID
{
"ClientToken": "6d1b56aa-2d25-4cae-bdd4-a94a593d37a6",
"NatGateway": {
"CreateTime": "2023-08-27T03:35:47+00:00",
"NatGatewayAddresses": [
{
"AllocationId": "eipalloc-05613c22cf3f36803",
"IsPrimary": true,
"Status": "associating"
}
],
"NatGatewayId": "nat-0c4da22f458fe7cbb",
"State": "pending",
"SubnetId": "subnet-0cdb542888b890f9f",
"VpcId": "vpc-0f504a74c4317f091",
"ConnectivityType": "public"
}
}
export NAT_ID=nat-0c4da22f458fe7cbb
# create route table
aws ec2 create-route-table --vpc-id $VPC_ID --query RouteTable.RouteTableId --output text
export ROUTETABLE_ID=rtb-0ff23a1fb426e1301
# create route to Nat GW
aws ec2 create-route --route-table-id $ROUTETABLE_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $NAT_ID
# associate route table with private subnet
aws ec2 associate-route-table --route-table-id $ROUTETABLE_ID --subnet-id $SUBNET_ID
{
"AssociationId": "rtbassoc-0a155c6354b7008c4",
"AssociationState": {
"State": "associated"
}
}Create VM
# create VM
# test external internet connectivity
finished with https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli
Attach VPG to VPC return to step 2 of https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create_gateways_and_vpn_connections_on_aws
export VPC_ID=vpc-0f504a74c4317f091
export AWS_SIDE_ASN=65501
aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn $AWS_SIDE_ASN
{
"VpnGateway": {
"State": "available",
"Type": "ipsec.1",
"VpcAttachments": [],
"VpnGatewayId": "vgw-0bd4928d84cd855cf",
"AmazonSideAsn": 65501
}
}
export VPN_GATEWAY_ID=vgw-0bd4928d84cd855cf
aws ec2 attach-vpn-gateway --vpn-gateway-id $VPN_GATEWAY_ID --vpc-id $VPC_ID
{
"VpcAttachment": {
"State": "attaching",
"VpcId": "vpc-0f504a74c4317f091"
}
}follow for options https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html#vpn-create-vpn-connection https://docs.aws.amazon.com/cli/latest/reference/ec2/create-vpn-connection.html https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNTunnels.html
generate preshared keys https://cloud.google.com/network-connectivity/docs/vpn/how-to/generating-pre-shared-key or use
openssl rand -base64 24export CUSTOMER_GATEWAY_1=cgw-0219d0daa1e0f6f02
export CUSTOMER_GATEWAY_2=cgw-066a98dc85373f724
export AWS_T1_IP=169.254.51.0/30
export AWS_T2_IP=169.254.52.0/30
export SHARED_SECRET_1=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G
export SHARED_SECRET_2=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0
#2030827: edit - was duplicated 52 and 53 but didn't use these anyway below - so no T2/T3 overalp
export AWS_T3_IP=169.254.53.0/30
export AWS_T4_IP=169.254.54.0/30
export SHARED_SECRET_3=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI
export SHARED_SECRET_4=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=$AWS_T1_IP,PreSharedKey=$SHARED_SECRET_1},{TunnelInsideCidr=$AWS_T2_IP,PreSharedKey=$SHARED_SECRET_2}]'
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=$AWS_T3_IP,PreSharedKey=$SHARED_SECRET_3},{TunnelInsideCidr=$AWS_T4_IP,PreSharedKey=$SHARED_SECRET_4}]'BUG:
BUG: shared key site generates / and + (invalid chars) - convert to . https://cloud.google.com/network-connectivity/docs/vpn/how-to/generating-pre-shared-key
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk+MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0+chcFdgh1UN0}]'
An error occurred (InvalidParameterValue) when calling the CreateVpnConnection operation: Value for parameter PreSharedKey is invalid.
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0}]'
{
"VpnConnection": {
"CustomerGatewayConfiguration": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<vpn_connection id=\"vpn-011bfe843f9f75e55\">\n <customer_gateway_id>cgw-0219d0daa1e0f6f02</customer_gateway_id>\n <vpn_gateway_id>vgw-0bd4928d84cd855cf</vpn_gateway_id>\n <vpn_connection_type>ipsec.1</vpn_connection_type>\n <ipsec_tunnel>\n <customer_gateway>\n <tunnel_outside_address>\n <ip_address>34.124.12.157</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.51.2</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65534</asn>\n <hold_time>30</hold_time>\n </bgp>\n </customer_gateway>\n <vpn_gateway>\n <tunnel_outside_address>\n <ip_address>34.232.42.137</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.51.1</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65501</asn>\n <hold_time>30</hold_time>\n </bgp>\n </vpn_gateway>\n <ike>\n <authentication_protocol>sha1</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>28800</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>main</mode>\n <pre_shared_key>4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G</pre_shared_key>\n </ike>\n <ipsec>\n <protocol>esp</protocol>\n <authentication_protocol>hmac-sha1-96</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>3600</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>tunnel</mode>\n <clear_df_bit>true</clear_df_bit>\n <fragmentation_before_encryption>true</fragmentation_before_encryption>\n <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n <dead_peer_detection>\n <interval>10</interval>\n <retries>3</retries>\n </dead_peer_detection>\n </ipsec>\n </ipsec_tunnel>\n <ipsec_tunnel>\n <customer_gateway>\n <tunnel_outside_address>\n <ip_address>34.124.12.157</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.52.2</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65534</asn>\n <hold_time>30</hold_time>\n </bgp>\n </customer_gateway>\n <vpn_gateway>\n <tunnel_outside_address>\n <ip_address>54.159.162.176</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.52.1</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65501</asn>\n <hold_time>30</hold_time>\n </bgp>\n </vpn_gateway>\n <ike>\n <authentication_protocol>sha1</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>28800</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>main</mode>\n <pre_shared_key>SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0</pre_shared_key>\n </ike>\n <ipsec>\n <protocol>esp</protocol>\n <authentication_protocol>hmac-sha1-96</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>3600</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>tunnel</mode>\n <clear_df_bit>true</clear_df_bit>\n <fragmentation_before_encryption>true</fragmentation_before_encryption>\n <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n <dead_peer_detection>\n <interval>10</interval>\n <retries>3</retries>\n </dead_peer_detection>\n </ipsec>\n </ipsec_tunnel>\n</vpn_connection>",
"CustomerGatewayId": "cgw-0219d0daa1e0f6f02",
"Category": "VPN",
"State": "pending",
"VpnConnectionId": "vpn-011bfe843f9f75e55",
"VpnGatewayId": "vgw-0bd4928d84cd855cf",
"GatewayAssociationState": "associated",
"Options": {
"EnableAcceleration": false,
"StaticRoutesOnly": false,
"LocalIpv4NetworkCidr": "0.0.0.0/0",
"RemoteIpv4NetworkCidr": "0.0.0.0/0",
"OutsideIpAddressType": "PublicIpv4",
"TunnelInsideIpVersion": "ipv4",
"TunnelOptions": [
{
"OutsideIpAddress": "34.232.42.137",
"TunnelInsideCidr": "169.254.51.0/30",
"PreSharedKey": "4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G",
"LogOptions": {
"CloudWatchLogOptions": {
"LogEnabled": false
}
}
},
{
"OutsideIpAddress": "54.159.162.176",
"TunnelInsideCidr": "169.254.52.0/30",
"PreSharedKey": "SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0",
"LogOptions": {
"CloudWatchLogOptions": {
"LogEnabled": false
}
}
}
]
},
"Routes": [],
"Tags": []
}
}aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0}]'
"VpnConnectionId": "vpn-011bfe843f9f75e55",
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.53.0/30,PreSharedKey=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI},{TunnelInsideCidr=169.254.54.0/30,PreSharedKey=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf}]'
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.53.0/30,PreSharedKey=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI},{TunnelInsideCidr=169.254.54.0/30,PreSharedKey=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf}]'
{
"VpnConnection": {
"CustomerGatewayConfiguration": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<vpn_connection id=\"vpn-09e6693b14fa31452\">\n <customer_gateway_id>cgw-066a98dc85373f724</customer_gateway_id>\n <vpn_gateway_id>vgw-0bd4928d84cd855cf</vpn_gateway_id>\n <vpn_connection_type>ipsec.1</vpn_connection_type>\n <ipsec_tunnel>\n <customer_gateway>\n <tunnel_outside_address>\n <ip_address>34.104.78.53</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.53.2</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65534</asn>\n <hold_time>30</hold_time>\n </bgp>\n </customer_gateway>\n <vpn_gateway>\n <tunnel_outside_address>\n <ip_address>44.215.227.192</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.53.1</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65501</asn>\n <hold_time>30</hold_time>\n </bgp>\n </vpn_gateway>\n <ike>\n <authentication_protocol>sha1</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>28800</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>main</mode>\n <pre_shared_key>ihwsnhelOYX8GAYQvxyszRDnKUigL7wI</pre_shared_key>\n </ike>\n <ipsec>\n <protocol>esp</protocol>\n <authentication_protocol>hmac-sha1-96</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>3600</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>tunnel</mode>\n <clear_df_bit>true</clear_df_bit>\n <fragmentation_before_encryption>true</fragmentation_before_encryption>\n <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n <dead_peer_detection>\n <interval>10</interval>\n <retries>3</retries>\n </dead_peer_detection>\n </ipsec>\n </ipsec_tunnel>\n <ipsec_tunnel>\n <customer_gateway>\n <tunnel_outside_address>\n <ip_address>34.104.78.53</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.54.2</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65534</asn>\n <hold_time>30</hold_time>\n </bgp>\n </customer_gateway>\n <vpn_gateway>\n <tunnel_outside_address>\n <ip_address>52.2.98.253</ip_address>\n </tunnel_outside_address>\n <tunnel_inside_address>\n <ip_address>169.254.54.1</ip_address>\n <network_mask>255.255.255.252</network_mask>\n <network_cidr>30</network_cidr>\n </tunnel_inside_address>\n <bgp>\n <asn>65501</asn>\n <hold_time>30</hold_time>\n </bgp>\n </vpn_gateway>\n <ike>\n <authentication_protocol>sha1</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>28800</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>main</mode>\n <pre_shared_key>7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf</pre_shared_key>\n </ike>\n <ipsec>\n <protocol>esp</protocol>\n <authentication_protocol>hmac-sha1-96</authentication_protocol>\n <encryption_protocol>aes-128-cbc</encryption_protocol>\n <lifetime>3600</lifetime>\n <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n <mode>tunnel</mode>\n <clear_df_bit>true</clear_df_bit>\n <fragmentation_before_encryption>true</fragmentation_before_encryption>\n <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n <dead_peer_detection>\n <interval>10</interval>\n <retries>3</retries>\n </dead_peer_detection>\n </ipsec>\n </ipsec_tunnel>\n</vpn_connection>",
"CustomerGatewayId": "cgw-066a98dc85373f724",
"Category": "VPN",
"State": "pending",
"VpnConnectionId": "vpn-09e6693b14fa31452",
"VpnGatewayId": "vgw-0bd4928d84cd855cf",
"GatewayAssociationState": "associated",
"Options": {
"EnableAcceleration": false,
"StaticRoutesOnly": false,
"LocalIpv4NetworkCidr": "0.0.0.0/0",
"RemoteIpv4NetworkCidr": "0.0.0.0/0",
"OutsideIpAddressType": "PublicIpv4",
"TunnelInsideIpVersion": "ipv4",
"TunnelOptions": [
{
"OutsideIpAddress": "44.215.227.192",
"TunnelInsideCidr": "169.254.53.0/30",
"PreSharedKey": "ihwsnhelOYX8GAYQvxyszRDnKUigL7wI",
"LogOptions": {
"CloudWatchLogOptions": {
"LogEnabled": false
}
}
},
{
"OutsideIpAddress": "52.2.98.253",
"TunnelInsideCidr": "169.254.54.0/30",
"PreSharedKey": "7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf",
"LogOptions": {
"CloudWatchLogOptions": {
"LogEnabled": false
}
}
}
]
},
"Routes": [],
"Tags": []
}
}
VPNs take a couple min to transition from pending
step GCP 2 - VPN Tunnels https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-vpn-tunnels-and-cloud-router-interfaces-on-google-cloud
export AWS_GW_IP_1=34.232.42.137
export AWS_GW_IP_2=54.159.162.176
export AWS_GW_IP_3=44.215.227.192
export AWS_GW_IP_4=52.2.98.253
export PEER_GATEWAY_NAME=aws-peer-vpn-gw
gcloud compute external-vpn-gateways create $PEER_GATEWAY_NAME --interfaces 0=$AWS_GW_IP_1,1=$AWS_GW_IP_2,2=$AWS_GW_IP_3,3=$AWS_GW_IP_4
Creating external VPN gateway...done.
NAME: aws-peer-vpn-gw
REDUNDANCY_TYPE: FOUR_IPS_REDUNDANCY4 tunnels
use ike-version=2 https://cloud.google.com/sdk/gcloud/reference/compute/vpn-tunnels/create
export REGION=northamerica-northeast1
export SHARED_SECRET_1=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G
export SHARED_SECRET_2=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0
export SHARED_SECRET_3=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI
export SHARED_SECRET_4=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf
export PROJECT_ID=vpn-aws-obs
export ROUTER_NAME=$PROJECT_ID-router
export HA_VPN_GATEWAY_NAME=$PROJECT_ID-vpn
export IKE_VERSION=2
gcloud compute vpn-tunnels create tunnel-1 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 0 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_1 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 0
Creating VPN tunnel...done.
NAME: tunnel-1
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 0
PEER_ADDRESS: 34.232.42.137
20230826 3 moregcp side
aws side - IP_SEC is up only so far until we setup BGP
20230827 Add 3 more tunnel https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-vpn-tunnels-and-cloud-router-interfaces-on-google-cloud
gcloud compute vpn-tunnels create tunnel-2 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 1 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_2 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 0
Creating VPN tunnel...done.
NAME: tunnel-2
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 0
PEER_ADDRESS: 54.159.162.176
gcloud compute vpn-tunnels create tunnel-3 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 2 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_3 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 1
Creating VPN tunnel...done.
NAME: tunnel-3
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 1
PEER_ADDRESS: 44.215.227.192
gcloud compute vpn-tunnels create tunnel-4 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 3 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_4 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 1
Creating VPN tunnel...done.
NAME: tunnel-4
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 1
PEER_ADDRESS: 52.2.98.253GCP side
AWS side
takes about 4 min for all 4
4 router interfaces
Get IPs from AWS VPN config (generic IKE2) - the customer gateway address
Inside IP Addresses
#export AWS_T1_IP=169.254.51.0/30
#export AWS_T2_IP=169.254.52.0/30
#export AWS_T3_IP=169.254.53.0/30
#export AWS_T4_IP=169.254.54.0/30
export PROJECT_ID=vpn-aws-obs
export ROUTER_NAME=$PROJECT_ID-router
export REGION=northamerica-northeast1
# take off the /30 - convert to non-CIDR
export GOOGLE_BGP_IP_TUNNEL_1=169.254.51.2
export GOOGLE_BGP_IP_TUNNEL_2=169.254.52.2
export GOOGLE_BGP_IP_TUNNEL_3=169.254.53.2
export GOOGLE_BGP_IP_TUNNEL_4=169.254.54.2
# inside addresses on gcp side match inside from aws above
gcloud compute routers add-interface $ROUTER_NAME --interface-name int-1 --vpn-tunnel tunnel-1 --ip-address $GOOGLE_BGP_IP_TUNNEL_1 --mask-length 30 --region $REGION
Updated [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/routers/vpn-aws-obs-router].
gcloud compute routers add-interface $ROUTER_NAME --interface-name int-2 --vpn-tunnel tunnel-2 --ip-address $GOOGLE_BGP_IP_TUNNEL_2 --mask-length 30 --region $REGION
gcloud compute routers add-interface $ROUTER_NAME --interface-name int-3 --vpn-tunnel tunnel-3 --ip-address $GOOGLE_BGP_IP_TUNNEL_3 --mask-length 30 --region $REGION
gcloud compute routers add-interface $ROUTER_NAME --interface-name int-4 --vpn-tunnel tunnel-4 --ip-address $GOOGLE_BGP_IP_TUNNEL_4 --mask-length 30 --region $REGION
mistake found - t2 and t3 CIDRS overlap - but I used direct strings and not the VARs above - so we are good
#export AWS_T2_IP=169.254.52.0/30
#export AWS_T3_IP=169.254.52.0/30
4 add BGP peers
--peer-ip-address: invalid ipv4 value: '169.254.51.0/30' The link-local address of the peer router. Must be a link-local IPv4 address belonging to the range 169.254.0.0/16. not in https://rajathithanrajasekar.medium.com/create-ha-vpn-in-google-cloud-using-gcloud-commands-66d691a8d8be
bug: CIDR not recognized - need IP like 169.254.51.1 from the vpn config - the BGP neighbor IP
--peer-ip-address AWS_T3_IP
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $AWS_T1_IP --region $REGION
ERROR: (gcloud.compute.routers.add-bgp-peer) argument --peer-ip-address: invalid ipv4 value: '169.254.51.0/30'
Usage: gcloud compute routers add-bgp-peer NAME --interface=INTERFACE --peer-asn=PEER_ASN --peer-name=PEER_NAME [optional flags]
is it the VPG
Inside IP Addresses
- Customer Gateway : 169.254.54.2/30
- Virtual Private Gateway : 169.254.54.1/30use as example
in another working VPN of my we use 169.254.0.2 - this is the GOOGLE_BGP_IP_TUNNEL_1 address
working with BGP neighbor IP
export AWS_SIDE_ASN=65501
export PEER_ASN=$AWS_SIDE_ASN
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $AWS_T1_IP --region $REGION
replace
AWS_T1_IP with GOOGLE_BGP_IP_TUNNEL_1
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $GOOGLE_BGP_IP_TUNNEL_1 --region $REGION
ERROR: (gcloud.compute.routers.add-bgp-peer) HTTPError 400: Invalid value for field 'resource.bgpPeers[0].peerIpAddress': '169.254.51.2'. In BGP Peer "aws-conn1-tunn1", peer IP address must be different from IP address 169.254.51.2.
use the VPG neighbor IP
Inside IP Addresses
- Customer Gateway : 169.254.54.2/30
- Virtual Private Gateway : 169.254.54.1/30
-
BGP Configuration Options:
- Customer Gateway ASN : 65534
- Virtual Private Gateway ASN : 65501
- Neighbor IP Address : 169.254.54.1
- Neighbor Hold Time : 30
working
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address 169.254.51.1 --region $REGION
Creating peer [aws-conn1-tunn1] in router [vpn-aws-obs-router]...done.
gcp side
aws side
Add 3 remaining BGP sessions
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn2 --peer-asn $PEER_ASN --interface int-2 --peer-ip-address 169.254.52.1 --region $REGION
NOTICE 2023-08-27T14:06:52.119687391Z BGP Event: Successfully added configuration for peers: 169.254.51.1, 169.254.52.1
INFO 2023-08-27T14:06:52.128174992Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
NOTICE 2023-08-27T14:06:53.069505725Z BGP Event: BGP peering with 169.254.52.1 came up 0 seconds ago, hold time is 60 seconds, graceful restart timer is 60 seconds
INFO 2023-08-27T14:06:53.203166433Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
NOTICE 2023-08-27T14:06:54.232914Z [protoPayload.serviceName: compute.googleapis.com] [protoPayload.methodName: v1.compute.routers.patch] [protoPayload.resourceName: projects/vpn-aws-obs/regions/northamerica-northeast1/routers/vpn-aws-obs-router] [protoPayload.authenticationInfo.principalEmail: michael@obrien.software] audit_log, method: "v1.compute.routers.patch", principal_email: "michael@obrien.software"
INFO 2023-08-27T14:07:50.515738103Z Prefix 10.51.0.0/16 NextHops 169.254.51.1,169.254.52.1 priority 200 received by cloud router
INFO 2023-08-27T14:07:54.116909103Z Prefix 10.51.0.0/16 NextHops 169.254.52.1 priority 100 received by cloud router
1007-1011 - 4 min for AWS side
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn2-tunn1 --peer-asn $PEER_ASN --interface int-3 --peer-ip-address 169.254.53.1 --region $REGION
NOTICE 2023-08-27T14:16:48.364249433Z BGP Event: BGP peering with 169.254.54.1 came up 0 seconds ago, hold time is 60 seconds, graceful restart timer is 60 seconds
INFO 2023-08-27T14:16:48.390937235Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
INFO 2023-08-27T14:16:48.506182843Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
last of 4 bgp tunnels
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn2-tunn2 --peer-asn $PEER_ASN --interface int-4 --peer-ip-address 169.254.54.1 --region $REGIONAll 4 tunnels up on both sides
GCP
AWS
check routes
use a bastion https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/
Verify BGP dynamic routes on both GCP and AWS sides
from AWS 51 subnet (101 is gcp)
INFO 2023-08-27T14:17:48.626770110Z Prefix 10.51.0.0/16 NextHops 169.254.53.1,169.254.54.1 priority 200 received by cloud router
INFO 2023-08-27T14:17:49.427057114Z Prefix 10.51.0.0/16 NextHops 169.254.54.1 priority 100 received by cloud router
gcloud compute routers get-status $ROUTER_NAME --region $REGION --format='flattened(result.bgpPeerStatus[].name, result.bgpPeerStatus[].ipAddress, result.bgpPeerStatus[].peerIpAddress)'
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers get-status $ROUTER_NAME --region $REGION --format='flattened(result.bgpPeerStatus[].name, result.bgpPeerStatus[].ipAddress, result.bgpPeerStatus[].peerIpAddress)'
result.bgpPeerStatus[0].ipAddress: 169.254.51.2
result.bgpPeerStatus[0].name: aws-conn1-tunn1
result.bgpPeerStatus[0].peerIpAddress: 169.254.51.1
result.bgpPeerStatus[1].ipAddress: 169.254.52.2
result.bgpPeerStatus[1].name: aws-conn1-tunn2
result.bgpPeerStatus[1].peerIpAddress: 169.254.52.1
result.bgpPeerStatus[2].ipAddress: 169.254.53.2
result.bgpPeerStatus[2].name: aws-conn2-tunn1
result.bgpPeerStatus[2].peerIpAddress: 169.254.53.1
result.bgpPeerStatus[3].ipAddress: 169.254.54.2
result.bgpPeerStatus[3].name: aws-conn2-tunn2
result.bgpPeerStatus[3].peerIpAddress: 169.254.54.1
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute vpn-tunnels list
NAME: tunnel-1
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 34.232.42.137
NAME: tunnel-2
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 54.159.162.176
NAME: tunnel-3
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 44.215.227.192
NAME: tunnel-4
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 52.2.98.253
gcloud compute vpn-tunnels describe tunnel-1 --region $REGION --format='flattened(status,detailedStatus)'
detailed_status: Tunnel is up and running.
status: ESTABLISHED
list dynamic routes
gcloud compute routers get-status $ROUTER_NAME --region $REGION --format="flattened(result.bestRoutes)"
result.bestRoutes[0].asPaths[0].asLists[0]: 65501
result.bestRoutes[0].asPaths[0].pathSegmentType: AS_SEQUENCE
result.bestRoutes[0].creationTimestamp: 2023-08-27T07:07:54.116-07:00
result.bestRoutes[0].destRange: 10.51.0.0/16
result.bestRoutes[0].kind: compute#route
result.bestRoutes[0].network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
result.bestRoutes[0].nextHopIp: 169.254.52.1
result.bestRoutes[0].nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
result.bestRoutes[0].priority: 100
result.bestRoutes[0].routeType: BGP
result.bestRoutes[1].asPaths[0].asLists[0]: 65501
result.bestRoutes[1].asPaths[0].pathSegmentType: AS_SEQUENCE
result.bestRoutes[1].creationTimestamp: 2023-08-27T07:17:49.427-07:00
result.bestRoutes[1].destRange: 10.51.0.0/16
result.bestRoutes[1].kind: compute#route
result.bestRoutes[1].network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
result.bestRoutes[1].nextHopIp: 169.254.54.1
result.bestRoutes[1].nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
result.bestRoutes[1].priority: 100
result.bestRoutes[1].routeType: BGP
Add public subnet and bastion for VM access on AWS
create internet gateway
attach IGW to VPC
aws ec2 attach-internet-gateway --vpc-id "vpc-0f504a74c4317f091" --internet-gateway-id "igw-0dec90e7440b2163c" --region us-east-1remove NATGW from private subnet - not needed - move to public subnet - remove black hole
create new NATGW with existing EIP
add route to IGW from public subnet
Create bastion
Been a while trying also https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html
IGW and NAT dont' have route table entries - adding 0.0.0.0/0 to IGW
fixed 1 of 2
regular ssh working now
michaelobrien@mbp7 pubsec-declarative-toolkit % ssh ubuntu@jump.obrienlabs.cloud
ssh: connect to host jump.obrienlabs.cloud port 22: Operation timed out
michaelobrien@mbp7 pubsec-declarative-toolkit % ssh ubuntu@jump.obrienlabs.cloud
The authenticity of host 'jump.obrienlabs.cloud (35.153.35.64)' can't be established.
ED25519 key fingerprint is SHA256:gXubQVYDc2kGW4PV1yJIn7x8DVWouqr1HgHKnfSAg04.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'jump.obrienlabs.cloud' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Sun Aug 27 15:20:44 UTC 2023
System load: 0.0 Processes: 95
Usage of /: 20.6% of 7.57GB Users logged in: 0
Memory usage: 23% IPv4 address for eth0: 10.51.2.201
Swap usage: 0%
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@ip-10-51-2-201:~$
ubuntu@ip-10-51-2-201:~$ dig www.google.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3028
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.google.com. IN A
;; ANSWER SECTION:
www.google.com. 139 IN A 172.253.122.104
www.google.com. 139 IN A 172.253.122.105
www.google.com. 139 IN A 172.253.122.106
www.google.com. 139 IN A 172.253.122.147
www.google.com. 139 IN A 172.253.122.99
www.google.com. 139 IN A 172.253.122.103
;; Query time: 4 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sun Aug 27 15:21:33 UTC 2023
;; MSG SIZE rcvd: 139
ubuntu@ip-10-51-2-201:~$ wget http://www.google.com
--2023-08-27 15:21:44-- http://www.google.com/
Resolving www.google.com (www.google.com)... 172.253.122.105, 172.253.122.106, 172.253.122.99, ...
Connecting to www.google.com (www.google.com)|172.253.122.105|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’
index.html [ <=> ] 18.17K --.-KB/s in 0.001s
2023-08-27 15:21:44 (34.8 MB/s) - ‘index.html’ saved [18611]
I see the private VM as expected on the private 1 subnet
ubuntu@ip-10-51-2-201:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=64 time=0.868 ms
on a mac
michaelobrien@mbp7 pubsec-declarative-toolkit % ssh -A ubuntu@jump.obrienlabs.cloud
on the bastion
ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.164
On the private VM
ubuntu@ip-10-51-1-164:~$
ubuntu@ip-10-51-1-164:~$ dig private.googleapis.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> private.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60142
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;private.googleapis.com. IN A
;; ANSWER SECTION:
private.googleapis.com. 300 IN A 199.36.153.8
private.googleapis.com. 300 IN A 199.36.153.9
private.googleapis.com. 300 IN A 199.36.153.10
private.googleapis.com. 300 IN A 199.36.153.11
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sun Aug 27 20:27:09 UTC 2023
;; MSG SIZE rcvd: 115
Fix nat on private subnet on aws - prior to viewing bgp dynamic routes from gcp
ping to google working on private subnet now
ubuntu@ip-10-51-1-164:~$ ping www.google.com
PING www.google.com (142.251.16.104) 56(84) bytes of data.
64 bytes from bl-in-f104.1e100.net (142.251.16.104): icmp_seq=1 ttl=51 time=3.15 ms
Add PGA route in AWS on private subnet not by pointing to the IGW but my advertising a custom route to 199.36.153.8/30 that is picked up by the VPN on the AWS side's BGP router
don't do below
Check Routes
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers get-status $ROUTER_NAME --region $REGION
kind: compute#routerStatusResponse
result:
bestRoutes:
- asPaths:
- asLists:
- 65501
pathSegmentType: AS_SEQUENCE
creationTimestamp: '2023-08-27T07:07:54.116-07:00'
destRange: 10.51.0.0/16
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.52.1
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
priority: 100
routeType: BGP
- asPaths:
- asLists:
- 65501
pathSegmentType: AS_SEQUENCE
creationTimestamp: '2023-08-27T07:17:49.427-07:00'
destRange: 10.51.0.0/16
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.1
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
bestRoutesForRouter:
- asPaths:
- asLists:
- 65501
pathSegmentType: AS_SEQUENCE
creationTimestamp: '2023-08-27T07:07:54.116-07:00'
destRange: 10.51.0.0/16
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.52.1
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
priority: 100
routeStatus: ACTIVE
routeType: BGP
- asPaths:
- asLists:
- 65501
pathSegmentType: AS_SEQUENCE
creationTimestamp: '2023-08-27T07:17:49.427-07:00'
destRange: 10.51.0.0/16
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.1
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeStatus: ACTIVE
routeType: BGP
bgpPeerStatus:
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.51.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.51.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
priority: 100
routeType: BGP
enableIpv6: false
ipAddress: 169.254.51.2
linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
md5AuthEnabled: false
name: aws-conn1-tunn1
numLearnedRoutes: 1
peerIpAddress: 169.254.51.1
state: Established
status: UP
uptime: 6 hours, 50 minutes, 8 seconds
uptimeSeconds: '24608'
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.52.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.52.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
priority: 100
routeType: BGP
enableIpv6: false
ipAddress: 169.254.52.2
linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
md5AuthEnabled: false
name: aws-conn1-tunn2
numLearnedRoutes: 1
peerIpAddress: 169.254.52.1
state: Established
status: UP
uptime: 6 hours, 44 minutes, 5 seconds
uptimeSeconds: '24245'
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.53.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.53.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
priority: 100
routeType: BGP
enableIpv6: false
ipAddress: 169.254.53.2
linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
md5AuthEnabled: false
name: aws-conn2-tunn1
numLearnedRoutes: 1
peerIpAddress: 169.254.53.1
state: Established
status: UP
uptime: 6 hours, 38 minutes, 10 seconds
uptimeSeconds: '23890'
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
enableIpv6: false
ipAddress: 169.254.54.2
linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
md5AuthEnabled: false
name: aws-conn2-tunn2
numLearnedRoutes: 1
peerIpAddress: 169.254.54.1
state: Established
status: UP
uptime: 6 hours, 34 minutes, 9 seconds
uptimeSeconds: '23649'
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
michael@cloudshell:~ (vpn-aws-obs)$ We can see the private google access CIDR in the first advertised route along with the private subnet in GCP VPC
- advertisedRoutes:
- destRange: 199.36.153.8/30
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGP
- destRange: 10.101.0.0/24
kind: compute#route
network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
nextHopIp: 169.254.54.2
nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
priority: 100
routeType: BGPWe should not need to - as it does not affect the on-prem network - but turn on PGA for the GCP private subnet
gcloud compute instances create vpn-jump \
--project=vpn-aws-obs \
--zone=northamerica-northeast1-a \
--machine-type=e2-small \
--network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=vpn-aws-obs-vpc-sn \
--maintenance-policy=MIGRATE \
--provisioning-model=STANDARD \
--service-account=1027024291560-compute@developer.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--tags=http-server,https-server \
--create-disk=auto-delete=yes,boot=yes,device-name=vpn-jump,image=projects/debian-cloud/global/images/debian-11-bullseye-v20230814,mode=rw,size=10,type=projects/vpn-aws-obs/zones/us-central1-a/diskTypes/pd-balanced \
--no-shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring \
--labels=goog-ec-src=vm_add-gcloud \
--reservation-affinity=any
on ssh
Please consider adding a firewall rule to allow ingress from the Cloud IAP for TCP forwarding netblock (from range 35.235.240.0/20) to the SSH port of your machine to start using Cloud IAP for TCP forwarding for better performance.
delete the default VPC
open the firewall
gcloud compute --project=vpn-aws-obs firewall-rules create allopen-ingress --direction=INGRESS --priority=1000 --network=vpn-aws-obs-vpc --action=ALLOW --rules=all --source-ranges=0.0.0.0/0
ssh to gcp vm
michael@vpn-jump:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
^C
--- 10.51.1.164 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2031ms
create a private instance - post PGA without public IP
gcloud compute instances create vpn-jump-private2 \
--project=vpn-aws-obs \
--zone=northamerica-northeast1-a \
--machine-type=e2-small \
--network-interface=stack-type=IPV4_ONLY,subnet=vpn-aws-obs-vpc-sn,no-address \
--maintenance-policy=MIGRATE \
--provisioning-model=STANDARD \
--service-account=1027024291560-compute@developer.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--tags=http-server,https-server \
--create-disk=auto-delete=yes,boot=yes,device-name=vpn-jump-private2,image=projects/debian-cloud/global/images/debian-11-bullseye-v20230814,mode=rw,size=10,type=projects/vpn-aws-obs/zones/us-central1-a/diskTypes/pd-balanced \
--no-shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring \
--labels=goog-ec-src=vm_add-gcloud \
--reservation-affinity=anyhowever we see AWS routes in GCP - but only the VPC - which may be subnet/vpc routing on the AWS side
vpn-aws-obs-router-aws-conn1-tunn2-1123490692 | Dynamic | IPv4 | 10.51.0.0/16 | 100 | None | VPN tunnel tunnel-2 |
-- | -- | -- | -- | -- | -- | -- | --
vpn-aws-obs-router-aws-conn2-tunn2-1369310470 | Dynamic | IPv4 | 10.51.0.0/16 | 100 | None | VPN tunnel tunnel-4
vpn-aws-obs-router-aws-conn1-tunn2-1123490692 Dynamic IPv4 10.51.0.0/16 100 None VPN tunnel [tunnel-2](https://console.cloud.google.com/hybrid/vpn/tunnels/details/northamerica-northeast1/tunnel-2?project=vpn-aws-obs)
vpn-aws-obs-router-aws-conn2-tunn2-1369310470 Dynamic IPv4 10.51.0.0/16 100 None VPN tunnel [tunnel-4](https://console.cloud.google.com/hybrid/vpn/tunnels/details/northamerica-northeast1/tunnel-4?project=vpn-aws-obs)
check netstat from both vms
AWS
ubuntu@ip-10-51-1-164:~$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.51.1.1 0.0.0.0 UG 0 0 0 eth0
10.51.0.2 10.51.1.1 255.255.255.255 UGH 0 0 0 eth0
10.51.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.51.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
GCP
michael@vpn-jump-private2:~$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.101.0.1 0.0.0.0 UG 0 0 0 ens4
10.101.0.1 0.0.0.0 255.255.255.255 UH 0 0 0 ens4
sudo apt install net-tools
first verify connectivity between VMs in both CSPs - spin up 2 more VMs
we are good on GCP
michael@vpn-jump-private2:~$ ping 10.101.0.5
PING 10.101.0.5 (10.101.0.5) 56(84) bytes of data.
64 bytes from 10.101.0.5: icmp_seq=1 ttl=64 time=1.56 ms
AWS good
ubuntu@ip-10-51-1-164:~$ ping 10.51.1.185
PING 10.51.1.185 (10.51.1.185) 56(84) bytes of data.
64 bytes from 10.51.1.185: icmp_seq=1 ttl=64 time=1.15 ms
ubuntu@ip-10-51-1-185:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=64 time=0.627 msChecking AWS propagation on the routes - off - this may be the issue
editing route propagation to route through the VPG
after
Routing working now from GCP to AWS
michael@vpn-jump-private2:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=63 time=17.7 mswe can ping from GCP to AWS (prem) 10.101.0.0/16 to 10.51.0.0/16
and we can ping the reverse AWS(prem) to GCP 10.51.0.0/16 to 10.101.0.0/16
ubuntu@ip-10-51-1-185:~$ ping 10.101.0.4
PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.
64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=16.7 ms
the other VM
Last login: Sun Aug 27 20:25:15 2023 from 10.51.2.201
ubuntu@ip-10-51-1-164:~$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.51.1.1 0.0.0.0 UG 0 0 0 eth0
10.51.0.2 10.51.1.1 255.255.255.255 UGH 0 0 0 eth0
10.51.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.51.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
ubuntu@ip-10-51-1-164:~$ ping 10.101.0.4
PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.
64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=17.0 msbaseline IP from AWS to GCP
ubuntu@ip-10-51-1-164:~$ curl https://199.36.153.8:443
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
ubuntu@ip-10-51-1-164:~$ curl http://199.36.153.8
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 401 (Unauthorized)!!1</title>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>401.</b> <ins>That’s an error.</ins>
<p>Your client does not have permission to the requested URL <code>/</code>. <ins>That’s all we know.</ins>https://us-east-1.console.aws.amazon.com/route53/v2/hostedzones?region=us-east-1#CreateHostedZone
Plural A records use CR/LF separators
private.googleapis.com = 199.36.153.8,199.36.153.9,199.36.153.10,199.36.153.11For each VPC that you associate with a private hosted zone, you must set the Amazon VPC settings enableDnsHostnames and enableDnsSupport to true.
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-updating
The "enable dns hostnames" was not set - we don't want it set - just "resolution" should be set
Do a dig on the private VM inside AWS
ubuntu@ip-10-51-1-164:~$ dig private.googleapis.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> private.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40574
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;private.googleapis.com. IN A
;; ANSWER SECTION:
private.googleapis.com. 300 IN A 199.36.153.10
private.googleapis.com. 300 IN A 199.36.153.11
private.googleapis.com. 300 IN A 199.36.153.8
private.googleapis.com. 300 IN A 199.36.153.9
;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Aug 28 02:24:41 UTC 2023
;; MSG SIZE rcvd: 115
check an record not already in the private zone before we add it
ubuntu@ip-10-51-1-164:~$ dig storage.googleapis.com
; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> storage.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42780
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;storage.googleapis.com. IN A
;; ANSWER SECTION:
storage.googleapis.com. 167 IN A 142.251.167.128
storage.googleapis.com. 167 IN A 172.253.62.128
storage.googleapis.com. 167 IN A 172.253.63.128
storage.googleapis.com. 167 IN A 172.253.115.128
storage.googleapis.com. 167 IN A 172.253.122.128
storage.googleapis.com. 167 IN A 142.250.31.128
storage.googleapis.com. 167 IN A 142.251.16.128
storage.googleapis.com. 167 IN A 142.251.111.128
storage.googleapis.com. 167 IN A 142.251.163.128
on my macbook
michaelobrien@mbp7 pubsec-declarative-toolkit % dig storage.googleapis.com
; <<>> DiG 9.10.6 <<>> storage.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31183
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;storage.googleapis.com. IN A
;; ANSWER SECTION:
storage.googleapis.com. 292 IN A 172.217.13.208
storage.googleapis.com. 292 IN A 172.217.13.112
storage.googleapis.com. 292 IN A 172.217.13.144
storage.googleapis.com. 292 IN A 172.217.13.176
private domain not having effect on the VPC - checking dhcp options
Turn on VPC flow logs
Checking traceroute
ubuntu@ip-10-51-1-185:~$ traceroute storage.googleapis.com
traceroute to storage.googleapis.com (142.251.163.128), 64 hops max
1 10.51.2.64 0.214ms 0.193ms 0.179ms
2 216.182.230.245 2.080ms 216.182.239.185 1.653ms 9.689ms
3 100.65.96.144 8.767ms 100.65.50.16 7.394ms 8.256ms
4 100.66.36.92 63.251ms 100.66.25.194 19.483ms 12.593ms
5 241.0.11.192 0.994ms 100.66.26.212 1.404ms 1.496ms
6 241.0.11.216 0.783ms 0.329ms 0.344ms
7 240.1.208.12 16.359ms 15.870ms 15.808ms
8 100.65.91.4 45.599ms 240.1.208.14 15.816ms 15.595ms
9 99.83.71.68 15.827ms 15.361ms 15.359ms
10 142.251.55.118 17.134ms 216.239.43.24 17.432ms 17.353ms
11 241.0.11.195 0.970ms * *
12 240.1.208.12 17.303ms 108.170.249.162 17.040ms 22.631ms
13 108.170.249.67 15.965ms 240.1.208.14 15.805ms 15.751ms
14 172.253.68.51 17.195ms 16.947ms 17.158ms
15 * * *
16 * * *
17 108.170.249.108 17.892ms 17.381ms 17.306ms
18 172.253.65.78 15.935ms 15.328ms 15.245ms
19 216.239.40.133 18.348ms 18.743ms 19.057ms
20 142.250.209.70 18.043ms * *
21 * * *
22 * * *
23 209.85.252.200 16.217ms 15.756ms 15.844ms
24 * * *
25 * * *
26 * * *
27 * * *
ubuntu@ip-10-51-1-185:~$ traceroute private.googleapis.com
traceroute to private.googleapis.com (199.36.153.9), 64 hops max
1 * * check /etc/hosts.txt override
199.36.153.14 storage.googleapis.com
break it on purpose first to verify we don't get through a public secondary DNS
ubuntu@ip-10-51-1-185:~$ sudo vi /etc/hosts
ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
INFO 0828 03:48:20.911258 retry_util.py] Retrying request, attempt #1...
^CCaught CTRL-C (signal 2) - exiting
ubuntu@ip-10-51-1-185:~$ ping storage.googleapis.com
PING storage.googleapis.com (199.36.153.28) 56(84) bytes of data.
then set to a valid ip
ubuntu@ip-10-51-1-185:~$ sudo vi /etc/hosts
ubuntu@ip-10-51-1-185:~$ ping storage.googleapis.com
PING storage.googleapis.com (199.36.153.8) 56(84) bytes of data.
^C
--- storage.googleapis.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1029ms
ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.png
ubuntu@ip-10-51-1-185:~$
working only through the private IP as 199.36.153.8 is not routable on the internetInstall gcloud cli https://cloud.google.com/sdk/docs/install#linux
ubuntu@ip-10-51-1-185:~$ sudo apt-get update
Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
ubuntu@ip-10-51-1-185:~$ curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-444.0.0-linux-x86_64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 178M 100 178M 0 0 7618k 0 0:00:24 0:00:24 --:--:-- 1582k
ubuntu@ip-10-51-1-185:~$ tar -xf google-cloud-cli-444.0.0-linux-x86_64.tar.gz
ubuntu@ip-10-51-1-185:~$ ./google-cloud-sdk/install.sh
Welcome to the Google Cloud CLI!
To help improve the quality of this product, we collect anonymized usage data
and anonymized stacktraces when crashes are encountered; additional information
is available at <https://cloud.google.com/sdk/usage-statistics>. This data is
handled in accordance with our privacy policy
<https://cloud.google.com/terms/cloud-privacy-notice>. You may choose to opt in this
collection now (by choosing 'Y' at the below prompt), or at any time in the
future by running the following command:
gcloud config set disable_usage_reporting false
Do you want to help improve the Google Cloud CLI (y/N)? y
Your current Google Cloud CLI version is: 444.0.0
The latest available version is: 444.0.0
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Components │
├───────────────┬──────────────────────────────────────────────────────┬──────────────────────────────┬──────────┤
│ Status │ Name │ ID │ Size │
├───────────────┼──────────────────────────────────────────────────────┼──────────────────────────────┼──────────┤
│ Not Installed │ App Engine Go Extensions │ app-engine-go │ 4.5 MiB │
│ Not Installed │ Appctl │ appctl │ 21.0 MiB │
│ Not Installed │ Artifact Registry Go Module Package Helper │ package-go-module │ < 1 MiB │
│ Not Installed │ Cloud Bigtable Command Line Tool │ cbt │ 11.4 MiB │
│ Not Installed │ Cloud Bigtable Emulator │ bigtable │ 7.0 MiB │
│ Not Installed │ Cloud Datastore Emulator │ cloud-datastore-emulator │ 36.2 MiB │
│ Not Installed │ Cloud Firestore Emulator │ cloud-firestore-emulator │ 42.5 MiB │
│ Not Installed │ Cloud Pub/Sub Emulator │ pubsub-emulator │ 61.2 MiB │
│ Not Installed │ Cloud Run Proxy │ cloud-run-proxy │ 13.4 MiB │
│ Not Installed │ Cloud SQL Proxy │ cloud_sql_proxy │ 7.8 MiB │
│ Not Installed │ Cloud Spanner Emulator │ cloud-spanner-emulator │ 31.4 MiB │
│ Not Installed │ Cloud Spanner Migration Tool │ harbourbridge │ 20.9 MiB │
│ Not Installed │ Google Container Registry's Docker credential helper │ docker-credential-gcr │ 1.8 MiB │
│ Not Installed │ Kustomize │ kustomize │ 4.3 MiB │
│ Not Installed │ Log Streaming │ log-streaming │ 13.9 MiB │
│ Not Installed │ Minikube │ minikube │ 34.6 MiB │
│ Not Installed │ Nomos CLI │ nomos │ 26.4 MiB │
│ Not Installed │ On-Demand Scanning API extraction helper │ local-extract │ 14.4 MiB │
│ Not Installed │ Skaffold │ skaffold │ 22.7 MiB │
│ Not Installed │ Spanner migration tool │ spanner-migration-tool │ 20.9 MiB │
│ Not Installed │ Terraform Tools │ terraform-tools │ 66.1 MiB │
│ Not Installed │ anthos-auth │ anthos-auth │ 20.4 MiB │
│ Not Installed │ config-connector │ config-connector │ 56.7 MiB │
│ Not Installed │ enterprise-certificate-proxy │ enterprise-certificate-proxy │ 8.2 MiB │
│ Not Installed │ gcloud Alpha Commands │ alpha │ < 1 MiB │
│ Not Installed │ gcloud Beta Commands │ beta │ < 1 MiB │
│ Not Installed │ gcloud app Java Extensions │ app-engine-java │ 65.1 MiB │
│ Not Installed │ gcloud app Python Extensions │ app-engine-python │ 8.5 MiB │
│ Not Installed │ gcloud app Python Extensions (Extra Libraries) │ app-engine-python-extras │ 27.3 MiB │
│ Not Installed │ gke-gcloud-auth-plugin │ gke-gcloud-auth-plugin │ 7.9 MiB │
│ Not Installed │ kpt │ kpt │ 14.5 MiB │
│ Not Installed │ kubectl │ kubectl │ < 1 MiB │
│ Not Installed │ kubectl-oidc │ kubectl-oidc │ 20.4 MiB │
│ Not Installed │ pkg │ pkg │ │
│ Installed │ BigQuery Command Line Tool │ bq │ 1.6 MiB │
│ Installed │ Bundled Python 3.9 │ bundled-python3-unix │ 63.6 MiB │
│ Installed │ Cloud Storage Command Line Tool │ gsutil │ 11.3 MiB │
│ Installed │ Google Cloud CLI Core Libraries │ core │ 21.6 MiB │
│ Installed │ Google Cloud CRC32C Hash Tool │ gcloud-crc32c │ 1.2 MiB │
└───────────────┴──────────────────────────────────────────────────────┴──────────────────────────────┴──────────┘
To install or remove components at your current SDK version [444.0.0], run:
$ gcloud components install COMPONENT_ID
$ gcloud components remove COMPONENT_ID
To update your SDK installation to the latest version [444.0.0], run:
$ gcloud components update
Modify profile to update your $PATH and enable shell command completion?
Last login: Mon Aug 28 03:02:10 2023 from 10.51.2.201
ubuntu@ip-10-51-1-185:~$ gcloud init
Welcome! This command will take you through the configuration of gcloud.
Your current configuration has been set to: [default]
You can skip diagnostics next time by using the following flag:
gcloud init --skip-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).
You must log in to continue. Would you like to log in (Y/n)?
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32555940559.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fauthcode.html&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=TNrMg266uDdOt4gnpc9Lg5yTRL5B9L&prompt=consent&access_type=offline&code_challenge=kKbf_2KEOisLiCLtxbK7toHy4M6iIHya_lUDJIc85qk&code_challenge_method=S256
You are logged in as: [michael@obrien.software].
Pick cloud project to use:
[1] all-services-obs
[2] bce-dev-so
[3] cs-host-6e8fdca189434184a30c0c
[4] cuda-obs
[5] dep1-team1-dev-os
[6] dep1-team1-nprod-os
[7] dns-sim-prem
[8] dns-sim-prod-host
[9] dns-sim-prod-service1
[10] logging-hh015-gz357
[11] manual-lz
[12] monitoring-dev-hh015-gz357
[13] monitoring-nonprod-hh015-gz357
[14] monitoring-prod-hh015-gz357
[15] vpc-host-nonprod-hh015-gz357
[16] vpc-host-prod-hh015-gz357
[17] vpn-aws-obs
[18] vpn-aws-svc-obs
[19] Enter a project ID
[20] Create a new project
Please enter numeric choice or text value (must exactly match list item): 17
Your current project has been set to: [vpn-aws-obs].
Do you want to configure a default Compute Region and Zone? (Y/n)? y
Which Google Compute Engine zone would you like to use as project default?
If you do not specify a zone via a command line flag while working with Compute Engine resources, the default is assumed.
[1] us-east1-b
... [115] us-west4-c
[116] Do not set default zone
Please enter numeric choice or text value (must exactly match list item): 92
Your project default Compute Engine zone has been set to [northamerica-northeast1-a].
You can change it by running [gcloud config set compute/zone NAME].
Your project default Compute Engine region has been set to [northamerica-northeast1].
You can change it by running [gcloud config set compute/region NAME].
Created a default .boto configuration file at [/home/ubuntu/.boto]. See this file and
[https://cloud.google.com/storage/docs/gsutil/commands/config] for more
information about configuring Google Cloud Storage.
Your Google Cloud SDK is configured and ready to use!
* Commands that require authentication will use michael@obrien.software by default
* Commands will reference project `vpn-aws-obs` by default
* Compute Engine commands will use region `northamerica-northeast1` by default
* Compute Engine commands will use zone `northamerica-northeast1-a` by default
Run `gcloud help config` to learn how to change individual settings
This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects.
Run `gcloud topic configurations` to learn more.
Some things to try next:
* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.
* Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting
* Run `gcloud cheat-sheet` to see a roster of go-to `gcloud` commands.
ubuntu@ip-10-51-1-185:~$ gcloud config set project vpn-aws-obs
Updated property [core/project].
How do I verify that googleapis.com traffic is not going through the NAT or IGW on the public subnet - check routes
| Destination | Target | Status | Propagated |
|---|---|---|---|
| 0.0.0.0/0 | nat-05be2cd683e8b153c | Active | No |
| 10.51.0.0/16 | local | Active | No |
| 10.101.0.0/24 | vgw-0bd4928d84cd855cf | Active | Yes |
| 199.36.153.8/30 | vgw-0bd4928d84cd855cf | Active | Yes |
Destination Target Status Propagated 0.0.0.0/0 nat-05be2cd683e8b153c Active No 10.51.0.0/16 local Active No 10.101.0.0/24 vgw-0bd4928d84cd855cf Active Yes 199.36.153.8/30 vgw-0bd4928d84cd855cf Active Yes
Add a GCS bucket to be able to list/update from AWS
from AWS
ubuntu@ip-10-51-1-185:~$ gsutil ls
gs://aws-transfer-obs/
ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.pngUpdate: PGA is working as expected from on-prem (simulated by an AWS VPC via VPN - which works well as a non-GCP on-prem/ground) following procedures in the github issue below. Essentially the main changes are what is detailed in the deck and docs. Verified 1 and 2
1 - Propagate routes on both sides but with an additional static route on the 199.36.153.8/30 CIDR from the GCP side so that googleapis.com calls can use that BGP route instead of the on prem IG public IPs (199.* is not publicly routable).
2 - Add a on-prem private zone (Route53 in AWS) to A record the 4 199.36.153.8/30 PGA IPs.
3 - optionally add forwarding private zone from prem to gcp
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494
on an AWS private VM (ping back to GCP VM, run a GCS ls) ubuntu@ip-10-51-1-185:~$ ping 10.101.0.4 PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=17.7 ms ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/ gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.png
obriensystems
commented
1 year ago Add PSC Private Service Connect capability through endpoints (for now focus on onprem to GCP)
Tasks for gcloud and awscli - (KRM/Terraform/CloudFormation later)
obriensystems
commented
1 year ago changes
new file: environments/prod/outputs.tf
new file: environments/prod/psc.tf
new file: modules/22-private-service-connect/README.md
new file: modules/22-private-service-connect/dns.tf
new file: modules/22-private-service-connect/main.tf
new file: modules/22-private-service-connect/metadata.yaml
new file: modules/22-private-service-connect/outputs.tf
new file: modules/22-private-service-connect/variables.tf
new file: modules/22-private-service-connect/versions.tf
Step #3 - "tf plan": Terraform will perform the following actions:
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.google_compute_global_address.private_service_connect will be created
Step #3 - "tf plan": + resource "google_compute_global_address" "private_service_connect" {
Step #3 - "tf plan": + address = "10.3.0.5"
Step #3 - "tf plan": + address_type = "INTERNAL"
Step #3 - "tf plan": + creation_timestamp = (known after apply)
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + label_fingerprint = (known after apply)
Step #3 - "tf plan": + name = "global-psconnect-ip"
Step #3 - "tf plan": + network = "projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc"
Step #3 - "tf plan": + prefix_length = (known after apply)
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + purpose = "PRIVATE_SERVICE_CONNECT"
Step #3 - "tf plan": + self_link = (known after apply)
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.google_compute_global_forwarding_rule.forwarding_rule_private_service_connect will be created
Step #3 - "tf plan": + resource "google_compute_global_forwarding_rule" "forwarding_rule_private_service_connect" {
Step #3 - "tf plan": + base_forwarding_rule = (known after apply)
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + ip_address = (known after apply)
Step #3 - "tf plan": + ip_protocol = (known after apply)
Step #3 - "tf plan": + label_fingerprint = (known after apply)
Step #3 - "tf plan": + name = "globalrule"
Step #3 - "tf plan": + network = "projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc"
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + psc_connection_id = (known after apply)
Step #3 - "tf plan": + psc_connection_status = (known after apply)
Step #3 - "tf plan": + self_link = (known after apply)
Step #3 - "tf plan": + subnetwork = (known after apply)
Step #3 - "tf plan": + target = "all-apis"
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.net-host-prj.module.project.google_project.project will be updated in-place
Step #3 - "tf plan": ~ resource "google_project" "project" {
Step #3 - "tf plan": id = "projects/tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": ~ labels = {
Step #3 - "tf plan": - "date_modified" = "2023-09-05"
Step #3 - "tf plan": } -> (known after apply)
Step #3 - "tf plan": name = "TzPe-tlz-tlzprod-host4"
Step #3 - "tf plan": # (5 unchanged attributes hidden)
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.gcr.google_dns_managed_zone.private[0] will be created
Step #3 - "tf plan": + resource "google_dns_managed_zone" "private" {
Step #3 - "tf plan": + description = "Private DNS zone to configure gcr.io"
Step #3 - "tf plan": + dns_name = "gcr.io."
Step #3 - "tf plan": + force_destroy = false
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + name = "dz-gcr"
Step #3 - "tf plan": + name_servers = (known after apply)
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + visibility = "private"
Step #3 - "tf plan":
Step #3 - "tf plan": + private_visibility_config {
Step #3 - "tf plan": + networks {
Step #3 - "tf plan": + network_url = "projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc"
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.gcr.google_dns_record_set.cloud-static-records["*/CNAME"] will be created
Step #3 - "tf plan": + resource "google_dns_record_set" "cloud-static-records" {
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + managed_zone = "dz-gcr"
Step #3 - "tf plan": + name = "*.gcr.io."
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + rrdatas = [
Step #3 - "tf plan": + "gcr.io.",
Step #3 - "tf plan": ]
Step #3 - "tf plan": + ttl = 300
Step #3 - "tf plan": + type = "CNAME"
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.gcr.google_dns_record_set.cloud-static-records["/A"] will be created
Step #3 - "tf plan": + resource "google_dns_record_set" "cloud-static-records" {
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + managed_zone = "dz-gcr"
Step #3 - "tf plan": + name = "gcr.io."
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + rrdatas = [
Step #3 - "tf plan": + "10.3.0.5",
Step #3 - "tf plan": ]
Step #3 - "tf plan": + ttl = 300
Step #3 - "tf plan": + type = "A"
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.googleapis.google_dns_managed_zone.private[0] will be created
Step #3 - "tf plan": + resource "google_dns_managed_zone" "private" {
Step #3 - "tf plan": + description = "Private DNS zone to configure private.googleapis.com."
Step #3 - "tf plan": + dns_name = "googleapis.com."
Step #3 - "tf plan": + force_destroy = false
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + name = "dz-apis"
Step #3 - "tf plan": + name_servers = (known after apply)
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + visibility = "private"
Step #3 - "tf plan":
Step #3 - "tf plan": + private_visibility_config {
Step #3 - "tf plan": + networks {
Step #3 - "tf plan": + network_url = "projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc"
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.googleapis.google_dns_record_set.cloud-static-records["*/CNAME"] will be created
Step #3 - "tf plan": + resource "google_dns_record_set" "cloud-static-records" {
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + managed_zone = "dz-apis"
Step #3 - "tf plan": + name = "*.googleapis.com."
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + rrdatas = [
Step #3 - "tf plan": + "private.googleapis.com.",
Step #3 - "tf plan": ]
Step #3 - "tf plan": + ttl = 300
Step #3 - "tf plan": + type = "CNAME"
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.googleapis.google_dns_record_set.cloud-static-records["private/A"] will be created
Step #3 - "tf plan": + resource "google_dns_record_set" "cloud-static-records" {
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + managed_zone = "dz-apis"
Step #3 - "tf plan": + name = "private.googleapis.com."
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + rrdatas = [
Step #3 - "tf plan": + "10.3.0.5",
Step #3 - "tf plan": ]
Step #3 - "tf plan": + ttl = 300
Step #3 - "tf plan": + type = "A"
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.pkg_dev.google_dns_managed_zone.private[0] will be created
Step #3 - "tf plan": + resource "google_dns_managed_zone" "private" {
Step #3 - "tf plan": + description = "Private DNS zone to configure pkg.dev"
Step #3 - "tf plan": + dns_name = "pkg.dev."
Step #3 - "tf plan": + force_destroy = false
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + name = "dz-pkg-dev"
Step #3 - "tf plan": + name_servers = (known after apply)
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + visibility = "private"
Step #3 - "tf plan":
Step #3 - "tf plan": + private_visibility_config {
Step #3 - "tf plan": + networks {
Step #3 - "tf plan": + network_url = "projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc"
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.pkg_dev.google_dns_record_set.cloud-static-records["*/CNAME"] will be created
Step #3 - "tf plan": + resource "google_dns_record_set" "cloud-static-records" {
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + managed_zone = "dz-pkg-dev"
Step #3 - "tf plan": + name = "*.pkg.dev."
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + rrdatas = [
Step #3 - "tf plan": + "pkg.dev.",
Step #3 - "tf plan": ]
Step #3 - "tf plan": + ttl = 300
Step #3 - "tf plan": + type = "CNAME"
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.module.pkg_dev.google_dns_record_set.cloud-static-records["/A"] will be created
Step #3 - "tf plan": + resource "google_dns_record_set" "cloud-static-records" {
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + managed_zone = "dz-pkg-dev"
Step #3 - "tf plan": + name = "pkg.dev."
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + rrdatas = [
Step #3 - "tf plan": + "10.3.0.5",
Step #3 - "tf plan": ]
Step #3 - "tf plan": + ttl = 300
Step #3 - "tf plan": + type = "A"
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.net-host-prj.module.network["tlzprod-svpc"].module.subnets["prsubnet02"].google_compute_subnetwork.subnetwork will be updated in-place
Step #3 - "tf plan": ~ resource "google_compute_subnetwork" "subnetwork" {
Step #3 - "tf plan": id = "projects/tzpe-tlz-tlzprod-host4/regions/northamerica-northeast1/subnetworks/tzpecnr-prsubnet02-host4-snet"
Step #3 - "tf plan": name = "tzpecnr-prsubnet02-host4-snet"
Step #3 - "tf plan": # (13 unchanged attributes hidden)
Step #3 - "tf plan":
Step #3 - "tf plan": ~ log_config {
Step #3 - "tf plan": - metadata = "EXCLUDE_ALL_METADATA" -> null
Step #3 - "tf plan": # (4 unchanged attributes hidden)
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": Plan: 11 to add, 2 to change, 0 to destroy.
Step #3 - "tf plan":
Step #3 - "tf plan": Changes to Outputs:
Step #3 - "tf plan": + dns_zone_gcr_name = "dz-gcr"
Step #3 - "tf plan": + dns_zone_googleapis_name = "dz-apis"
Step #3 - "tf plan": + dns_zone_pkg_dev_name = "dz-pkg-dev"
Step #3 - "tf plan": + forwarding_rule_name = "globalrule"
Step #3 - "tf plan": + forwarding_rule_target = "all-apis"
Step #3 - "tf plan": + global_address_id = (known after apply)
Step #3 - "tf plan": + private_service_connect_ip = "10.3.0.5"
Step #3 - "tf plan": + private_service_connect_name = "global-psconnect-ip"PSC
DNS
Missing Service Directory
need to enable
servicedirectory.googleapis.com
networkconnectivity.googleapis.com
dns.googleapis.com (already in)See related forwarding rule issue https://github.com/GoogleCloudPlatform/magic-modules/pull/7480 https://github.com/hashicorp/terraform-provider-google/issues/9758
after servicedirectory enablement - we get the service directory entry - although with a default region - to be fixed
Verify static internal IP for the PSC endpoint
switch from the global google_compute_global_forwarding_rule to google_compute_forwarding_rule https://github.com/hashicorp/terraform-provider-google-beta/blob/main/website/docs/r/compute_forwarding_rule.html.markdown https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_global_forwarding_rule.html https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_forwarding_rule.html
For regional change to the PSC endpoint with
# example from https://github.com/terraform-google-modules/terraform-google-network/tree/master/examples/private_service_connect
module "private_service_connect" {
source = "../../modules/22-private-service-connect"
project_id = module.net-host-prj.project_id# var.project_id
# need array of subnets
# module.net-host-prj.network_name is object with 1 attribute "tlzprod-svpc"
# │ var.prod_host_net.networks[0].network_name is "tlzprod-svpc"
#subnetwork_self_link = module.net-host-prj.network_name[var.prod_host_net.networks[0].network_name].subnets[0].subnet_name
#network_self_link = module.net-host-prj.network_self_link #module.simple_vpc.network_self_link
network_self_link = "projects/${module.net-host-prj.project_id}/global/networks/${module.net-host-prj.network_name[var.prod_host_net.networks[0].network_name]}"
# module.net-host-prj.network_name[var.prod_host_net.networks[0].network_name]
private_service_connect_ip = "10.3.0.5"
forwarding_rule_target = "all-apis"
# unsupported - https://github.com/hashicorp/terraform-provider-google/issues/9758
region = "northamerica-northeast1"
}
p #3 - "tf plan":
Step #3 - "tf plan": Terraform will perform the following actions:
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.google_compute_forwarding_rule.forwarding_rule_private_service_connect will be created
Step #3 - "tf plan": + resource "google_compute_forwarding_rule" "forwarding_rule_private_service_connect" {
Step #3 - "tf plan": + base_forwarding_rule = (known after apply)
Step #3 - "tf plan": + creation_timestamp = (known after apply)
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + ip_address = "projects/tzpe-tlz-tlzprod-host4/global/addresses/global-psconnect-ip"
Step #3 - "tf plan": + ip_protocol = (known after apply)
Step #3 - "tf plan": + ip_version = (known after apply)
Step #3 - "tf plan": + label_fingerprint = (known after apply)
Step #3 - "tf plan": + name = "l7-ilb-forwarding-rule"
Step #3 - "tf plan": + network = "projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc"
Step #3 - "tf plan": + network_tier = "PREMIUM"
Step #3 - "tf plan": + port_range = (known after apply)
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + psc_connection_id = (known after apply)
Step #3 - "tf plan": + psc_connection_status = (known after apply)
Step #3 - "tf plan": + region = "northamerica-northeast1"
Step #3 - "tf plan": + self_link = (known after apply)
Step #3 - "tf plan": + service_name = (known after apply)
Step #3 - "tf plan": + subnetwork = (known after apply)
Step #3 - "tf plan":
Step #3 - "tf plan": + service_directory_registrations {
Step #3 - "tf plan": + namespace = (known after apply)
Step #3 - "tf plan": + service = (known after apply)
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.google_compute_global_forwarding_rule.forwarding_rule_private_service_connect will be destroyed
Step #3 - "tf plan": - resource "google_compute_global_forwarding_rule" "forwarding_rule_private_service_connect" {
Step #3 - "tf plan": - allow_psc_global_access = false -> null
Step #3 - "tf plan": - id = "projects/tzpe-tlz-tlzprod-host4/global/forwardingRules/globalrule" -> null
Step #3 - "tf plan": - ip_address = "10.3.0.5" -> null
Step #3 - "tf plan": - ip_protocol = "TCP" -> null
Step #3 - "tf plan": - label_fingerprint = "42WmSpB8rSM=" -> null
Step #3 - "tf plan": - labels = {} -> null
Step #3 - "tf plan": - name = "globalrule" -> null
Step #3 - "tf plan": - network = "https://www.googleapis.com/compute/v1/projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc" -> null
Step #3 - "tf plan": - project = "tzpe-tlz-tlzprod-host4" -> null
Step #3 - "tf plan": - psc_connection_id = "36394505062252549" -> null
Step #3 - "tf plan": - self_link = "https://www.googleapis.com/compute/v1/projects/tzpe-tlz-tlzprod-host4/global/forwardingRules/globalrule" -> null
Step #3 - "tf plan": - source_ip_ranges = [] -> null
Step #3 - "tf plan": - target = "all-apis" -> null
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.net-host-prj.module.project.google_project.project will be updated in-place
Step #3 - "tf plan": ~ resource "google_project" "project" {
Step #3 - "tf plan": id = "projects/tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": ~ labels = {
Step #3 - "tf plan": - "date_modified" = "2023-09-05"
Step #3 - "tf plan": } -> (known after apply)
Step #3 - "tf plan": name = "TzPe-tlz-tlzprod-host4"
Step #3 - "tf plan": # (5 unchanged attributes hidden)
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.net-host-prj.module.network["tlzprod-svpc"].module.subnets["prsubnet02"].google_compute_subnetwork.subnetwork will be updated in-place
Step #3 - "tf plan": ~ resource "google_compute_subnetwork" "subnetwork" {
Step #3 - "tf plan": id = "projects/tzpe-tlz-tlzprod-host4/regions/northamerica-northeast1/subnetworks/tzpecnr-prsubnet02-host4-snet"
Step #3 - "tf plan": name = "tzpecnr-prsubnet02-host4-snet"
Step #3 - "tf plan": # (13 unchanged attributes hidden)
Step #3 - "tf plan":
Step #3 - "tf plan": ~ log_config {
Step #3 - "tf plan": - metadata = "EXCLUDE_ALL_METADATA" -> null
Step #3 - "tf plan": # (4 unchanged attributes hidden)
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": Plan: 1 to add, 2 to change, 1 to destroy.
Step #4 - "tf apply": │ Error: Error creating ForwardingRule: googleapi: Error 400: Invalid value for field 'resource.target': ''. No target or backend service specified for forwarding rule., invalid
Step #4 - "tf apply": │
Step #4 - "tf apply": │ with module.private_service_connect.google_compute_forwarding_rule.forwarding_rule_private_service_connect,
Step #4 - "tf apply": │ on ../../modules/22-private-service-connect/main.tf line 37, in resource "google_compute_forwarding_rule" "forwarding_rule_private_service_connect":
Step #4 - "tf apply": │ 37: resource "google_compute_forwarding_rule" "forwarding_rule_private_service_connect" {
fix - swap in
name = var.forwarding_rule_name
target = var.forwarding_rule_target
Step #4 - "tf apply": │ Error: Error creating ForwardingRule: googleapi: Error 400: Invalid value for field 'resource.target': 'https://compute.googleapis.com/compute/beta/projects/tzpe-tlz-tlzprod-host4/regions/northamerica-northeast1/all-apis'. Invalid target. Must be either a valid In-Project Forwarding Rule Target URL, a valid Service Attachment URL, or a supported Google API bundle (global-only)., invalid
Step #4 - "tf apply": │
Step #4 - "tf apply": │ with module.private_service_connect.google_compute_forwarding_rule.forwarding_rule_private_service_connect,
Step #4 - "tf apply": │ on ../../modules/22-private-service-connect/main.tf line 37, in resource "google_compute_forwarding_rule" "forwarding_rule_private_service_connect":
Step #4 - "tf apply": │ 37: resource "google_compute_forwarding_rule" "forwarding_rule_private_service_connect" {
pending
PSC IP added to router advertisements - DONE
wildcard cname for *.googleapis.com
verify pga on for prod vpc - DONE
tf for dns ingress proxy policy (dns server policy) - google_dns_policy - DONE
https://cloud.google.com/vpc/docs/dns-vpc-hosted-services#auto-dns-consumer
https://cloud.google.com/vpc/docs/configure-private-google-access
obriensystems
commented
1 year ago PSC IP added to router advertisements - test results for PR
Step #3 - "tf plan": Terraform will perform the following actions:
Step #3 - "tf plan":
Step #3 - "tf plan": # module.partner-interconnect-primary.google_compute_router.router1 will be updated in-place
Step #3 - "tf plan": ~ resource "google_compute_router" "router1" {
Step #3 - "tf plan": id = "projects/tzpe-tlz-tlzprod-host4/regions/northamerica-northeast1/routers/interconnect-prod-router"
Step #3 - "tf plan": name = "interconnect-prod-router"
Step #3 - "tf plan": # (5 unchanged attributes hidden)
Step #3 - "tf plan":
Step #3 - "tf plan": ~ bgp {
Step #3 - "tf plan": # (3 unchanged attributes hidden)
Step #3 - "tf plan":
Step #3 - "tf plan": + advertised_ip_ranges {
Step #3 - "tf plan": + description = "PSC endpoint ip"
Step #3 - "tf plan": + range = "10.3.0.5"
Step #3 - "tf plan": }
Step #3 - "tf plan": # (1 unchanged block hidden)
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.net-host-prj.module.project.google_project.project will be updated in-place
Step #3 - "tf plan": ~ resource "google_project" "project" {
Step #3 - "tf plan": id = "projects/tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": ~ labels = {
Step #3 - "tf plan": - "date_modified" = "2023-09-05"
Step #3 - "tf plan": } -> (known after apply)
Step #3 - "tf plan": name = "TzPe-tlz-tlzprod-host4"
Step #3 - "tf plan": # (5 unchanged attributes hidden)
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.net-host-prj.module.network["tlzprod-svpc"].module.subnets["prsubnet02"].google_compute_subnetwork.subnetwork will be updated in-place
Step #3 - "tf plan": ~ resource "google_compute_subnetwork" "subnetwork" {
Step #3 - "tf plan": id = "projects/tzpe-tlz-tlzprod-host4/regions/northamerica-northeast1/subnetworks/tzpecnr-prsubnet02-host4-snet"
Step #3 - "tf plan": name = "tzpecnr-prsubnet02-host4-snet"
Step #3 - "tf plan": # (13 unchanged attributes hidden)
Step #3 - "tf plan":
Step #3 - "tf plan": ~ log_config {
Step #3 - "tf plan": - metadata = "EXCLUDE_ALL_METADATA" -> null
Step #3 - "tf plan": # (4 unchanged attributes hidden)
Step #3 - "tf plan": }
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": Plan: 0 to add, 3 to change, 0 to destroy.
Step #3 - "tf plan": ╷will add /32 for cidr
Step #3 - "tf plan": ~ bgp {
Step #3 - "tf plan": # (3 unchanged attributes hidden)
Step #3 - "tf plan":
Step #3 - "tf plan": ~ advertised_ip_ranges {
Step #3 - "tf plan": ~ range = "10.3.0.5" -> "10.3.0.5/32"Test results for DNS ingress policy for PSC endpoint The DNS server policy is separate from PSC creation. The IP for the policy is created on the fly
Step #3 - "tf plan": Terraform will perform the following actions:
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.google_compute_global_forwarding_rule.forwarding_rule_private_service_connect will be created
Step #3 - "tf plan": + resource "google_compute_global_forwarding_rule" "forwarding_rule_private_service_connect" {
Step #3 - "tf plan": + base_forwarding_rule = (known after apply)
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + ip_address = "projects/tzpe-tlz-tlzprod-host4/global/addresses/global-psconnect-ip"
Step #3 - "tf plan": + ip_protocol = (known after apply)
Step #3 - "tf plan": + label_fingerprint = (known after apply)
Step #3 - "tf plan": + name = "pscincoming"
Step #3 - "tf plan": + network = "projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc"
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan": + psc_connection_id = (known after apply)
Step #3 - "tf plan": + psc_connection_status = (known after apply)
Step #3 - "tf plan": + self_link = (known after apply)
Step #3 - "tf plan": + subnetwork = (known after apply)
Step #3 - "tf plan": + target = "all-apis"
Step #3 - "tf plan": }
Step #3 - "tf plan":
Step #3 - "tf plan": # module.private_service_connect.google_dns_policy.default_policy will be created
Step #3 - "tf plan": + resource "google_dns_policy" "default_policy" {
Step #3 - "tf plan": + description = "Managed by Terraform"
Step #3 - "tf plan": + enable_inbound_forwarding = true
Step #3 - "tf plan": + enable_logging = false
Step #3 - "tf plan": + id = (known after apply)
Step #3 - "tf plan": + name = "psc-ingress-policy"
Step #3 - "tf plan": + project = "tzpe-tlz-tlzprod-host4"
Step #3 - "tf plan":
Step #3 - "tf plan": + networks {
Step #3 - "tf plan": + network_url = "projects/tzpe-tlz-tlzprod-host4/global/networks/tzpecnr-tlzprod-svpc-vpc"
Step #3 - "tf plan": }
Step #3 - "tf plan": }DNS server policy
Details on IP association
Reserved internal IP
obriensystems
commented
1 year ago Testing within gcloud
test internet route
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ gcloud config set project tzpe-tlz-tlzprod-host4
Updated property [core/project].
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (tzpe-tlz-tlzprod-host4)$ gsutil ls
gs://testbucket-tls/
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (tzpe-tlz-tlzprod-host4)$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage.googleapis.com/storage/v1/b?project=$(gcloud config get-value project)"
Your active configuration is: [cloudshell-23641]
{
"kind": "storage#buckets",
"items": [
{
"kind": "storage#bucket",
"selfLink": "https://www.googleapis.com/storage/v1/b/testbucket-tls",
"id": "testbucket-tls",
"name": "testbucket-tls",
"projectNumber": "604049845861",
"metageneration": "1",
"location": "NORTHAMERICA-NORTHEAST1",
"storageClass": "STANDARD",
"etag": "CAE=",
"timeCreated": "2023-09-06T16:43:27.996Z",
"updated": "2023-09-06T16:43:27.996Z",
"iamConfiguration": {
"bucketPolicyOnly": {
"enabled": true,
"lockedTime": "2023-12-05T16:43:27.996Z"
},
"uniformBucketLevelAccess": {
"enabled": true,
"lockedTime": "2023-12-05T16:43:27.996Z"
},
"publicAccessPrevention": "enforced"
},
"locationType": "region"
}
]
}test private route root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (tzpe-tlz-tlzprod-host4)$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage-pscincoming.p.googleapis.com/storage/v1/b?project=$(gcloud config get-value project)" Your active configuration is: [cloudshell-23641] { "kind": "storage#buckets", "items": [ { "kind": "storage#bucket", "selfLink": "https://www.googleapis.com/storage/v1/b/testbucket-tls", "id": "testbucket-tls", "name": "testbucket-tls", "projectNumber": "604049845861", "metageneration": "1", "location": "NORTHAMERICA-NORTHEAST1", "storageClass": "STANDARD", "etag": "CAE=", "timeCreated": "2023-09-06T16:43:27.996Z", "updated": "2023-09-06T16:43:27.996Z", "iamConfiguration": { "bucketPolicyOnly": { "enabled": true, "lockedTime": "2023-12-05T16:43:27.996Z" }, "uniformBucketLevelAccess": { "enabled": true, "lockedTime": "2023-12-05T16:43:27.996Z" }, "publicAccessPrevention": "enforced" }, "locationType": "region" } ] }
testing on premTesting the PSC private endpoint in VPC and on-prem - on shadow manual org
GCP based
internet based
ichael@cloudshell:~ (vpn-aws-obs)$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage.googleapis.com/storage/v1/b?project=$(gcloud config get-value project)"
Your active configuration is: [cloudshell-2578]
{
"kind": "storage#buckets",
"items": [....
private based (notice that the CNAME record is wildcarded - so any psc application name works
michael@cloudshell:~ (vpn-aws-obs)$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage-anyrandomstr.p.googleapis.com/storage/v1/b?project=$(gcloud config get-value project)"
Your active configuration is: [cloudshell-2578]
{
"kind": "storage#buckets",
"items": [
{
"kind": "storage#bucket",
"selfLink": "https://www.googleapis.com/storage/v1/b/aws-transfer-obs",
"id": "aws-transfer-obs",
"name": "aws-transfer-obs",
"projectNumber": "1027024291560",
"metageneration": "1",
"location": "NORTHAMERICA-NORTHEAST1",
"storageClass": "STANDARD",
"etag": "CAE=",
"timeCreated": "2023-08-28T03:24:27.491Z",
"updated": "2023-08-28T03:24:27.491Z",
"iamConfiguration": {
"bucketPolicyOnly": {
"enabled": true,
"lockedTime": "2023-11-26T03:24:27.491Z"
},
"uniformBucketLevelAccess": {
"enabled": true,
"lockedTime": "2023-11-26T03:24:27.491Z"
},
"publicAccessPrevention": "enforced"
},
"locationType": "region"
},
{
"kind": "storage#bucket",
"selfLink": "https://www.googleapis.com/storage/v1/b/df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
"id": "df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
"name": "df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
"projectNumber": "1027024291560",
"metageneration": "1",
"location": "NORTHAMERICA-NORTHEAST1",
"storageClass": "STANDARD",
"etag": "CAE=",
"timeCreated": "2023-08-29T14:52:20.813Z",
"updated": "2023-08-29T14:52:20.813Z",
"lifecycle": {
"rule": [
{
"action": {
"type": "Delete"
},
"condition": {
"daysSinceCustomTime": 20
}
}
]
},
"labels": {
"cdf_instance": "test"
},
"iamConfiguration": {
"bucketPolicyOnly": {
"enabled": false
},
"uniformBucketLevelAccess": {
"enabled": false
},
"publicAccessPrevention": "inherited"
},
"locationType": "region"
}
]
}
michael@cloudshell:~ (vpn-aws-obs)$ add CNAME record *.googleapis.com pointing to private.googleapis.com
add A record private.googleapis.com pointing to the IP for the PSC endpoint
delete the PGA 199.36.153.8/30 first
then add one to 10.102.0.2
Missing on manual PSC setup - the googleapis.com private zone with CNAME and A record to private.googleapis.com (see TF created PSC)
gcloud dns --project=vpn-aws-obs managed-zones create dz-apis --description="Private DNS zone to configure private.googleapis.com" --dns-name="googleapis.com." --visibility="private" --networks="https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc"
gcloud dns --project=vpn-aws-obs record-sets create *.googleapis.com. --zone="dz-apis" --type="CNAME" --ttl="300" --rrdatas="private.googleapis.com."
gcloud dns --project=vpn-aws-obs record-sets create private.googleapis.com. --zone="dz-apis" --type="A" --ttl="300" --rrdatas="10.102.0.2"
wait for DNS propagation
ubuntu@ip-10-51-1-185:~$ nslookup private.googleapis.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: private.googleapis.com
Address: 199.36.153.10
Name: private.googleapis.com
Address: 199.36.153.11
Name: private.googleapis.com
Address: 199.36.153.8
Name: private.googleapis.com
Address: 199.36.153.9
AWS based
michaelobrien@mbp7 pbmm-on-gcp-onboarding % ssh -A ubuntu@jump.obrienlabs.cloud
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)
*** System restart required ***
Last login: Tue Sep 5 13:21:59 2023 from 174.112.128.160
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.185
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)
*** System restart required ***
Last login: Tue Sep 5 13:22:16 2023 from 10.51.2.201
change route advertised for the PSC endpoint ip
on the private AWS vm - need a pre-generated token for the -H header
ubuntu@ip-10-51-1-185:~$ curl -X GET "https://10.102.0.2/storage/v1/b?project=vpn-aws-obs"
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
fmichaelobrien
commented
1 year ago By default global dynamic routing on the vpc is on/global to accommodate PSC mode global over regional https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/environments/prod/prod-network.auto.tfvars#L28
obriensystems
commented
1 year ago On AWS private VM back to GCP via public - via PSC
ubuntu@ip-10-51-1-185:~$ nslookup bigquery.googleapis.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
bigquery.googleapis.com canonical name = private.googleapis.com.
Name: private.googleapis.com
Address: 10.102.0.2GCP: 10.102.0.2 is the PSC endpoint 2 on GCP
AWS: Because we have a CNAME from the wildcard *.googleapis.com to private.googleapis.com - the A record 10.102.0.2 in Route53
michaelobrien@mbp7 pbmm-on-gcp-onboarding % ssh -A ubuntu@jump.obrienlabs.cloud
ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.185
ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"kind": "bigquery#tableDataList",
"etag": "xTHaKWTlPPkd2/tRo1kgDQ==",
"totalRows": "4331",
"pageToken": "BHQG7VMKRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAW5TQNYWWC53TFVXWE4YR5ADGWH7PAAAAAEQLOJXWY3DFOJRGYYLEMUNCINZRGMYTONRXGYWTEZRSGIWTIMRUGYWTQZLEMYWTINDGGM4WKNDEGQ4WGYISBNZG63DMMVZGE3DBMRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
"rows": [
{
"f": [
{
"v": "45.424712"
},
{
"v": "-75.698209"
}
]
},
{
"f": [
{
"v": "45.424712"
},
{
"v": "-75.6949"
}
]
}
]
}
re-authenticate on each login
Add service account for bigquery
biqueryAdmin roleadd serviceAccountTokenCreator role to adminssee https://cloud.google.com/bigquery/docs/use-service-accounts https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev
ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=76......0-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fapplicationdefaultauthcode.html&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login&state=a7fI6fW8EBiD3skC32FHcB6Qr5t46L&prompt=consent&access_type=offline&code_challenge=qN6I0UEBHl0SQ8dSQ7qbPO_y7pdxU54Zuant3PSf9xc&code_challenge_method=S256
Enter authorization code: 4/0Adeu5.........AS-6OWiDm4Nr36BJHlNhIiCFHVi1RALV0wS1XhcKdnkCcr7g
Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]
These credentials will be used by any library that requests Application Default Credentials (ADC).
ubuntu@ip-10-51-1-185:~$ cat /home/ubuntu/.config/gcloud/application_default_credentials.json
{
"delegates": [],
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/big....a@vpn-aws-obs.iam.gserviceaccount.com:generateAccessToken",
"source_credentials": {
"client_id": "76408.........6pt8ejuq83di341hur.apps.googleusercontent.com",
"client_secret": "d-FL.............HD0Ty",
"refresh_token": "1//0.....................AESNgF-L9IrXpW9FfxsR8BZmLIorBGxG5Bxf2HlTZW6Vpg_9muB8CSwLgy4U-pmR04Aje0ppUdUDg",
"type": "authorized_user"
},
"type": "impersonated_service_account"
ubuntu@ip-10-51-1-185:~$ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"kind": "bigquery#tableDataList",
"etag": "lj66QnFhOWfLoLt73uLkyw==",
"totalRows": "4331",
"pageToken": "BF6E6O4PRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAW5TQNYWWC53TFVXWE4YR5ADGWH7PAAAAAEQLOJXWY3DFOJRGYYLEMUNCINZRGMYTONRXGYWTEZRSGIWTIMRUGYWTQZLEMYWTINDGGM4WKNDEGQ4WGYISBNZG63DMMVZGE3DBMRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
"rows": [
{
"f": [
{
"v": "45.424712"
test access by temporarily deleting bigquery.admin role
Reauthenticate impersonation to check biqquery deny
ubuntu@ip-10-51-1-185:~$gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=76408....&state=OZUfD7okVYuyMyKGDuBCTEgDvNA3g4&prompt=consent&access_type=offline&code_challenge=ET3rQo9WuB86WSG7vwQkKXdwA0Tl2uce_B7FvBm59jg&code_challenge_method=S256
Enter authorization code: 4/0Adeu5BU1Gmjm...........
Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]
These credentials will be used by any library that requests Application Default Credentials (ADC).
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"error": {
"code": 403,
"message": "Access Denied: Table vpn-aws-obs:rollerblade.rollerblade: Permission bigquery.tables.getData denied on table vpn-aws-obs:rollerblade.rollerblade (or it may not exist).",
"errors": [
{
"message": "Access Denied: Table vpn-aws-obs:rollerblade.rollerblade: Permission bigquery.tables.getData denied on table vpn-aws-obs:rollerblade.rollerblade (or it may not exist).",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"Reinstate bigquery.admin
ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=764086.....in&state=VTg2YpRi4woOr5xsOPKCINRCQemoqU&prompt=consent&access_type=offline&code_challenge=9cDrYAw20PjQ087WE-vhkRTSTKTZIvxpQphne3AaPGk&code_challenge_method=S256
Enter authorization code: 4/0Ade...5DKUtVunUt6FiW8iPw
Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]
These credentials will be used by any library that requests Application Default Credentials (ADC).
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
"kind": "bigquery#tableDataList",
"etag": "z+UPOJmTYBEcLen4sYqTAA==",
"totalRows": "4331",
"pageToken": "BFJDEQMPRIA...MRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
"rows": [
{
"f": [
{
"v": "45.424712"
},
{
"v": "-75.698209"
}
20240406: Closing issue during retrofit/rebase of this TEF V1 based/modified repo to TEF V4 standards This issue may participate in the LZ refactor after rebase Query on all issues related to the older V1 version via the tag https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/labels/2024-pre-tef-v4
shadow https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/286 and https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/468
20230827:2300: pivot to PSC from PGA https://cloud.google.com/vpc/docs/about-accessing-vpc-hosted-services-endpoints
https://cloud.google.com/vpc/docs/configure-private-service-connect-apis
Document and simulate GCP + Customer procedure: shadow https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494 follow https://cloud.google.com/vpc/docs/private-access-options
Requirements
Asset Inventory - GCP side
Notes:
https://docs.google.com/presentation/d/13sjT2tJ4yLIYGRREE3wBrylB1OvcEMpKdquVuJB_nX4/edit?resourcekey=0-N3DruQaiutFvZ98HTT7-vQ#slide=id.g1154b3b950f_2_3458 slide 27 https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid https://cloud.google.com/vpc/docs/configure-private-service-connect-apis#on-premises
Customer environment consists of already created interconnect/VPN where there is a BGP route for the DNS proxy egress from onprem Use case is one where google APIs and googledomains.com queries into GCP both resolve and are kept private on the premium google network
https://cloud.google.com/vpc/docs/private-service-connect#:~:text=Similarly%2C%20a%20Private%20Service%20Connect,internal%20IP%20addresses%20for%20endpoints.
Reference: procedures