Open snuggie12 opened 1 year ago
I also discovered that ServiceDirectoryService
is missing the type
field where you can say the service is a PRIVATE_SERVICE_CONNECT
. Not sure if that creates the forwarding rule for you or not.
Hi, curious if there is any update on this? We ended up creating a PSC via other means as well, but using CC is as well our preference so just a quick ping to mention we'd be interested in this, thanks in advance!
I just found out that there is no way to create a forwarding rule pointing to a serviceAttachment (PSC), I really need it too :/
@snuggie12 Are you asking the target
field in ComputeForwardingRule
to support ComputeServiceAttachement
?
Something like below?
spec:
target:
targetServiceAttachmentRef:
external: "projects/destination-project/regions/us-east1/serviceAttachments/target-service
yes @diviner524 , We too expect this behavior
Also in the config connector docs, we could see only the below targets and not for ServiceAttachmentRef
@Dineshvcetster: So ideally we want to add targetServiceAttachmentRef
to support referencing a ComputeServiceAttachment resource.
Having said that, I believe we can use the external
field of any other existing ref fields as a workaround, for example:
spec:
target:
targetTCPProxyRef:
external: "projects/destination-project/regions/us-east1/serviceAttachments/target-service
On top of the workaround above, there are a few other known issues with this combination, @justinsb has a recent fix for this scenario, which should be included in our next release (v1.111.0). I suggest you wait until the release is out and then apply the workaround above to see if it works.
Our strategy here BTW is to try to make sure we have test coverage, and now that we are fully OSS (i.e. all development happens on github) to ensure that we coverage in the mockgcp layer so that we can run our tests on github without relying on the "real" GCP APIs. I believe I got to forwardingRule in some of my WIP PRs, so now it's a matter of getting that all merge-ready (breaking it into smaller PRs) and reviewed.
In addition to the "quick" mockgcp tests, we also run tests against the real GCP APIs "behind the scenes" - it's quite a thorny issue to run tests against real infrastructure for community pull requests.
I think we should try to make sure we have test coverage for external
first (if we don't already), and then we can also add test coverage for the other scenarios identified here (and ensure that they work!)
/assign
(Assigning to myself, though if anyone else wants to work on it, please feel free and comment here to avoid duplication of effort!)
@diviner524, we tried with config connector v1.111.0 version but ended with below error.
As you mentioned, i tried all other targets but no luck
Error creating ForwardingRule: googleapi: Error 400: Invalid value for field 'resource.IPAddress': ''. The URL is malformed., invalid
I tried with addressType Internal, External and allowed psc to be global but still getting same issue.
Below Config i have used in my case.
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeForwardingRule
metadata:
labels:
label-one: "value-one"
name: computeforwardingrule-regional
namespace: cnrm-gcp-infra
spec:
description: "A regional forwarding rule"
target:
targetVPNGatewayRef:
external: <serviceAttachmentURI>
location: europe-west1
ipProtocol: "ESP"
ipAddress:
addressRef:
name: computeforwardingrule-dep-regional
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeAddress
metadata:
name: computeforwardingrule-dep-regional
namespace: cnrm-gcp-infra
labels:
label-one: "value-one"
spec:
description: IP Address for PSC Endpoint
location: europe-west1
@Dineshvcetster did you check the ComputeAddress resource computeforwardingrule-dep-regional
and does it have a valid address
value in its KRM resource spec?
kubectl describe ComputeAddress computeforwardingrule-dep-regional -n cnrm-gcp-infra
@diviner524 , Please find the spec
Name: computeforwardingrule-dep-regional
Namespace: cnrm-gcp-infra
Labels: label-one=value-one
API Version: compute.cnrm.cloud.google.com/v1beta1
Kind: ComputeAddress
Metadata:
Creation Timestamp: 2023-11-03T06:44:24Z
Finalizers:
cnrm.cloud.google.com/finalizer
cnrm.cloud.google.com/deletion-defender
Generation: 2
Resource Version: 37089726
UID: 082572f0-4705-4933-828b-9676447add7a
Spec:
Address: 10.11.130.18
Address Type: INTERNAL
Description: IP Address for PSC Endpoint
Location: global
Network Ref:
External: <VPN Name>
Purpose: PRIVATE_SERVICE_CONNECT
Resource ID: computeforwardingrule-dep-regional
Status:
Conditions:
Last Transition Time: 2023-11-03T06:44:37Z
Message: The resource is up to date
Reason: UpToDate
Status: True
Type: Ready
Creation Timestamp: 2023-11-02T23:44:25.716-07:00
Label Fingerprint: DvLa3Bl79lw=
Observed Generation: 2
Self Link: https://www.googleapis.com/compute/v1/projects/<project>/global/addresses/computeforwardingrule-dep-regional
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Updating 35s computeaddress-controller Update in progress
Normal UpToDate 22s (x2 over 23s) computeaddress-controller The resource is up to date
The 400 error message seems to indicate the API was getting an empty IP address value. However I do see 10.11.130.18
in your ComputeAddress spec.
Have you tried to change:
ipAddress:
addressRef:
name: computeforwardingrule-dep-regional
To:
ipAddress:
addressRef:
external: "10.11.130.18"
Also @justinsb might be able to provide some sample YAMLs to show how we can configure a forwarding rule with PSC.
@diviner524 , i tried and it didnt work. serviceAttachmentURI refers to cloudSQL which was created for our testing purpose with psc enabled and project allowed config
When i use addressType EXTERNAL and use targetVPNGatewayRef with external refers to the serviceAttachmentURI, i am getting below error.
Error creating GlobalForwardingRule: googleapi: Error 400: Invalid value for field 'resource.target': ''. No target or backend service specified for forwarding rule., invalid
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeForwardingRule
metadata:
labels:
label-one: "value-one"
name: computeforwardingrule-regional
spec:
description: "A regional forwarding rule"
target:
targetVPNGatewayRef:
external: <serviceAttachmentURI>
ipProtocol: "ESP"
location: global
ipAddress:
addressRef:
external: "10.11.130.18"
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeAddress
metadata:
name: computeforwardingrule-dep-regional
namespace: cnrm-gcp-infra
labels:
label-one: "value-one"
spec:
description: IP Address for PSC Endpoint
location: global
@Dineshvcetster There is a bug somewhere, the error message is not the real one but I have no idea what the real error is. If you attach the compute address using an external ref with its url instead of an internal reference it will work (we do that).
(external expects a gcp url, not the ip address)
@schmurfy, Thanks for the input.
After some trail and error, i ended with the below error.
FYI, I switched to regional instead of global
Update call failed: error applying desired state: summary: Error waiting to create ForwardingRule: Error waiting for Creating ForwardingRule:
Do we really need serviceDirectoryRegistrations?
I tried with below config but no luck
serviceDirectoryRegistrations:
- namespace: goog-psc-default
If we use global forwarding rule, we are getting
summary: Error creating GlobalForwardingRule: googleapi: Error 400: Invalid value for field 'resource.target': ''. Unrecognized forwarding rule target specified SERVICE_ATTACHMENT, invalid.
@justinsb , @diviner524 , Could you please share the sample yaml which you have used for serviceAttachment?
@Dineshvcetster - This may not be exactly what you are looking for, however my team recently got PSC working with Config Connector and I wanted to share what the configuration looks like. For our use case, PSC is providing connectivity from a consumer project to an internal GKE endpoint in another project with out having to peer the VPC networks. This is how we got it to work:
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeSubnetwork
metadata:
name: pscnetworkcrd-uscentral
spec:
ipCidrRange: X.X.X.X/27
region: us-central1
description: psc
privateIpGoogleAccess: false
purpose: PRIVATE_SERVICE_CONNECT
networkRef:
name: VPC_NAME
logConfig:
aggregationInterval: INTERVAL_10_MIN
flowSampling: 0.5
metadata: INCLUDE_ALL_METADATA
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeServiceAttachment
metadata:
name: computeserviceattachment-crd
spec:
location: us-central1
description: "A sample service attachment"
targetServiceRef:
external: "projects/PROJECT_ID_PRODUCER/regions/us-central1/forwardingRules/INTERNAL_LB_ID"
connectionPreference: "ACCEPT_MANUAL"
natSubnets:
- name: "pscnetworkcrd-uscentral"
enableProxyProtocol: false
consumerAcceptLists:
- projectRef:
external: "PROJECT_ID_CONSUMER"
connectionLimit: 100
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeAddress
metadata:
name: testconsumecrd-sec
spec:
description: Static IP
addressType: INTERNAL
location: us-central1
ipVersion: IPV4
subnetworkRef:
name: VPC_SUBNET_NAME
namespace: PROJECT_NAMESPACE
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeForwardingRule
metadata:
labels:
name: computeforwardingrule-crd
spec:
loadBalancingScheme: ""
networkRef:
name: VPC_NAME
target:
targetTCPProxyRef:
external: projects/PROJECT_ID_PRODUCER/regions/us-central1/serviceAttachments/computeserviceattachment-crd
location: us-central1
ipAddress:
addressRef:
external: "projects/PROJECT_ID_CONSUMER/regions/us-central1/addresses/testconsumecrd-sec"
Note: to get this to work, we had to set loadBalancingScheme to an empty string in the ComputeForwardingRule.
@tedelwartowski-bestbuy, thanks for your effort. I also ended with the similar config except the computeaddress with namespace: PROJECT_NAMESPACE
But I have no idea why we need servicedirectory(an optional field)?
Below error I am getting with
Creating ForwardingRule: APPLICATION_ERROR;google.cloud.servicedirectory.v1beta1/ManagedResourceService.AddServiceBundle;Permission 'servicedirectory.services.create' denied on resource 'projects//locations/europe-west1/namespaces/goog-psc-default'.
It is working now. Thanks @diviner524 @schmurfy @tedelwartowski-bestbuy
apiVersion: compute.cnrm.cloud.google.com/v1beta1 kind: ComputeServiceAttachment metadata: name: computeserviceattachment-crd spec: ... targetServiceRef: external: "projects/PROJECT_ID_PRODUCER/regions/us-central1/forwardingRules/INTERNAL_LB_ID"
The bit I'm failing to figure out is how you reference the forwarding-rule via targetServiceRef if it has been generated by the GKE Gateway API Controller. The value for INTERNAL_LB_ID
is generated, e.g. gkegw1-cdeo-gatewaynamespace-gatewayname-lwu3nrtyr2n0
. I must be missing a step here.
Checklist
Describe the feature or resource
According to TF docs a consumer can more or less be created by creating the following resources:
The
ComputeForwardingRule
requires a new target type to hit the service attachment.However, when you create a consumer using the console additional objects get created:
PSC Connection ID
so I presume it's an object of some sort.Weirdly the service directory API doesn't need enabled to work, but I turned it on to see which objects were getting created.
I'm willing to try and create all these objects separately but at the very least I think the forwarding rule needs updated in order to talk to a service attachment.
Additional information
https://cloud.google.com/vpc/docs/configure-private-service-connect-services#create-endpoint shows similar instructions to the terraform docs.
Importance
We are currently testing this feature out so it's not a blocker since this can be created via other means, but using kcc is our preference with these things.