Cannot specify an api_endpoint, returns E1003/E1020

[alice-rc]

This works:

from google.cloud import secretmanager_v1 as secretmanager
from google.api_core.client_options import ClientOptions
from google.oauth2 import service_account

...
options_clean = ClientOptions()
sm_client_clean = secretmanager.SecretManagerServiceClient(credentials=account, client_options=options_clean)
response_clean = sm_client_clean.access_secret_version(request=sm_version_request)

This does not:

from google.cloud import secretmanager_v1 as secretmanager
from google.api_core.client_options import ClientOptions
from google.oauth2 import service_account

...
options_priv = ClientOptions(api_endpoint="private.googleapis.com")
sm_client_priv = secretmanager.SecretManagerServiceClient(credentials=account, client_options=options_priv)
response_priv = sm_client_priv.access_secret_version(request=sm_version_request)

It generates the error:

E1003 09:09:35.205845398 1521508 hpack_parser.cc:999]                  Error parsing 'content-type' metadata: invalid value
*** google.api_core.exceptions.Unknown: None Stream removed

[alice-rc]

$ pip freeze
cachetools==5.3.1
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.3.0
cryptography==41.0.4
google-api-core==2.12.0
google-auth==2.23.2
google-cloud-secret-manager==2.16.4
google-cloud-vision==3.4.4
google-crc32c==1.5.0
googleapis-common-protos==1.60.0
grpc-google-iam-v1==0.12.6
grpcio==1.59.0
grpcio-status==1.59.0
idna==3.4
importlib-resources==5.0.7
Jinja2==3.1.2
MarkupSafe==2.1.3
packaging==23.2
proto-plus==1.22.3
protobuf==4.24.3
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
PyYAML==6.0.1
requests==2.31.0
resolvelib==1.0.1
rsa==4.9
urllib3==2.0.6

$ nslookup private.googleapis.com
Server:     X.X.12.211
Address:    X.X.12.211#53

Non-authoritative answer:
Name:   private.googleapis.com
Address: X.X.153.10
Name:   private.googleapis.com
Address: X.X.153.8
Name:   private.googleapis.com
Address: X.X.153.11
Name:   private.googleapis.com
Address: X.X.153.9

[parthea]

Hi @alice-rc, It's not clear about the reason that private.googleapis.com is used instead of the API hostname secretmanager.googleapis.com defined here. Could you please share more context? If you are a Googler, please file an issue internally so that we can dig into the specifics of this issue.

[alice-rc]

So this is extremely frustrating. I did log a ticket with Google Support and I keep getting the answer from them "Note that we don’t provide code-level support since it is outside the scope of Google Cloud Platform support based on our Technical Support Services Guidelines [1]. So there they tell me 'Not us' and here you tell me 'use the guys that say it's not them'...

The reason I need to use the private.googleapis.com endpoint is the systems are air-gapped with no public internet access. We do have the private endpoint setup as can be seen from the provided nslookup. Other teams within the company have used that endpoint (either just private.googleapis.com or private.googleapis.com:443) with both Java and Go implementations, it seem that only the python library implementation doesn't work when setting the api_endpoint to private.googleapis.com.

[parthea]

I'm working on getting more information to help debug the issue (Googlers see b/303342495)

[alice-rc]

Just a bit more information. If, for experimentation purposes, I setup /etc/hosts on a test server and add:

X.X.153.9    secretmanager.googleapis.com
X.X.153.9    private.googleapis.com

And then do the API calls without setting ClientOptions(api_endpoint="private.googleapis.com") everything works as expected. But If I set ClientOptions(api_endpoint="private.googleapis.com") it does not work.

To me that means that there is nothing actually wrong with the endpoint services at X.X.153.9 which work when the API call thinks that it is the default endpoint, but then has an issue using an alternate endpoint when api_endpoint is set to private.googleapis.com.

[parthea]

Other teams within the company have used that endpoint (either just private.googleapis.com or private.googleapis.com:443) with both Java and Go implementations, it seem that only the python library implementation doesn't work when setting the api_endpoint to private.googleapis.com.

Please can you share a snippet of the code used for Java and Go which are working ?

[alice-rc]

Here is a Java example:

package com.ringcentral.ops.gcsmdemo;

import com.google.api.gax.core.CredentialsProvider;
import com.google.auth.oauth2.ServiceAccountCredentials;
import com.google.cloud.secretmanager.v1.AccessSecretVersionResponse;
import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
import com.google.cloud.secretmanager.v1.SecretManagerServiceSettings;
import com.google.cloud.secretmanager.v1.SecretVersionName;
import com.google.cloud.secretmanager.v1.stub.SecretManagerServiceStubSettings;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import java.io.FileInputStream;
import java.io.IOException;

@Slf4j
@SpringBootApplication
public class GcsmDemoApplication implements CommandLineRunner {

    public static void main(String[] args) {
        SpringApplication.run(GcsmDemoApplication.class, args);
    }

    @Override
    public void run(String... args) throws Exception {
        log.info("access secret version ...");

        String credentialsFileName = args[0];
        String endpoint = args[1];
        String projectId = args[2];
        String secret = args[3];
        String version = args[4];

        CredentialsProvider serviceAccountCredentials = () -> ServiceAccountCredentials.fromStream(new FileInputStream(credentialsFileName));
        SecretManagerServiceSettings settings = getSecretManagerServiceSettings(serviceAccountCredentials, endpoint);

        try (SecretManagerServiceClient client = SecretManagerServiceClient.create(settings)) {
            AccessSecretVersionResponse accessSecretVersionResponse = client.accessSecretVersion(SecretVersionName.format(projectId, secret, version));
            log.info("secret = {}", accessSecretVersionResponse.getPayload().getData().toStringUtf8());
        }
    }

    private static SecretManagerServiceSettings getSecretManagerServiceSettings(CredentialsProvider serviceAccountCredentials, String endpoint) throws IOException {
        SecretManagerServiceSettings.Builder builder = SecretManagerServiceSettings.newBuilder();
        builder.setCredentialsProvider(serviceAccountCredentials);
        builder.setEndpoint(endpoint);
        builder.setTransportChannelProvider(SecretManagerServiceSettings.defaultGrpcTransportProviderBuilder()
                .setChannelConfigurator(input -> input.overrideAuthority(SecretManagerServiceStubSettings.getDefaultEndpoint()))
                .build());
        return builder.build();
    }
}

args[1] (endpoint) is set to "private.googleapis.com:443"

[alice-rc]

Here is a Go example:

package main

import (
    secretmanager "cloud.google.com/go/secretmanager/apiv1"
    "cloud.google.com/go/secretmanager/apiv1/secretmanagerpb"
    "context"
    "flag"
    "fmt"
    "golang.org/x/oauth2"
    "golang.org/x/oauth2/google"
    "google.golang.org/api/option"
    "google.golang.org/grpc"
    "log"
    "net"
    "net/http"
    "os"
    "os/signal"
    "time"
)

func getPrivateEndpointOptions(configFile string) ([]option.ClientOption, error) {
    if err := os.Setenv("GOOGLE_APPLICATION_CREDENTIALS", configFile); err != nil {
        return nil, fmt.Errorf("set env GOOGLE_APPLICATION_CREDENTIALS: %w", err)
    }
    options := make([]option.ClientOption, 0)
    dialContextPrivateAPI := func(ctx context.Context, network, address string) (net.Conn, error) {
        dialer := &net.Dialer{
            Timeout:   30 * time.Second,
            KeepAlive: 30 * time.Second,
            DualStack: true,
        }
        return dialer.DialContext(ctx, network, "private.googleapis.com:443")
    }

    const oauth2scope = "https://www.googleapis.com/auth/cloud-platform"

    gRPCDialFunc := func(ctx context.Context, address string) (net.Conn, error) {
        return dialContextPrivateAPI(ctx, "tcp", address)
    }
    options = append(options, option.WithGRPCDialOption(grpc.WithContextDialer(gRPCDialFunc)))
    oauth2httpTransport := http.DefaultTransport.(*http.Transport)
    oauth2httpTransport.DialContext = dialContextPrivateAPI
    oauth2httpClient := &http.Client{Transport: oauth2httpTransport, Timeout: 30 * time.Second}
    oauth2ctx := context.WithValue(context.Background(), oauth2.HTTPClient, oauth2httpClient)
    oauth2tokenSource, err := google.DefaultTokenSource(oauth2ctx, oauth2scope)
    if err != nil {
        return nil, fmt.Errorf("set DefaultTokenSource: %w", err)
    }

    options = append(options, option.WithTokenSource(oauth2tokenSource))
    return options, nil
}

func main() {
    var credentials string
    var private bool
    var project string
    var secret string
    var version string

    flag.StringVar(&credentials, "credentials", "", "path to credentials file")
    flag.BoolVar(&private, "private", false, "use private api endpoint")
    flag.StringVar(&project, "project", "", "project id")
    flag.StringVar(&secret, "secret", "", "secret id")
    flag.StringVar(&version, "version", "latest", "version id")
    flag.Parse()

    if credentials == "" || project == "" || secret == "" || version == "" {
        flag.PrintDefaults()
        return
    }

    ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
    defer cancel()
    termC := make(chan os.Signal, 1)
    go func() {
        select {
        case <-termC:
            log.Println("terminated")
            cancel()
        case <-ctx.Done():
            log.Println("timeout")
        }
    }()
    signal.Notify(termC, os.Kill, os.Interrupt)

    options := make([]option.ClientOption, 0)

    if private {
        endpointOptions, err := getPrivateEndpointOptions(credentials)
        if err != nil {
            log.Printf("failed to get endpoint options: %v", err)
            return
        }
        options = append(options, endpointOptions...)
    } else {
        options = append(options, option.WithCredentialsFile(credentials))
    }

    client, err := secretmanager.NewClient(ctx, options...)
    if err != nil {
        log.Printf("failed to setup client: %v", err)
        return
    }
    defer func(client *secretmanager.Client) {
        if cerr := client.Close(); cerr != nil {
            log.Printf("failed to close client: %v", cerr)
        }
    }(client)

    req := &secretmanagerpb.AccessSecretVersionRequest{
        Name: fmt.Sprintf("projects/%s/secrets/%s/versions/%s", project, secret, version),
    }
    resp, err := client.AccessSecretVersion(ctx, req)
    if err != nil {
        log.Printf("failed to access secret version: %v\n", err)
    } else {
        log.Printf("secret = %s\n", string(resp.Payload.Data))
    }
}

[parthea]

Thanks @alice-rc. ~Please can you also share the runnable python snippet that fails?~ please disregard this request for another snippet. The issue appears to be in grpc. See the linked bugs below.

[parthea]

Possibly related : https://github.com/grpc/grpc/issues/29706

[parthea]

Also see suggestions in https://stackoverflow.com/a/74992074

[parthea]

Also see https://github.com/grpc/grpc/issues/34076

[alice-rc]

Since I am not getting the exact same error message, and you have referenced this ticket in the others, I'll provide the output of my run with debug logging on just in the off chance it will help get this diagnosed and fixed.

Private endpoint: True
Version id: latest
E1020 15:09:59.114208589 1747587 hpack_parser.cc:999]                  Error parsing 'content-type' metadata: invalid value
Traceback (most recent call last):
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "<REDACTED>/lib64/python3.9/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "<REDACTED>/lib64/python3.9/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Stream removed"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-10-20T15:09:59.11433511-07:00", grpc_status:2, grpc_message:"Stream removed"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<REDACTED>/gcsm_secret_test.py", line 108, in <module>
    main()
  File "<REDACTED>/gcsm_secret_test.py", line 85, in main
    access_version = sm_client.access_secret_version(request={'name': secret_version_path})
  File "<REDACTED>/lib64/python3.9/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1518, in access_secret_version
    response = rpc(
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/retry.py", line 366, in retry_wrapped_func
    return retry_target(
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/retry.py", line 204, in retry_target
    return target()
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.Unknown: None Stream removed

[alice-rc]

Since this is a blocker on a major initiative to move to Google Cloud Secret Manager, can you tell me what versions of the google libraries I can use which do NOT have this issue?

[parthea]

Since this is a blocker on a major initiative to move to Google Cloud Secret Manager, can you tell me what versions of the google libraries I can use which do NOT have this issue?

The fix for this issue is outside of the client library code in this repository. The error Error parsing 'content-type' metadata: invalid value I believe is the same as the one in https://github.com/grpc/grpc/issues/34076 and https://github.com/grpc/grpc/issues/29706.

Please can you check if this comment helps? https://github.com/grpc/grpc/issues/29706#issuecomment-1741840899

In the meantime, there is a comment from the grpc team here which mentions that they are working on a fix in parallel.

We will keep this issue open, but the issue must either be fixed by using one of the following solutions

• https://github.com/grpc/grpc/issues/29706#issuecomment-1741840899
• https://github.com/grpc/grpc/issues/34076#issuecomment-1758482912
• More debugging via the suggestions in https://stackoverflow.com/questions/74984332/error-parsing-metadata-using-python-datacatalog/74992074#74992074

[parthea]

Please could you also try using REST instead of GRPC in case there is a more helpful error message that isn't surfaced when using GRPC? You can set the transport argument of SecretManagerServiceClient.

https://github.com/googleapis/google-cloud-python/blob/2d1eb364224c54bd1ec3b294de87387e21eff18f/packages/google-cloud-secret-manager/google/cloud/secretmanager_v1/services/secret_manager_service/client.py#L409-L411

from google.cloud import secretmanager_v1 as secretmanager
from google.api_core.client_options import ClientOptions
from google.oauth2 import service_account

...
options_priv = ClientOptions(api_endpoint="private.googleapis.com")
sm_client_priv = secretmanager.SecretManagerServiceClient(credentials=account, client_options=options_priv, transport="rest")
response_priv = sm_client_priv.access_secret_version(request=sm_version_request)

[alice-rc]

Updated the code to use secretmanager.SecretManagerServiceClient(credentials=sm_account, client_options=sm_client_options, transport="rest")

Output:

Private endpoint: True
2023-10-23 13:44:00,634 [DEBUG] Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
Version id: latest
2023-10-23 13:44:00,640 [DEBUG] Starting new HTTPS connection (1): private.googleapis.com:443
2023-10-23 13:44:00,971 [DEBUG] https://private.googleapis.com:443 "GET /v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest:access?%24alt=json%3Benum-encoding%3Dint HTTP/1.1" 404 1644
Traceback (most recent call last):
  File "<REDACTED>/gcsm_secret_test.py", line 108, in <module>
    main()
  File "<REDACTED>/gcsm_secret_test.py", line 85, in main
    access_version = sm_client.access_secret_version(request={'name': secret_version_path})
  File "<REDACTED>/lib64/python3.9/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1518, in access_secret_version
    response = rpc(
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/retry.py", line 366, in retry_wrapped_func
    return retry_target(
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/retry.py", line 204, in retry_target
    return target()
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "<REDACTED>/lib64/python3.9/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "<REDACTED>/lib64/python3.9/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/transports/rest.py", line 690, in __call__
    raise core_exceptions.from_http_response(response)
google.api_core.exceptions.NotFound: 404 GET https://private.googleapis.com:443/v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest:access?%24alt=json%3Benum-encoding%3Dint: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:ur
l(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest:access</code> was not found on this server.  <ins>That’s all we know.</ins>

With secretmanager.SecretManagerServiceClient(credentials=sm_account, transport="rest") Output:

Private endpoint: False
2023-10-23 13:50:46,535 [DEBUG] Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
Version id: latest
2023-10-23 13:50:46,540 [DEBUG] Starting new HTTPS connection (1): secretmanager.googleapis.com:443
2023-10-23 13:50:46,839 [DEBUG] https://secretmanager.googleapis.com:443 "GET /v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest:access?%24alt=json%3Benum-encoding%3Dint HTTP/1.1" 200 None
2023-10-23 13:50:46,963 [DEBUG] https://secretmanager.googleapis.com:443 "GET /v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest?%24alt=json%3Benum-encoding%3Dint HTTP/1.1" 200 None

{"action": "retrieve", "changed": false, "create_time": "2023-08-21T18:19:37.149381+00:00", "name": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "replication_status": "automatic", "version": "1", "secret": "<REDACTED>", "invocation": {"module_args": {"credentials": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "project_id": "<REDACTED>", "secret_id": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "action": "retrieve", "private_endpoint": false, "version_id": "latest", "value": null}}}

Both secretmanager.googleapis.com and private.googleapis.com are set to the same IP on this dev system:

$ tail -2 /etc/hosts
X.X.153.9    secretmanager.googleapis.com
X.X.153.9    private.googleapis.com

[parthea]

Using transport="rest" shows the actual error which is google.api_core.exceptions.NotFound: 404 GET https://private.googleapis.com:443/v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest

You can see from the error message that the client library is attempting to use the endpoint which is configured. This is not an issue with the client library but rather the configuration on the dev system or local infrastructure.

Did you receive documentation from Google on how to configure the API on the dev system? If so, please reach out to whoever sent the documentation to double check the configuration, as my team doesn't maintain the docs/support for running APIs on dev systems.

[alice-rc]

I'm utterly confused. If I take a step back from any particular code base and just try to use the curl command it appears that the endpoint at X.X.153.9 recognizes a request that uses secretmanager.googleapis.com as the URL but nothing else, despite all of them resolving to the same IP address.

$ curl "https://secretmanager.googleapis.com:443/v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest"
{
  "error": {
    "code": 401,
    "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
    "status": "UNAUTHENTICATED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "CREDENTIALS_MISSING",
        "domain": "googleapis.com",
        "metadata": {
          "method": "google.cloud.secretmanager.v1.SecretManagerService.GetSecretVersion",
          "service": "secretmanager.googleapis.com"
        }
      }
    ]
  }
}

But is I use any other hostname, mapped to that same IP I get:

$ curl "https://private.googleapis.com:443/v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest"
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/v1/projects/<REDACTED>/secrets/<REDACTED>/versions/latest</code> was not found on this server.  <ins>That’s all we know.</ins>

But, what is truly baffling is that the go code that I provided above works on the very same dev server without issues using the private endpoint. No changes to /etc/hosts, no special configuration, it just works.

I'm not a go developer so I'm unsure of exactly what it is doing but it is clearly doing something internally in creating the request that the python code is not. I could likely setup java on this same server and it too would work, given the feedback from other colleagues. So given that the other code bases work, without doing some specific 'configuration on the dev system or local infrastructure' I'm unsure as to how to proceed. I did not receive documentation from Google on how to configure the API on ANY system. Who would provide that? Why would python require that when other code bases do not?

[parthea]

I checked with another team member regarding the golang code and they mentioned that the WithEndpoint option needs to be set to change the API endpoint. We believe that the golang code is not using private.googleapis.com. To use a different endpoint in golang , you'll need to pass the WithEndpoint option before calling secretmanager.NewClient(ctx, options...).

Use

option.WithEndpoint("private.googleapis.com")

https://pkg.go.dev/google.golang.org/api/option#WithEndpoint

[alice-rc]

I don't think that is correct. I just tested the following: Comment out the secretmanager.googleapis.com line in /etc/hosts. Run the go code without setting the private endpoint:

$ go run main.go -credentials credentials.json -project <REDACTED> -secret <REDACTED>
2023/10/24 11:59:00 timeout
2023/10/24 11:59:00 failed to access secret version: context deadline exceeded

That result is expected as this system does not have internet access.

Run the code with the private endpoint:

$ go run main.go -credentials credentials.json -project <REDACTED> -secret <REDACTED> -private true
2023/10/24 12:00:45 secret = <REDACTED>

[alice-rc]

Any updates on when this might get resolved?

[parthea]

I've reached out to colleague who is more familiar with the authentication library to look into this further. I will update this issue once I hear back.

[parthea]

Hi @alice-rc ,

Please can you share more information about the setup in your development environment with instructions so that I can reproduce the issue? If you'd prefer to send this information in private, please send the information via Google Support and reference this issue.

[alice-rc]

I'm not sure what information you need, as I've provided the pip freeze and the nslookup above. If you have specific questions please ask. General info Python version 3.9.13 go version go1.19.13 linux/amd64

[alice-rc]

Any progress here at all? Do you still need environment info from me?

[parthea]

Hi @alice-rc,

I don't believe this is a configuration that we support. Please see the following documentation on configuring on-premises systems to connect to Google APIs and services using a Virtual Private Cloud: https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid

There is a public issue tracker for Virtual Private Cloud here where you can ask specific questions about setting up a Virtual Private Cloud . There is also a codelab to help with this specific configuration: https://codelabs.developers.google.com/codelabs/cloudnet-psc?hl=en#0

[ygazaryan]

Hi, @parthea @alice-rc

I have the same issue in python lib:

import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/tmp/gcp/sample.json'

from google.cloud import secretmanager_v1
from google.api_core.client_options import ClientOptions
json_key_path = '/tmp/gcp/ops-rc-dbaops-e583aa5f7d91.json'
secret_id = 'test'
secret_name = f'projects/***/secrets/{secret_id}/versions/1'
client_options = ClientOptions(api_endpoint='private.googleapis.com')
client2 = secretmanager_v1.SecretManagerServiceClient(client_options=client_options).from_service_account_file(json_key_path)
response2 = client2.access_secret_version(request={"name": secret_name})

Trace info

Traceback (most recent call last):
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/grpc_helpers.py", line 50, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses; last error: UNKNOWN: tcp handshaker shutdown"
    debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-11-08T07:16:48.672716028+00:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: tcp handshaker shutdown {created_time:"2023-11-08T07:16:48.672704349+00:00", grpc_status:14}]}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/grpc_helpers.py", line 52, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses; last error: UNKNOWN: tcp handshaker shutdown

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1443, in access_secret_version
    metadata=metadata,
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func
    on_error=on_error,
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/retry.py", line 210, in retry_target
    ) from last_exc
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 failed to connect to all addresses; last error: UNKNOWN: tcp handshaker shutdown

Try different option:

import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/tmp/gcp/sample.json'
.....
client_options = ClientOptions(api_endpoint='private.googleapis.com')
client2 = secretmanager_v1.SecretManagerServiceClient(client_options=client_options)
response2 = client2.access_secret_version(request={"name": secret_name})

E1108 07:14:08.811693473   28066 hpack_parser.cc:1235]       Error parsing metadata: error=invalid value key=content-type value=text/html; charset=UTF-8
Traceback (most recent call last):
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/grpc_helpers.py", line 50, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Stream removed"
    debug_error_string = "UNKNOWN:Error received from peer ipv4:199.36.153.10:443 {created_time:"2023-11-08T07:14:08.811936421+00:00", grpc_status:2, grpc_message:"Stream removed"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1443, in access_secret_version
    metadata=metadata,
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func
    on_error=on_error,
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/tmp/gcp/venv/lib64/python3.6/site-packages/google/api_core/grpc_helpers.py", line 52, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.Unknown: None Stream removed

Network access to private.googleapis.com works fine:

telnet private.googleapis.com 443
Trying 199.36.153.9...
Connected to private.googleapis.com.
Escape character is '^]'.

telnet 199.36.153.10 443
Trying 199.36.153.10...
Connected to 199.36.153.10.
Escape character is '^]'.
^]

In Go the same sample code works fine.

[parthea]

Hi @ygazaryan,

As mentioned in https://github.com/googleapis/google-cloud-python/issues/11781#issuecomment-1800911683, please follow the steps to use Virtual Private Cloud where we have documentation for this use case, as well as a public issue tracker for Virtual Private Cloud.

[parthea]

I'm going to close this issue as there is already a public issue tracker for Virtual Private Cloud here for this use case. Virtual Private Cloud is the recommended way to configure Private Google Access for on-premises hosts.

[alice-rc]

So after 6 weeks for this issue it is determined that this is not something that is supported? What is the api_endpoint option for then?

[parthea]

https://github.com/googleapis/google-cloud-python/issues/11781#issuecomment-1800911683 has guidance on the recommended next steps which is to use Virtual Private Cloud which is supported via the issue tracker.

What is the api_endpoint option for then?

In some APIs a regional endpoint is required. In that case the api_endpoint argument of ClientOptions needs to be set to use a regional endpoint. For example, this is the case for Speech To Text V2: https://cloud.google.com/speech-to-text/v2/docs/migration#migrating_in_api

    # Instantiates a client to a regionalized Speech endpoint.
    client = SpeechClient(
        client_options=ClientOptions(
            api_endpoint=f"{location}-speech.googleapis.com",
        )
    )

[alice-rc]

You do realize, I assume, that the documentation that you directed me to at https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid specifically states that "You must direct Google APIs and services traffic sent by on-premises systems to the IP addresses associated with either the private.googleapis.com or the restricted.googleapis.com special domain names." Which is exactly what we did and is the subject of this issue.

[parthea]

If you have followed all of the steps in https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid for Virtual Private Cloud, and it's still not working, please file an issue in the public issue tracker for the Virtual Private Cloud product. There is also a codelab to which you can work through for the configuration: https://codelabs.developers.google.com/codelabs/cloudnet-psc?hl=en#0