proxy-agent `No valid URLs found for zone: southamerica-east1-c`

[andreclaudino]

What steps did you take:

I deployed kubeflow-pipelines standalone in private gke following documentation.

What happened:

The pod proxy-agents keeps CrashLoopBackOff, getting logs, I can see the message ValueError: No valid URLs found for zone: southamerica-east1-c

What did you expect to happen:

The pod should run normaly

Environment:

How did you deploy Kubeflow Pipelines (KFP)? I deployed version 0.5.1 following instructions in https://www.kubeflow.org/docs/pipelines/installation/standalone-deployment/

KFP version: 0.5.1 (9c16e12)

KFP SDK version:

kfp                               0.5.1
kfp-server-api                    0.3.0

Anything else you would like to add:

This is the complete log:

+++ dirname /opt/proxy/attempt-register-vm-on-proxy.sh
++ cd /opt/proxy
++ pwd
+ DIR=/opt/proxy
+ [[ ! -z '' ]]
++ curl http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    48  100    48    0     0   8692      0 --:--:-- --:--:-- --:--:--  9600
+ INSTANCE_ZONE=/projects/474790423518/zones/southamerica-east1-c
+ INSTANCE_ZONE=southamerica-east1-c
+ wget https://storage.googleapis.com/ml-pipeline/proxy-agent-config.json
--2020-05-20 14:28:48--  https://storage.googleapis.com/ml-pipeline/proxy-agent-config.json
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.30.48, 2800:3f0:4001:802::2010
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.30.48|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1820 (1.8K) [application/json]
Saving to: 'proxy-agent-config.json'

     0K .                                                     100% 23.2M=0s

2020-05-20 14:28:48 (23.2 MB/s) - 'proxy-agent-config.json' saved [1820/1820]

++ python /opt/proxy/get_proxy_url.py --config-file-path proxy-agent-config.json --location southamerica-east1-c --version latest
Traceback (most recent call last):
  File "/opt/proxy/get_proxy_url.py", line 111, in <module>
    main()
  File "/opt/proxy/get_proxy_url.py", line 93, in main
    raise ValueError("No valid URLs found for zone: {}".format(location))
ValueError: No valid URLs found for zone: southamerica-east1-c
+ PROXY_URL=

In fact, the file https://storage.googleapis.com/ml-pipeline/proxy-agent-config.json doesn't have an entry for southamerica-east1-c, this looks the problem.

/kind bug // /area frontend /area backend // /area sdk // /area testing // /area engprod -->

[numerology]

Ah yes... It's likely that inverse proxy does not support South America, yet.

Is it a necessity to deploy clusters over there for you? If so we'll file a FR to inverse proxy team.

[numerology]

@andreclaudino Below is the currently supported regions. @rmgogogo I think we need to document that in the website as well.

us-central1
us-central2
us-west1
us-east1
us-east4
northamerica-northeast1
europe-west1
europe-west2
europe-west6
asia-southeast1
asia-northeast1
asia-east1
australia-southeast1

[rmgogogo]

+1 for document it in website. @joeliedtke

I will also send out an email to proxy team on whether flexible to add more region.

[andreclaudino]

Ah yes... It's likely that inverse proxy does not support South America, yet.

Is it a necessity to deploy clusters over there for you? If so we'll file a FR to inverse proxy team.

Thanks for answering. Yes, we need to use this in South America. We are using Kubeflow in a production cluster with many other applications from our company, it's impossible for us to change the cluster region only to support kubeflow.

What do we needed to support other regions? I look at the file, and there is an standard in filenames, maybe we can just create these entries automatically for all gcp regions.

[numerology]

What do we needed to support other regions

Unfortunately I don't think there is an easy way for our customers to make changes to support more regions. Inverse proxy is a managed service hosted as a Google managed service, the mapping in https://storage.googleapis.com/ml-pipeline/proxy-agent-config.json is the list of currently supported GCE regions.

We've filed a FR to inverse proxy team to expand their supported regions. Meanwhile, to use KFP on an arbitrary region, I think one workaround is to deploy a full-fledged Kubeflow, see this guide. KFP is one component of it.

@rmgogogo do we have any other option?

[andreclaudino]

What do we needed to support other regions

Unfortunately I don't think there is an easy way for our customers to make changes to support more regions. Inverse proxy is a managed service hosted as a Google managed service, the mapping in https://storage.googleapis.com/ml-pipeline/proxy-agent-config.json is the list of currently supported GCE regions.

We've filed a FR to inverse proxy team to expand their supported regions. Meanwhile, to use KFP on an arbitrary region, I think one workaround is to deploy a full-fledged Kubeflow, see this guide. KFP is one component of it.

@rmgogogo do we have any other option?

@numerology, the problem persists if deploying entire kubeflow. In fact, I had many other problems installing kubeflow, then decide use only kfp.

[rmgogogo]

No cheap option for now. Reply from proxy team is that support a new region requires 2-4 weeks (didn't commit to add southameraca-east1 yet but we can help request if no other options).

To get more background, any reason why must use southameraca-east1? is it for some legit consideration?

[numerology]

@rmgogogo I think @andreclaudino answered a similar question above

Thanks for answering. Yes, we need to use this in South America. We are using Kubeflow in a production cluster with many other applications from our company, it's impossible for us to change the cluster region only to support kubeflow.

[andreclaudino]

No cheap option for now. Reply from proxy team is that support a new region requires 2-4 weeks (didn't commit to add southameraca-east1 yet but we can help request if no other options).

To get more background, any reason why must use southameraca-east1? is it for some legit consideration?

As @numerology already said, I have answered your question:

We are using Kubeflow in a production cluster with many other applications from our company, it's impossible for us to change the cluster region only to support kubeflow.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.