something went wrong

[MathieuGrosso]

I have followed all the tutorial, was able to log on the mlflow server but then i get this message "something went wrong" on the uI interface. Do you know why ? In logs i have an error of engine, the password is seen as the port. I have followed the same path as you provide for engine but it does not seem to work.

[mkarpicz]

Hey Mathieu,

Thanks for reporting the issue. Soon we will release a new version of this repository. Meanwhile, @Barabara could you check this?

All the best, Maciej

On Fri, Jun 30, 2023 at 7:56 PM MathieuGrosso @.***> wrote:

I have followed all the tutorial, was able to log on the mlflow server but then i get this message "something went wrong" on the uI interface. Do you know why ? In console i have these messages:

FetchUtils.js:82 Fetch failed: Response m @ FetchUtils.js:82 RequestStateWrapper.js:93 ERROR Array(1)0: {id: '1688147721577mw1d3mv2', active: false, error: o}length: 1[[Prototype]]: Array(0) u @ RequestStateWrapper.js:93 react-dom.production.min.js:216 Error: A request error occurred. at u (RequestStateWrapper.js:94:9) at c.getRenderedContent (RequestStateWrapper.js:77:9) at c.render (RequestStateWrapper.js:87:17) at Wa (react-dom.production.min.js:187:188) at qa (react-dom.production.min.js:186:173) at Ks (react-dom.production.min.js:269:427) at Pl (react-dom.production.min.js:250:347) at Tl (react-dom.production.min.js:250:278) at Al (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) us @ react-dom.production.min.js:216 AppErrorBoundary.js:28 Error: A request error occurred. at u (RequestStateWrapper.js:94:9) at c.getRenderedContent (RequestStateWrapper.js:77:9) at c.render (RequestStateWrapper.js:87:17) at Wa (react-dom.production.min.js:187:188) at qa (react-dom.production.min.js:186:173) at Ks (react-dom.production.min.js:269:427) at Pl (react-dom.production.min.js:250:347) at Tl (react-dom.production.min.js:250:278) at Al (react-dom.production.min.js:250:138) at bl (react-dom.production.min.js:243:163) Object componentDidCatch @ AppErrorBoundary.js:28 /#/:1 Uncaught (in promise) o manifest.json:1 Failed to load resource: the server responded with a status of 401 () manifest.json:1 Manifest: Line: 1, column: 1, Syntax error. manifest.json:1 Failed to load resource: the server re

— Reply to this email directly, view it on GitHub https://github.com/dlabsai/mlflow-for-gcp/issues/9, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELV7NNGCDV6VGOXR6VVAIDXN4HLJANCNFSM6AAAAAAZ2EQKMM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[basia-c]

Hello! Looks like your MlFlow is not able to connect to the database, there may be a few different reasons:

1. API is not activated, the following APIs need to be activated (it may take up to 24h):

   • https://console.developers.google.com/apis/api/secretmanager.googleapis.com
   • https://console.cloud.google.com/apis/library/containerregistry.googleapis.com
   • https://console.developers.google.com/apis/api/run.googleapis.com
   • https://console.developers.google.com/apis/api/sqladmin.googleapis.com

2. Database is down (it needs to be up and running)

3. Service account used by CloudRun does not have the right privileges - it needs to have access to the secrets created for this project (roles/secretmanager.secretAccessor) and to the database (roles/cloudsql.client)

4. If it still does not work at this point try updating requirements.txt file as follows :

google-cloud-storage==2.9.0 google-cloud-secret-manager==2.16.1 mlflow==2.4.1 click==8.1.3 pg8000==1.29.3 wsgi-basic-auth==1.1.0 gunicorn==20.1.0 psycopg2-binary==2.9.6

[BioGeek]

I had the same issue.

The tutorial writes that the mlflow_database_url Cloud SQL connection string must be in SQLAlchemy-format and gives as example:

postgresql+pg8000://<dbuser>:<dbpass>@/<dbname>?unix_sock=/cloudsql/tc-global-dlabs:europe-west3:mlfow/.s.PGSQL.5432 

After much head scratching I found this stackoverflow post that shows you need to use psycopg2 instead:

postgresql+psycopg2://<username>:<password>@<dbname>?host=/cloudsql/<my-project>:<us-central1>:<dbinstance>

Some other places where I deviated from the tutorial:

• Container Registry is deprecated, so I used Artifactory registry
• The get_secret.py script is not needed. You can expose secrets as environment variables in Cloud Run. So I modified the entry-point.sh script.

Also, it is unclear why I needed to download the client-cert.pem, client-key.pem and server-ca.pem files because the tutorial doesn't seem to use them anywhere.

[mkarpicz]

Hey Jeroen,

thanks so much! Currently I’m maintaining this repo and will update once back at the end of the next week.

All the best, Maciej

W dniu pt., 29.09.2023 o 03:19 Jeroen Van Goey @.***> napisał(a):

I had the same issue.

The tutorial writes that the mlflow_database_url Cloud SQL connection string must be in SQLAlchemy-format and gives as example:

postgresql+pg8000://dbuser>:<dbpass>@/<dbname?unix_sock=/cloudsql/tc-global-dlabs:europe-west3:mlfow/.s.PGSQL.5432

After much head scratching I found this stackoverflow post https://stackoverflow.com/questions/72411618/mlflow-and-postgres-getting-bad-request-error that shows you need to use psycopg2 instead:

postgresql+psycopg2://username>:<password>@<dbname?host=/cloudsql/::

Some other places where I deviated from the tutorial:

• Container Registry is deprecated, so I used Artifactory registry
• The get_secret.py script is not needed. You can expose secrets as environment variables in Cloud Run. So I modified the entry-point.sh script.

— Reply to this email directly, view it on GitHub https://github.com/dlabsai/mlflow-for-gcp/issues/9#issuecomment-1740034568, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELV7NJBDJKHIKL2ETOZP7TX4XUMTANCNFSM6AAAAAAZ2EQKMM . You are receiving this because you commented.Message ID: @.***>

[mz-dlabs]

Hello @BioGeek,

I've updated repository documentation with changes related to Artifact Registry etc.

If MLflow cannot connect to the database, try one of the following:

1. Make sure that in Cloud Run configuration, in Container(s), Volumes, Networking, Security accordion, Cloud SQL connections section, you added your database instance
2. Try using Cloud SQL without SSL
3. For the database URL, I use a string like the following - make sure yours is similar

   postgresql+pg8000://username:password@/database_name?unix_sock=/cloudsql/project_name:database_region:database_name/.s.PGSQL.5432

Let us know if you have successfully launched the application

Regards Maciej