bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.88k stars 9.16k forks source link

[bitnami/mlflow] Chart does not map S3 host to tracking pod in MLFLow #27088

Closed rojo1997 closed 2 months ago

rojo1997 commented 3 months ago

Name and Version

bitnami/mlflow 1.4.4

What architecture are you using?

amd64

What steps will reproduce the bug?

I am using the GCP Kubernetes service and trying to store MLFlow artifacts in Google Storage.

Are you using any custom parameters or values?

tracking:
  enabled: true
  replicaCount: 1
  auth:
    enabled: true
    existingSecret: "mlflow"
    existingSecretUserKey: "MLFLOW_TRACKING_USERNAME"
    existingSecretPasswordKey: "MLFLOW_TRACKING_PASSWORD"
  service:
    type: ClusterIP
    ports:
      http: 80
  persistence:
    enabled: true
    mountPath: /bitnami/mlflow
    size: 8Gi
postgresql:
  enabled: true
  auth:
    username: bn_mlflow
    password: ""
    database: bitnami_mlflow
minio:
  enabled: false
externalS3:
  host: "storage.googleapis.com"
  port: 443
  existingSecret: "gcp-s3"
  existingSecretAccessKeyIDKey: "ACCESS_KEY"
  existingSecretKeySecretKey: "SECRET"
  protocol: "https"
  bucket: '{{ .Values | get "mlflow.tracking.externalS3.bucket" }}'
  serveArtifacts: true

What is the expected behavior?

MLFlow is able to save the artifacts in Google Storage.

What do you see instead?

MLFlow is not recognising the Google Storage host and is trying to connect to AWS S3.

Additional information

I managed to solve it by adding the following environment variable:

tracking:
  extraEnvVars:
    - name: MLFLOW_S3_ENDPOINT_URL
      value: "https://storage.googleapis.com"
javsalgar commented 3 months ago

Hi!

Thank you so much for reporting. Would you like to submit a PR fixing the issue?

angegar commented 3 months ago

Did you create a custom mlflow container with google-cloud-storage ?

I just tried what you proposed but it does not work with the out of the box mlflow container.

github-actions[bot] commented 2 months ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 2 months ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.