search

feast-dev / feast

The Open Source Feature Store for Machine Learning
https://feast.dev
Apache License 2.0
5.47k stars 977 forks source link

Environment variable not changing problem.. (FEAST_S3_ENDPOINT_URL) #1358

Closed devJackie closed 3 years ago

devJackie commented 3 years ago

Hello, I am a developer who wants to configure MLOps' feature store with Feast :)

Expected Behavior

Expect environment variables to be changed in k8s like local docker.

Current Behavior

Setting the local docker environment variable works correctly. os.environ["FEAST_S3_ENDPOINT_URL"] = "http://172.22.0.7:9000"

but, After deploying k8s, environment variable setting does not work properly. environment variables setting (jupyter - minimal_ride_hailing.ipynb) os.environ["FEAST_S3_ENDPOINT_URL"] = "http://minio.xxx-ml-dev.is.xxx-corp.com"

It is always set to "https://feast.s3.amazonaws.com", even though I change the setting. 스크린샷 2021-02-26 오후 5 24 16

Steps to reproduce

install Feast -> Kubernetes (with Helm) https://docs.feast.dev/getting-started/install-feast/kubernetes-with-helm helm version: 3.5.2 minio version: minio-8.0.10

Specifications

Any help would be greatly appreciated.

woop commented 3 years ago

@devJackie at what point in the notebook do you set the variable? The notebook has a cell at the start where env vars are printed out, is your S3_ENDPOINT_URL printed out?

devJackie commented 3 years ago

@woop Thanks for the quick answer :)

Yes, it is output to the notebook cell. (Refer to the image below)

(Edit with necessary environment variables) image

The environment variable was set automatically, but I just in case set the "FEAST_S3_ENDPOINT_URL" env again. image

It is always set to "https://feast.s3.amazonaws.com", even though I change the env setting(os.environ["FEAST_S3_ENDPOINT_URL"] = "http://minio.xxx-ml-dev.is.xxx-corp.com"). 😭

devJackie commented 3 years ago

@woop I think I solved the problem in an unusual way.

  1. Connect with kubectl kubectl exec -it -n feast feast-release-feast-jupyter-5b54cf46-r4xl7 -- /bin/bash

  2. env check (FEAST_S3_ENDPOINT_URL does not exist) (base) jovyan@feast-release-feast-jupyter-5b54cf46-r4xl7:~$ env

  3. add env setting (FEAST_S3_ENDPOINT_URL) I solved the problem when export to FEAST_S3_ENDPOINT_URL. (base) jovyan@feast-release-feast-jupyter-5b54cf46-r4xl7:~$ export FEAST_S3_ENDPOINT_URL=http://minio.xxx-ml-dev.is.xxxcorp.com image

  4. It was uploaded to minio. image

Is the correct way to export FEAST_S3_ENDPOINT_URL after connecting to the container(kubectl)? Are there any other alternatives?

devJackie commented 3 years ago

This seems to be resolved. I installed spark-on-k8s-operator with the namespace "feast" installed on top of k8s. 😊

woop commented 3 years ago

Thanks @devJackie!