search

Azure / feast-azure

Azure plugins for Feast (FEAture STore)
MIT License
82 stars 52 forks source link

Conda Environment Creation Fails for Deployment #12

Closed tarockey closed 2 years ago

tarockey commented 2 years ago

Summary: When attempting to create the conda environment "feast-env" as found in the training and deployment notebook, the environment creation stalls out after an hour during "installing pip dependencies" and never completes.

Details: First occurrence of this issue was during attempts to deploy a scoring container, which timed out after an hour. Looking at the logs, it simply stops at the line "Installing pip dependencies: ... working ...", with no further detail.

Attempting to build the conda environment from the "model_service_env.yml" in a local terminal fail in the same spot. Manually removing the pip packages from the yml file and manually installing those packages in the conda environment does not fail. Also, removing just the "feast-azure-provider" package causes to build to succeed, and it can then be manually installed. However, since Azure ML does not allow the pip installs to be done in separate steps, this is not a viable workaround for the container deployment.

samuel100 commented 2 years ago

@tarockey thanks for raising! There appears to be a bug in conda - we are looking into a workaround.

samuel100 commented 2 years ago

@tarockey this has now been fixed. We changed the deployment to be a custom docker build - feast has dependencies on gcc and unixodbc-dev that needed to be installed on the docker image. The bug in Conda caused pip to hang when it hit the install error.