search

snowflakedb / snowflake-ml-python

Apache License 2.0
43 stars 12 forks source link

Add option for custom conda channel for checking packages exist #125

Closed danielhstahl closed 2 weeks ago

danielhstahl commented 3 weeks ago

Currently, the Snowflake Conda channel is hardcoded. This is an issue in private/isolated environments that don't allow internet connectivity or when proxies such as JFrog are used.

This PR allows for an environment variable SNOWFLAKE_CONDA_CHANNEL_URL to be set for an alternative Conda channel such as a JFrog proxy.

Default behavior is unchanged.

sfc-gh-wzhao commented 2 weeks ago

Hi @danielhstahl , Thank you for your PR. However, this constant is used in many place other than get_matched_package_versions_in_snowflake_conda_channel function so they could not be changed. Furthermore, get_matched_package_versions_in_snowflake_conda_channel is designed as best-effort and non-blocking so it should not fail if that URL is not accessiable.

However, we are working on a change to replace get_matched_package_versions_in_snowflake_conda_channel with get_matched_package_versions_in_information_schema so that the package availablity check will always use information schema, making it easier to control.

danielhstahl commented 2 weeks ago

Hi @danielhstahl , Thank you for your PR. However, this constant is used in many place other than get_matched_package_versions_in_snowflake_conda_channel function so they could not be changed. Furthermore, get_matched_package_versions_in_snowflake_conda_channel is designed as best-effort and non-blocking so it should not fail if that URL is not accessiable.

However, we are working on a change to replace get_matched_package_versions_in_snowflake_conda_channel with get_matched_package_versions_in_information_schema so that the package availablity check will always use information schema, making it easier to control.

Thank you. The problem is that the function will hang for 1-2 hours, even though the error itself doesn't break or block the flow. I can PR a workaround which uses a flag to ignore this function.

danielhstahl commented 2 weeks ago

Opened a PR here: https://github.com/snowflakedb/snowflake-ml-python/pull/129