[ISSUE] Conflict dependencies when using snowflake-ml-python with others `fsspec` protocols

[nct74]

My system is running with these libraries in my .toml that supports read/write files, based on fsspec in Python. fsspec_dnanexus = ">=0.2.3" s3fs = "== 2024.3.1" adlfs = "== 2024.4.0" xgboost = ">= 1.7.6" python = ">= 3.9,< 3.10"

Issue: It raises under errors when I add snowflake-ml-python = ">= 1.5.3" in my.toml file; I need to import from snowflake.ml.fileset import sfcfs, following this document (https://docs.snowflake.com/en/developer-guide/snowpark-ml/reference/latest/api/fileset/snowflake.ml.fileset.sfcfs.SFFileSystem) I have tried to downgrade/upgrade those modules (s3fs, adlfs, fsspec_dnanexus) to get compatible versions, but it has not worked.

I would appreciate your letting me know which version I should add to my system.

[nct74]

Hi @sfc-gh-sdas, @sfc-gh-wzhao, I appreciate it if you could help me check on my issue. Many thanks for considering my request.

[sfc-gh-sdas]

Sorry for late response.

I presume I am missing some variables like, which OS, etc?

Here is what I tried on Mac:

$ python3.9 -m venv test-109
$ source test-109/bin/activate
(test-109) $ pip install "snowflake-ml-python>= 1.5.3" fsspec_dnanexus s3fs adlfs "xgboost>= 1.7.6"
(test-109) $ pip list
Package                    Version
-------------------------- -----------
...
adlfs                      2023.8.0
fsspec                     2023.4.0
fsspec-dnanexus            0.1.0
s3fs                       2023.4.0
xgboost                    1.7.6
...

(test-109) $ python
Python 3.9.13 (v3.9.13:6de2ca5339, May 17 2022, 11:37:23) 
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from snowflake.ml.fileset import sfcfs
>>> from snowflake.ml.version import VERSION
>>> print(VERSION)
1.5.4

I might be missing something.

[nct74]

Hi @sfc-gh-sdas, thank you for your response, I can install it, but the version is very down (e.g. fsspec_dnanexus from 0.2.3 to 0.1.0). I would appreciate it if we could have a new version snowflake-ml-python that can be compatible with newer packages. Ref: https://pypi.org/project/fsspec-dnanexus/#history

[nct74]

Hello, @sfc-gh-sdas. I'd appreciate it if you could keep me informed about the plans for new versions that address my issue. We created a lot of features that rely on the 'fsspec' libraries, and I could not downgrade to fit with `snowflake-ml-python', which may have possible issues when we do that.