[BUG]pyscenic version conflict when using feather v2 vs v1

Describe the bug

If I following the install procedure through conda create -n scenic_protocol python=3.6, I can only install pyscenic=0.11.2. Under python 3.6, I could not upgrade ctxcore to 2.0.0 and pyarrow >= 8.0.0. Then, I could not use the feather file v2. It would lead to the following error when try the following error.

pyscenic ctx adj.csv mm10_10kbp_up_10kbp_down_full_tx_v10_clust.genes_vs_motifs.rankings.feather mm10_500bp_up_100bp_down_full_tx_v10_clust.genes_vs_motifs.rankings.feather --annotations_fname motifs-v10nr_
clust-nr.mgi-m0.001-o0.0.tbl --output reg.csv --mask_dropouts --num_workers 10 --mode dask_multiprocessing --chunk_size 100

/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/numba/np/ufunc/parallel.py:365: NumbaWarning: The TBB threading layer requires TBB version 2019.5 or later i.e., TBB_INTERFACE_VERSION >= 11005. Found TBB_INTERFACE_VERSION = 10002. The TBB threading layer is disabled.
  warnings.warn(problem)
Traceback (most recent call last):
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/bin/pyscenic", line 8, in <module>
    sys.exit(main())
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 675, in main
    args.func(args)
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 230, in prune_targets_command
    num_workers=args.num_workers,
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/prune.py", line 410, in prune2df
    module_chunksize,
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/prune.py", line 334, in _distributed_calc
    scheduler='processes', num_workers=num_workers if num_workers else cpu_count()
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/base.py", line 283, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/base.py", line 565, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/multiprocessing.py", line 230, in get
    **kwargs
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/local.py", line 487, in get_async
    raise_exception(exc, tb)
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/local.py", line 316, in reraise
    raise exc.with_traceback(tb)
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/transform.py", line 301, in modules2df
    [module2df(db, module, motif_annotations, weighted_recovery, False, module2features_func) for module in modules]
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/transform.py", line 301, in <listcomp>
    [module2df(db, module, motif_annotations, weighted_recovery, False, module2features_func) for module in modules]
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/transform.py", line 231, in module2df
    db, module, motif_annotations, weighted_recovery=weighted_recovery
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/transform.py", line 152, in module2features_auc1st_impl
    df = db.load(module)
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/ctxcore/rnkdb.py", line 318, in load
    gene_set = self.geneset.intersection(set(gs.genes))
  File "cytoolz/functoolz.pyx", line 476, in cytoolz.functoolz._memoize.__call__
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/ctxcore/rnkdb.py", line 98, in geneset
    return set(self.genes)
  File "cytoolz/functoolz.pyx", line 476, in cytoolz.functoolz._memoize.__call__
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/ctxcore/rnkdb.py", line 296, in genes
    reader = FeatherReader(self._fname)
  File "/home/chun-hao.chao-umw/.conda/envs/scenic_protocol/lib/python3.6/site-packages/pyarrow/feather.py", line 40, in __init__
    self.open(source)
  File "pyarrow/feather.pxi", line 83, in pyarrow.lib.FeatherReader.open
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not a feather file

Please complete the following information:

pySCENIC version: 0.11.2
Installation method: conda and pip
Run environment: HPC cluster
OS: linux
Package versions: Package Version

aiohttp 3.8.6 aiosignal 1.2.0 arboreto 0.1.6 async-timeout 4.0.2 asynctest 0.13.0 attrs 22.2.0 bokeh 2.3.3 boltons 23.1.1 cached-property 1.5.2 certifi 2024.8.30 charset-normalizer 2.0.12 click 8.0.4 cloudpickle 2.2.1 contextvars 2.4 ctxcore 0.1.1 cycler 0.11.0 cytoolz 0.11.0 dask 2021.3.0 decorator 4.4.2 dill 0.3.4 distributed 2021.3.0 frozendict 2.4.6 frozenlist 1.2.0 fsspec 2022.1.0 h5py 3.1.0 HeapDict 1.0.1 idna 3.10 idna-ssl 1.1.0 immutables 0.19 importlib-metadata 4.8.3 importlib-resources 5.4.0 interlap 0.2.7 Jinja2 3.0.3 joblib 1.1.1 kiwisolver 1.3.1 llvmlite 0.36.0 locket 1.0.0 loompy 3.0.7 MarkupSafe 2.0.1 matplotlib 3.3.4 msgpack 1.0.5 multidict 5.2.0 multiprocessing-on-dill 3.5.0a4 networkx 2.5.1 numba 0.53.1 numpy 1.19.5 numpy-groupies 0.9.22 packaging 21.3 pandas 1.1.5 partd 1.2.0 patsy 0.5.6 Pillow 8.4.0 pip 21.3.1 psutil 6.1.0 pyarrow 0.16.0 pynndescent 0.5.13 pyparsing 3.1.4 pyscenic 0.11.2 python-dateutil 2.9.0.post0 pytz 2024.2 PyYAML 6.0.1 requests 2.27.1 scikit-learn 0.24.2 scipy 1.5.4 seaborn 0.11.2 setuptools 58.0.4 six 1.16.0 sortedcontainers 2.4.0 statsmodels 0.12.2 tblib 1.7.0 threadpoolctl 3.1.0 toolz 0.11.2 tornado 6.1 tqdm 4.64.1 typing_extensions 4.1.1 umap-learn 0.5.7 urllib3 1.26.20 wheel 0.37.1 yarl 1.7.2 zict 2.1.0 zipp 3.6.0

However, I realize the feather file contain version 1 and version 2. Only if I tried to create conda environment python >=3.7. I can install newer version of pyscenic =0.12.1 and also ctxcore=2.0.0 and pyarrow= 17.0.0. I expected to be work on feather v2.

I just tried pyscenic -h. I repeatedly encounter the same error in following.

Traceback (most recent call last):
  File "/home/chun-hao.chao-umw/.conda/envs/py39/bin/pyscenic", line 5, in <module>
    from pyscenic.cli.pyscenic import main
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/pyscenic/cli/pyscenic.py", line 19, in <module>
    from arboreto.algo import genie3, grnboost2
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/arboreto/algo.py", line 7, in <module>
    from arboreto.core import create_graph, SGBM_KWARGS, RF_KWARGS, EARLY_STOP_WINDOW_LENGTH
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/arboreto/core.py", line 10, in <module>
    from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor, ExtraTreesRegressor
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/__init__.py", line 87, in <module>
    from .base import clone
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/base.py", line 19, in <module>
    from .utils import _IS_32BIT
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/utils/__init__.py", line 22, in <module>
    from ._estimator_html_repr import estimator_html_repr
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/utils/_estimator_html_repr.py", line 10, in <module>
    from .fixes import parse_version
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/utils/fixes.py", line 17, in <module>
    import scipy.stats
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/stats/__init__.py", line 608, in <module>
    from ._stats_py import *
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/stats/_stats_py.py", line 39, in <module>
    from scipy.spatial.distance import cdist
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/spatial/__init__.py", line 110, in <module>
    from ._kdtree import *
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/spatial/_kdtree.py", line 4, in <module>
    from ._ckdtree import cKDTree, cKDTreeNode
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Traceback (most recent call last):
  File "/home/chun-hao.chao-umw/.conda/envs/py39/bin/pyscenic", line 5, in <module>
    from pyscenic.cli.pyscenic import main
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/pyscenic/cli/pyscenic.py", line 19, in <module>
    from arboreto.algo import genie3, grnboost2
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/arboreto/algo.py", line 7, in <module>
    from arboreto.core import create_graph, SGBM_KWARGS, RF_KWARGS, EARLY_STOP_WINDOW_LENGTH
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/arboreto/core.py", line 10, in <module>
    from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor, ExtraTreesRegressor
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/__init__.py", line 87, in <module>
    from .base import clone
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/base.py", line 19, in <module>
    from .utils import _IS_32BIT
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/utils/__init__.py", line 22, in <module>
    from ._estimator_html_repr import estimator_html_repr
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/utils/_estimator_html_repr.py", line 10, in <module>
    from .fixes import parse_version
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/sklearn/utils/fixes.py", line 17, in <module>
    import scipy.stats
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/stats/__init__.py", line 608, in <module>
    from ._stats_py import *
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/stats/_stats_py.py", line 39, in <module>
    from scipy.spatial.distance import cdist
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/spatial/__init__.py", line 110, in <module>
    from ._kdtree import *
  File "/home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/spatial/_kdtree.py", line 4, in <module>
    from ._ckdtree import cKDTree, cKDTreeNode
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/chun-hao.chao-umw/.conda/envs/py39/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)

I guess it still the version conflict issue. I tried to reinstall through conda install -c conda-forge libstdcxx-ng. It is still the same.

Please complete the following information:

pySCENIC version: 0.12.1
Installation method: conda and pip
Run environment: HPC cluster
OS: linux

Package versions:

Package                 Version
----------------------- ---------
aiohappyeyeballs        2.4.3
aiohttp                 3.10.10
aiosignal               1.3.1
arboreto                0.1.6
async-timeout           4.0.3
attrs                   24.2.0
bokeh                   3.1.1
boltons                 24.0.0
Brotli                  1.0.9
certifi                 2024.8.30
cffi                    1.17.0
charset-normalizer      3.4.0
click                   8.1.7
cloudpickle             3.1.0
contourpy               1.1.1
ctxcore                 0.2.0
cycler                  0.12.1
cytoolz                 0.12.2
dask                    2023.5.0
dill                    0.3.9
distributed             2023.5.0
fonttools               4.53.1
frozendict              2.4.6
frozenlist              1.5.0
fsspec                  2024.10.0
h2                      4.1.0
h5py                    3.11.0
hpack                   4.0.0
hyperframe              6.0.1
idna                    3.10
importlib_metadata      8.5.0
importlib_resources     6.4.5
interlap                0.2.7
Jinja2                  3.1.4
joblib                  1.4.2
kiwisolver              1.4.5
llvmlite                0.41.1
locket                  1.0.0
loompy                  3.0.7
lz4                     4.3.3
MarkupSafe              2.1.5
matplotlib              3.7.3
msgpack                 1.1.0
multidict               6.1.0
multiprocessing_on_dill 3.5.0a4
networkx                3.1
numba                   0.58.1
numexpr                 2.8.4
numpy                   1.24.4
numpy_groupies          0.9.22
packaging               24.1
pandas                  1.3.5
partd                   1.4.1
patsy                   0.5.6
pillow                  10.4.0
pip                     24.3.1
platformdirs            4.3.6
ply                     3.11
pooch                   1.8.2
propcache               0.2.0
psutil                  6.1.0
py-cpuinfo              9.0.0
pyarrow                 17.0.0
pycparser               2.22
pynndescent             0.5.13
pyparsing               3.1.4
PyQt5                   5.15.9
PyQt5-sip               12.12.2
pyscenic                0.12.1
PySocks                 1.7.1
python-dateutil         2.9.0
pytz                    2024.2
PyYAML                  6.0.2
requests                2.32.3
scikit-learn            1.3.2
scipy                   1.10.1
seaborn                 0.13.2
setuptools              75.1.0
sip                     6.7.12
six                     1.16.0
sortedcontainers        2.4.0
statsmodels             0.14.1
tables                  3.8.0
tblib                   3.0.0
threadpoolctl           3.5.0
toml                    0.10.2
tomli                   2.0.2
toolz                   0.12.0
tornado                 6.4.1
tqdm                    4.66.6
typing_extensions       4.12.2
tzdata                  2024.2
umap-learn              0.5.7
unicodedata2            15.1.0
urllib3                 2.2.3
wheel                   0.44.0
xyzservices             2024.9.0
yarl                    1.15.2
zict                    3.0.0
zipp                    3.20.2
zstandard               0.23.0

What would be the ideal version to install on pyscenic? Or I just use feather file version1 to fix the problem.

aertslab / pySCENIC

[BUG]pyscenic version conflict when using feather v2 vs v1 #590