scverse / scvi-tools

Deep probabilistic analysis of single-cell and spatial omics data
http://scvi-tools.org/
BSD 3-Clause "New" or "Revised" License
1.2k stars 344 forks source link

In Jupyter Notebook on Windows, the kernel keeps crashing when running Model hyperparameter tuning with scVI. #2917

Closed GLking123 closed 1 month ago

GLking123 commented 1 month ago

Hi

The running code is as follows:

`model_cls = scvi.model.SCVI model_cls.setup_anndata(adata, categorical_covariate_keys = ['Sample'], continuous_covariate_keys=['pct_counts_mt'])

tuner = ModelTuner(model_cls)

search_space = { "n_hidden": tune.choice([92, 128, 192, 256]), "n_latent": tune.choice([10, 20, 30, 40, 50, 60,70,80]), "n_layers": tune.choice([1, 2, 3]), "lr": tune.loguniform(1e-4, 1e-2), "gene_likelihood": tune.choice(["nb", "zinb"])}

results = tuner.fit( adata, metric="validation_loss", search_space=search_space, num_samples=30, max_epochs=20, resources={"cpu": 10, "gpu": 1}, ) `

The following error message is displayed: symbolize_win32.inc : 53] raw: syminitialize() failed: 87 fatal python error: aborted

For the above question, could you provide some debugging suggestions? Thank you for your valuable time and assistance. I sincerely look forward to your response!

canergen commented 1 month ago

Hi, it's hard for me to test except our CI tests for Windows in GitHub. Let's try to reduce the reasons for failure. Can you provide whether just the below raises the issue or the train runner:

model_cls = scvi.model.SCVI
model_cls.setup_anndata(adata, categorical_covariate_keys = ['Sample'],
continuous_covariate_keys=['pct_counts_mt'])

tuner = ModelTuner(model_cls)

search_space = {
"n_hidden": tune.choice([92, 128, 192, 256]),
"n_latent": tune.choice([10, 20, 30, 40, 50, 60,70,80]),
"n_layers": tune.choice([1, 2, 3]),
"lr": tune.loguniform(1e-4, 1e-2),
"gene_likelihood": tune.choice(["nb", "zinb"])}

I would then disable the GPU. Can you train a model using GPU only in the same environment? Does the same code run in a Colab environment? The error code tells us that there is a wrong parameter somewhere in the code. Can you check: https://github.com/scverse/scvi-tools/issues/2908? We will release a patch fix in the coming days.

GLking123 commented 1 month ago

The issue with RunConfig(local_dir) argument is deprecated is due to the Ray version being too high. When I downgraded to version 1.12.0, the error didn't occur. However, with version 1.12.0, running ray.init() causes the kernel to crash and displays: 2024-07-29 19:41:51,397 INFO worker.py:1752 -- Started a local Ray instance. After that, the kernel crashes. When I run the above code in the command prompt, it shows: symbolize_win32.inc: 53] raw: syminitialize() failed: 87 fatal python error: aborted

I used resources={"gpu": 1} and got the same result.

The following is a more detailed error report: `2024-07-29 17:16:11,242 INFO worker.py:1752 -- Started a local Ray instance. [symbolize_win32.inc : 53] RAW: SymInitialize() failed: 87 Fatal Python error: Aborted

Stack (most recent call first): File "K:\anaconda3\envs\gl\lib\site-packages\ray_private\worker.py", line 2391 in connect File "K:\anaconda3\envs\gl\lib\site-packages\ray_private\worker.py", line 1754 in init File "K:\anaconda3\envs\gl\lib\site-packages\ray_private\client_mode_hook.py", line 103 in wrapper File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\tune.py", line 262 in _ray_auto_init File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\tune.py", line 538 in run File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\impl\tuner_internal.py", line 596 in _fit_internal File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\impl\tuner_internal.py", line 477 in fit File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\tuner.py", line 379 in fit File "K:\anaconda3\envs\gl\lib\site-packages\scvi\autotune_tuner.py", line 116 in fit File "K:\single_cell_gl\new_scvi.py", line 126 in `

canergen commented 1 month ago

So does initializing ray also create this error outside of scvi-tools? So just running ray.init() throws this error? If ray is correctly installed and you can follow their tutorials, can you try installing from our main branch? The whole function is rewritten and it might be things are working there.

GLking123 commented 1 month ago

Hi

Thank you for your explanation. As long as I run ray.init(), an error is thrown. I will try installing the latest main branch. Thank you again.

canergen commented 1 month ago

It looks like a problem with your ray installation then. Please try running one of their cases first like https://docs.ray.io/en/latest/tune/getting-started.html. If it fails, please request more information from the ray[tune] developers. Closing here. Please reopen if the tutorials works but scvi-tools hypertuning fails.

Pinolinoo commented 1 month ago

Hi, I have a similar issue.

When I run: model_cls = scvi.model.SCVI model_cls.setup_anndata(adata_subset, categorical_covariate_keys = ['Sample'],continuous_covariate_keys=['pct_counts_mt', 'pct_counts_ribo'])

The kernel dies when I perform the setup_anndata. I run on a cluster with GPU and the OS is RHEl9.

ray.init() works fine!

Thanks very much in advance for your help!

canergen commented 1 month ago

Hi. This looks like a different issue (the first one is supposedly an error within ray). Can you first try one of our tutorials to exclude it’s something with your specific data. There’s not much happening at setup_anndata. Do you get any message in the command line instance that runs the notebook? Could you check the code directly in command line to provide more insights?

Pinolinoo commented 1 month ago

Hi can, thanks so much for your reply. I tried with an adata from your tutorials

(adata_path = os.path.join(save_dir.name, "lung_atlas.h5ad")

adata = sc.read( adata_path, backup_url="https://figshare.com/ndownloader/files/24539942", )

I get the same kernel crash when I do: model_cls.setup_anndata(adata, categorical_covariate_keys = ['batch'],continuous_covariate_keys=['percent.mito'])

All I get in the terminal from the jupyter session is: WARNING:root:kernel 829f8cb5-256f-431c-84bf-e54521da75c0 restarted

Right now I unfortunately don't have time to test the code directly in the command line, will do this next though if the others infos aren't enough for now! Thank you so much!

Best Philipp

Pinolinoo commented 1 month ago

Ok update:

I tried again directly in the command line with the scvi tutorials lung atlas dataset and the error I get is: Segmentation fault (core dumped)

I have 100GB free Memory however in the interactive session and when I tried on a Jupyter Kernel Session with 512GB free Memory the kernel also crashed.

canergen commented 1 month ago

Thanks it’s a hard one to debug. Can you paste the output when enabling faulthandler: e.g. python3 -q -X faulthandler

Pinolinoo commented 1 month ago

Sure, here you go! Thanks a lot for your hard work!

`Thread 0x00007fa883d91640 (most recent call first): File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/threading.py", line 359 in wait File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/threading.py", line 655 in wait File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x00007fa98c397740 (most recent call first): File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 704 in register_plugin File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax_plugins/xla_cuda12/init.py", line 79 in initialize File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 623 in discover_pjrt_plugins File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 758 in _discover_and_register_pjrt_plugins File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 855 in backends File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1002 in _get_backend_uncached File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1023 in get_backend File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1089 in devices File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/_utils.py", line 272 in _check_nonnegative_integers File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/fields/_layer_field.py", line 87 in validate_field File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/fields/_base_field.py", line 64 in register_field File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/fields/_layer_field.py", line 114 in register_field File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/_manager.py", line 223 in _add_field File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/_manager.py", line 180 in register_fields File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/model/_scvi.py", line 219 in setup_anndata File "", line 1 in

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, PIL._imaging, kiwisolver._cext, sklearn.utils._random, _brotli, zstandard.backend_c, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, gmpy2.gmpy2, jaxlib.cpu_feature_guard, msgpack._cmsgpack, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.utils._fast_dict, sklearn.cluster._hierarchical_fast, sklearn.cluster._k_means_common, sklearn.cluster._k_means_elkan, sklearn.cluster._k_means_lloyd, sklearn.cluster._k_means_minibatch, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, sklearn.cluster._dbscan_inner, sklearn.cluster._hdbscan._tree, sklearn.cluster._hdbscan._linkage, sklearn.cluster._hdbscan._reachability, sklearn._isotonic, sklearn.tree._utils, sklearn.tree._tree, sklearn.tree._splitter, sklearn.tree._criterion, sklearn.neighbors._quad_tree, sklearn.manifold._barnes_hut_tsne, sklearn.manifold._utils, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering (total: 272) Segmentation fault (core dumped)`

canergen commented 1 month ago

Thanks, we’re getting there. It looks like your JAX installation is broken. Can you try:

def selu(x, alpha=1.67, lmbda=1.05):
  return lmbda * jnp.where(x > 0, x, alpha * jnp.exp(x) - alpha)

x = jnp.arange(5.0)
print(selu(x))
canergen commented 1 month ago

And:

from jax import jit

selu_jit = jit(selu)
_ = selu_jit(x)  # compiles on first call
%timeit selu_jit(x).block_until_ready()
Pinolinoo commented 1 month ago

def selu(x, alpha=1.67, lmbda=1.05): ... return lmbda jnp.where(x > 0, x, alpha jnp.exp(x) - alpha) ...

x = jnp.arange(5.0) Fatal Python error: Segmentation fault

Current thread 0x00007f3597f0d740 (most recent call first): File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 704 in register_plugin File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax_plugins/xla_cuda12/init.py", line 79 in initialize File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 623 in discover_pjrt_plugins File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 758 in _discover_and_register_pjrt_plugins File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 855 in backends File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1002 in _get_backend_uncached File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1023 in get_backend File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1147 in local_devices File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/interpreters/pxla.py", line 1671 in _get_default_device File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/interpreters/pxla.py", line 1716 in _get_and_check_device_assignment File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/interpreters/pxla.py", line 1680 in call File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1365 in _resolve_in_shardings File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1443 in _resolve_and_lower File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1460 in _pjit_call_impl_python File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1534 in call_impl_cache_miss File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1558 in _pjit_call_impl File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 879 in process_primitive File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 391 in bind_with_trace File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 2789 in bind File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 182 in _python_pjit_helper File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 305 in cache_miss File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/traceback_util.py", line 179 in reraise_with_filtered_traceback File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/dispatch.py", line 86 in apply_primitive File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 879 in process_primitive File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 391 in bind_with_trace File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 387 in bind File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/lax/lax.py", line 1292 in broadcasted_iota File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/lax/lax.py", line 1282 in iota File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/numpy/lax_numpy.py", line 2952 in arange File "", line 1 in

Extension modules: h5py._errors, h5py.defs, h5py._objects, h5py.h5, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, PIL._imaging, kiwisolver._cext, sklearn.utils._random, _brotli, zstandard.backend_c, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, gmpy2.gmpy2, jaxlib.cpu_feature_guard, msgpack._cmsgpack, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.utils._fast_dict, sklearn.cluster._hierarchical_fast, sklearn.cluster._k_means_common, sklearn.cluster._k_means_elkan, sklearn.cluster._k_means_lloyd, sklearn.cluster._k_means_minibatch, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, sklearn.cluster._dbscan_inner, sklearn.cluster._hdbscan._tree, sklearn.cluster._hdbscan._linkage, sklearn.cluster._hdbscan._reachability, sklearn._isotonic, sklearn.tree._utils, sklearn.tree._tree, sklearn.tree._splitter, sklearn.tree._criterion, sklearn.neighbors._quad_tree, sklearn.manifold._barnes_hut_tsne, sklearn.manifold._utils, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering (total: 272) Segmentation fault (core dumped)

Pinolinoo commented 1 month ago

Seems like just the jnp.arrang() gives the mistake already!

canergen commented 1 month ago

I would recommend to set up a new conda environment and follow the installation instructions on the JAX homepage (CPU version will be more stable if you only need it for scVI). If this doesn’t fix it, can you create an issue there. Can you install scVI after installing and testing JAX?

Pinolinoo commented 1 month ago

Ok I will do that. I want to use scVI for an integration of datasets and wanted to perform a hyperparameter search before. Wouldn't it be better to do that on the gpu and thus also use JAX for gpu?

canergen commented 1 month ago

JAX is used for the MrVI model and JaxSCVI, all other models don't benefit from a GPU-enabled JAX (installing CUDA-enabled JAX is sometimes a bit of a pain). However, it should also work to use a GPU-enabled version of JAX. They have guides on how to install it on their webpage. Please follow this and verify your installation. Closing here as it is an issue not related to scvi-tools.

Pinolinoo commented 1 month ago

I did like you suggested and installed Jax first. The tests were then working however when I install scvi afterwards and then try to import scvi I get this error:

RuntimeError Traceback (most recent call last) Cell In[1], line 1 ----> 1 import scvi 2 from scvi.autotune import ModelTuner 3 from ray import tune

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/init.py:11 8 from ._settings import settings 10 # this import needs to come after prior imports to prevent circular import ---> 11 from . import data, model, external, utils 13 from importlib.metadata import version 15 package_name = "scvi-tools"

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/data/init.py:3 1 from anndata import read_csv, read_h5ad, read_loom, read_text ----> 3 from ._anntorchdataset import AnnTorchDataset 4 from ._datasets import ( 5 annotation_simulation, 6 brainlarge_dataset, (...) 22 synthetic_iid, 23 ) 24 from ._manager import AnnDataManager, AnnDataManagerValidationCheck

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/data/_anntorchdataset.py:23 20 from torch.utils.data import Dataset 22 from scvi._constants import REGISTRY_KEYS ---> 23 from scvi.utils._exceptions import InvalidParameterError 25 if TYPE_CHECKING: 26 from ._manager import AnnDataManager

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/utils/init.py:6 4 from ._docstrings import de_dsp, setup_anndata_dsp 5 from ._exceptions import InvalidParameterError ----> 6 from ._jax import device_selecting_PRNGKey 7 from ._track import track 9 all = [ 10 "track", 11 "setup_anndata_dsp", (...) 18 "dependencies", 19 ]

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/utils/_jax.py:3 1 from typing import Callable ----> 3 import jax 4 from jax import random 7 def device_selecting_PRNGKey(use_cpu: bool = True) -> Callable:

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/init.py:37 34 del _cloud_tpu_init 36 # Force early import, allowing use of jax.core after importing jax. ---> 37 import jax.core as _core 38 del _core 40 # Note: import as is required for names to be exported. 41 # See PEP 484 & https://github.com/google/jax/issues/7570

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/core.py:18 1 # Copyright 2022 The JAX Authors. 2 # 3 # Licensed under the Apache License, Version 2.0 (the "License"); (...) 15 # Note: import as is required for names to be exported. 16 # See PEP 484 & https://github.com/google/jax/issues/7570 ---> 18 from jax._src.core import ( 19 AbstractToken as AbstractToken, 20 AbstractValue as AbstractValue, 21 Atom as Atom, 22 AxisSize as AxisSize, 23 CallPrimitive as CallPrimitive, 24 ClosedJaxpr as ClosedJaxpr, 25 ConcreteArray as ConcreteArray, 26 ConcretizationTypeError as ConcretizationTypeError, 27 DShapedArray as DShapedArray, 28 DropVar as DropVar, 29 Effect as Effect, 30 Effects as Effects, 31 EvalTrace as EvalTrace, 32 InDBIdx as InDBIdx, 33 InconclusiveDimensionOperation as InconclusiveDimensionOperation, 34 InputType as InputType, 35 Jaxpr as Jaxpr, 36 JaxprDebugInfo as JaxprDebugInfo, 37 JaxprEqn as JaxprEqn, 38 JaxprPpContext as JaxprPpContext, 39 JaxprPpSettings as JaxprPpSettings, 40 JaxprTypeError as JaxprTypeError, 41 Literal as Literal, 42 MainTrace as MainTrace, 43 MapPrimitive as MapPrimitive, 44 NameGatheringSubst as NameGatheringSubst, 45 NamedShape as NamedShape, 46 OutDBIdx as OutDBIdx, 47 OutputType as OutputType, 48 ParamDict as ParamDict, 49 Primitive as Primitive, 50 ShapedArray as ShapedArray, 51 Sublevel as Sublevel, 52 TRACER_LEAK_DEBUGGER_WARNING as TRACER_LEAK_DEBUGGER_WARNING, 53 ThreadLocalState as ThreadLocalState, 54 Token as Token, 55 Trace as Trace, 56 TraceStack as TraceStack, 57 TraceState as TraceState, 58 Tracer as Tracer, 59 UnshapedArray as UnshapedArray, 60 Value as Value, 61 Var as Var, 62 abstract_token as abstract_token, 63 apply_todos as apply_todos, 64 as_named_shape as as_named_shape, 65 aval_mapping_handlers as aval_mapping_handlers, 66 axis_frame as axis_frame, 67 call as call, 68 call_bind_with_continuation as call_bind_with_continuation, 69 call_impl as call_impl, 70 call_p as call_p, 71 canonicalize_shape as _deprecated_canonicalize_shape, 72 check_eqn as check_eqn, 73 check_jaxpr as check_jaxpr, 74 check_type as check_type, 75 check_valid_jaxtype as check_valid_jaxtype, 76 closed_call_p as closed_call_p, 77 concrete_aval as concrete_aval, 78 concrete_or_error as concrete_or_error, 79 concretization_function_error as concretization_function_error, 80 cur_sublevel as cur_sublevel, 81 custom_typechecks as custom_typechecks, 82 dedup_referents as dedup_referents, 83 definitely_equal as _deprecated_definitely_equal, 84 dimension_as_value as _deprecated_dimension_as_value, 85 do_subst_axis_names_jaxpr as do_subst_axis_names_jaxpr, 86 ensure_compile_time_eval as ensure_compile_time_eval, 87 escaped_tracer_error as escaped_tracer_error, 88 eval_context as eval_context, 89 eval_jaxpr as eval_jaxpr, 90 extend_axis_env as extend_axis_env, 91 extend_axis_env_nd as extend_axis_env_nd, 92 find_top_trace as find_top_trace, 93 full_lower as full_lower, 94 gensym as gensym, 95 get_aval as get_aval, 96 get_referent as get_referent, 97 is_constant_dim as is_constant_dim, 98 is_constant_shape as is_constant_shape, 99 jaxpr_as_fun as jaxpr_as_fun, 100 jaxpr_uses_outfeed as jaxpr_uses_outfeed, 101 jaxprs_in_params as jaxprs_in_params, 102 join_effects as join_effects, 103 join_named_shapes as join_named_shapes, 104 lattice_join as lattice_join, 105 leaked_tracer_error as leaked_tracer_error, 106 literalable_types as literalable_types, 107 map_bind as map_bind, 108 map_bind_with_continuation as map_bind_with_continuation, 109 mapped_aval as mapped_aval, 110 maybe_find_leaked_tracers as maybe_find_leaked_tracers, 111 max_dim as max_dim, 112 min_dim as min_dim, 113 new_base_main as new_base_main, 114 new_jaxpr_eqn as new_jaxpr_eqn, 115 new_main as new_main, 116 new_sublevel as new_sublevel, 117 no_axis_name as no_axis_name, 118 no_effects as no_effects, 119 non_negative_dim as _deprecated_non_negative_dim, 120 outfeed_primitives as outfeed_primitives, 121 pp_aval as pp_aval, 122 pp_eqn as pp_eqn, 123 pp_eqn_rules as pp_eqn_rules, 124 pp_eqns as pp_eqns, 125 pp_jaxpr as pp_jaxpr, 126 pp_jaxpr_eqn_range as pp_jaxpr_eqn_range, 127 pp_jaxpr_skeleton as pp_jaxpr_skeleton, 128 pp_jaxprs as pp_jaxprs, 129 pp_kv_pair as pp_kv_pair, 130 pp_kv_pairs as pp_kv_pairs, 131 pp_var as pp_var, 132 pp_vars as pp_vars, 133 primal_dtype_to_tangent_dtype as primal_dtype_to_tangent_dtype, 134 primitive_uses_outfeed as primitive_uses_outfeed, 135 process_env_traces_call as process_env_traces_call, 136 process_env_traces_map as process_env_traces_map, 137 pytype_aval_mappings as pytype_aval_mappings, 138 raise_as_much_as_possible as raise_as_much_as_possible, 139 raise_to_shaped as raise_to_shaped, 140 raise_to_shaped_mappings as raise_to_shaped_mappings, 141 reset_trace_state as reset_trace_state, 142 stash_axis_env as stash_axis_env, 143 str_eqn_compact as str_eqn_compact, 144 subjaxprs as subjaxprs, 145 subst_axis_names as subst_axis_names, 146 subst_axis_names_eqn as subst_axis_names_eqn, 147 subst_axis_names_jaxpr as subst_axis_names_jaxpr, 148 subst_axis_names_var as subst_axis_names_var, 149 substitute_vars_in_output_ty as substitute_vars_in_output_ty, 150 thread_local_state as thread_local_state, 151 trace_state_clean as trace_state_clean, 152 traverse_jaxpr_params as traverse_jaxpr_params, 153 typecheck as typecheck, 154 typecompat as typecompat, 155 typematch as typematch, 156 unmapped_aval as unmapped_aval, 157 used_axis_names as used_axis_names, 158 used_axis_names_jaxpr as used_axis_names_jaxpr, 159 valid_jaxtype as valid_jaxtype, 160 ) 163 from jax._src import core as _src_core 164 _deprecations = { 165 # Added Oct 11, 2023: 166 "DimSize": ( (...) 190 ), 191 }

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/core.py:39 35 from weakref import ref 37 import numpy as np ---> 39 from jax._src import dtypes 40 from jax._src import config 41 from jax._src import effects

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/dtypes.py:33 30 import ml_dtypes 31 import numpy as np ---> 33 from jax._src import config 34 from jax._src.typing import DType, DTypeLike 35 from jax._src.util import set_module, StrictABC

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/config.py:27 24 import threading 25 from typing import Any, Callable, Generic, NamedTuple, NoReturn, TypeVar, cast ---> 27 from jax._src import lib 28 from jax._src.lib import jax_jit 29 from jax._src.lib import transfer_guard_lib

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/lib/init.py:75 71 return _jaxlib_version 74 version_str = jaxlib.version.version ---> 75 version = check_jaxlib_version( 76 jax_version=jax.version.version, 77 jaxlib_version=jaxlib.version.version, 78 minimum_jaxlib_version=jax.version._minimum_jaxlib_version) 80 # Before importing any C compiled modules from jaxlib, first import the CPU 81 # feature guard module to verify that jaxlib was compiled in a way that only 82 # uses instructions that are present on this machine. 83 import jaxlib.cpu_feature_guard as cpu_feature_guard

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/lib/init.py:67, in check_jaxlib_version(jax_version, jaxlib_version, minimum_jaxlib_version) 64 raise RuntimeError(msg) 66 if _jaxlib_version > _jax_version: ---> 67 raise RuntimeError( 68 f'jaxlib version {jaxlib_version} is newer than and ' 69 f'incompatible with jax version {jax_version}. Please ' 70 'update your jax and/or jaxlib packages.') 71 return _jaxlib_version

RuntimeError: jaxlib version 0.4.31 is newer than and incompatible with jax version 0.4.27. Please update your jax and/or jaxlib packages.

canergen commented 1 month ago

It seems the Windows prebuilt are experimental for jax. You have to install jaxlib and jax in the same version pip install jaxlib==0.4.27. It's not straightforward to track this in the pyproject.toml file as far as I'm aware.

Pinolinoo commented 1 month ago

Thank you, I managed to get everything installed and the setup.anndata() works now however later when I run

model_cls = scvi.model.SCVI
model_cls.setup_anndata(adata, categorical_covariate_keys = ['Sample'],continuous_covariate_keys=['pct_counts_mt', 'pct_counts_ribo'])

tuner = ModelTuner(model_cls)

results = tuner.fit(adata, metric="validation_loss",
                    resources = {'gpu': 1}, #have to specify gpu or might not use
                    search_space = search_space,
                    num_samples = 100,
                    max_epochs = 20)

I get this error:

---------------------------------------------------------------------------
DeprecationWarning                        Traceback (most recent call last)
Cell In[16], [line 1](vscode-notebook-cell:?execution_count=16&line=1)
----> [1](vscode-notebook-cell:?execution_count=16&line=1) results = tuner.fit(adata, metric="validation_loss",
      [2](vscode-notebook-cell:?execution_count=16&line=2)                     resources = {'gpu': 1}, #have to specify gpu or might not use
      [3](vscode-notebook-cell:?execution_count=16&line=3)                     search_space = search_space,
      [4](vscode-notebook-cell:?execution_count=16&line=4)                     num_samples = 100,
      [5](vscode-notebook-cell:?execution_count=16&line=5)                     max_epochs = 20)

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:115, in ModelTuner.fit(self, adata, **kwargs)
     [33](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:33) def fit(self, adata: AnnOrMuData, **kwargs) -> None:
     [34](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:34)     """Run a specified hyperparameter sweep for the associated model class.
     [35](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:35) 
     [36](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:36)     Parameters
   (...)
    [113](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:113)         A dataclass containing the results of the tuning experiment.
    [114](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:114)     """
--> [115](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:115)     tuner, config = self._manager._get_tuner(adata, **kwargs)
    [116](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:116)     results = tuner.fit()
    [117](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:117)     return self._manager._get_analysis(results, config)

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:558, in TunerManager._get_tuner(self, adata, metric, additional_metrics, search_space, model_kwargs, train_kwargs, use_defaults, num_samples, max_epochs, scheduler, scheduler_kwargs, searcher, searcher_kwargs, reporter, resources, seed, experiment_name, logging_dir, monitor_device_stats)
    [538](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:538) _trainable = self._get_trainable(
    [539](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:539)     adata,
    [540](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:540)     _metrics,
   (...)
    [550](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:550)     monitor_device_stats=monitor_device_stats,
    [551](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:551) )
    [553](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:553) tune_config = tune.tune_config.TuneConfig(
    [554](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:554)     scheduler=_scheduler,
    [555](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:555)     search_alg=_searcher,
    [556](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:556)     num_samples=num_samples,
    [557](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:557) )
--> [558](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:558) run_config = air.config.RunConfig(
    [559](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:559)     name=_experiment_name,
    [560](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:560)     local_dir=_logging_dir,
    [561](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:561)     progress_reporter=_reporter,
    [562](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:562)     log_to_file=True,
    [563](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:563)     verbose=1,
    [564](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:564) )
    [565](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:565) tuner = tune.Tuner(
    [566](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:566)     trainable=_trainable,
    [567](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:567)     param_space=_search_space,
    [568](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:568)     tune_config=tune_config,
    [569](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:569)     run_config=run_config,
    [570](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:570) )
    [571](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:571) config = {
    [572](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:572)     "metrics": _metrics,
    [573](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:573)     "search_space": _search_space,
    [574](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:574) }

File <string>:15, in __init__(self, name, storage_path, storage_filesystem, failure_config, checkpoint_config, sync_config, verbose, stop, callbacks, progress_reporter, log_to_file, local_dir)

File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:665, in RunConfig.__post_init__(self)
    [662](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:662) from ray.tune.experimental.output import AirVerbosity, get_air_verbosity
    [664](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:664) if self.local_dir is not None:
--> [665](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:665)     raise DeprecationWarning(
    [666](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:666)         "The `RunConfig(local_dir)` argument is deprecated. "
    [667](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:667)         "You should set the `RunConfig(storage_path)` instead."
    [668](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:668)         "See the docs: https://docs.ray.io/en/latest/train/user-guides/"
    [669](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:669)         "persistent-storage.html#setting-the-local-staging-directory"
    [670](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:670)     )
    [672](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:672) if self.storage_path is None:
    [673](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:673)     # TODO(justinvyu): [Deprecated] Remove in 2.30
    [674](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:674)     self.storage_path = DEFAULT_STORAGE_PATH

DeprecationWarning: The `RunConfig(local_dir)` argument is deprecated. You should set the `RunConfig(storage_path)` instead.See the docs: https://docs.ray.io/en/latest/train/user-guides/persistent-storage.html#setting-the-local-staging-directory
canergen commented 1 month ago

Great that it worked. This is a duplicate of #2908. Please see there for fixes.