Closed GLking123 closed 3 months ago
Hi, it's hard for me to test except our CI tests for Windows in GitHub. Let's try to reduce the reasons for failure. Can you provide whether just the below raises the issue or the train runner:
model_cls = scvi.model.SCVI
model_cls.setup_anndata(adata, categorical_covariate_keys = ['Sample'],
continuous_covariate_keys=['pct_counts_mt'])
tuner = ModelTuner(model_cls)
search_space = {
"n_hidden": tune.choice([92, 128, 192, 256]),
"n_latent": tune.choice([10, 20, 30, 40, 50, 60,70,80]),
"n_layers": tune.choice([1, 2, 3]),
"lr": tune.loguniform(1e-4, 1e-2),
"gene_likelihood": tune.choice(["nb", "zinb"])}
I would then disable the GPU. Can you train a model using GPU only in the same environment? Does the same code run in a Colab environment? The error code tells us that there is a wrong parameter somewhere in the code. Can you check: https://github.com/scverse/scvi-tools/issues/2908? We will release a patch fix in the coming days.
The issue with RunConfig(local_dir) argument is deprecated is due to the Ray version being too high.
When I downgraded to version 1.12.0, the error didn't occur.
However, with version 1.12.0, running ray.init() causes the kernel to crash and displays:
2024-07-29 19:41:51,397 INFO worker.py:1752 -- Started a local Ray instance.
After that, the kernel crashes. When I run the above code in the command prompt, it shows:
symbolize_win32.inc: 53] raw: syminitialize() failed: 87 fatal python error: aborted
I used resources={"gpu": 1} and got the same result.
The following is a more detailed error report: `2024-07-29 17:16:11,242 INFO worker.py:1752 -- Started a local Ray instance. [symbolize_win32.inc : 53] RAW: SymInitialize() failed: 87 Fatal Python error: Aborted
Stack (most recent call first):
File "K:\anaconda3\envs\gl\lib\site-packages\ray_private\worker.py", line 2391 in connect
File "K:\anaconda3\envs\gl\lib\site-packages\ray_private\worker.py", line 1754 in init
File "K:\anaconda3\envs\gl\lib\site-packages\ray_private\client_mode_hook.py", line 103 in wrapper
File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\tune.py", line 262 in _ray_auto_init
File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\tune.py", line 538 in run
File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\impl\tuner_internal.py", line 596 in _fit_internal
File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\impl\tuner_internal.py", line 477 in fit
File "K:\anaconda3\envs\gl\lib\site-packages\ray\tune\tuner.py", line 379 in fit
File "K:\anaconda3\envs\gl\lib\site-packages\scvi\autotune_tuner.py", line 116 in fit
File "K:\single_cell_gl\new_scvi.py", line 126 in
So does initializing ray also create this error outside of scvi-tools? So just running ray.init()
throws this error?
If ray is correctly installed and you can follow their tutorials, can you try installing from our main branch? The whole function is rewritten and it might be things are working there.
Hi
Thank you for your explanation. As long as I run ray.init(), an error is thrown. I will try installing the latest main branch. Thank you again.
It looks like a problem with your ray installation then. Please try running one of their cases first like https://docs.ray.io/en/latest/tune/getting-started.html. If it fails, please request more information from the ray[tune] developers. Closing here. Please reopen if the tutorials works but scvi-tools hypertuning fails.
Hi, I have a similar issue.
When I run: model_cls = scvi.model.SCVI model_cls.setup_anndata(adata_subset, categorical_covariate_keys = ['Sample'],continuous_covariate_keys=['pct_counts_mt', 'pct_counts_ribo'])
The kernel dies when I perform the setup_anndata. I run on a cluster with GPU and the OS is RHEl9.
ray.init() works fine!
Thanks very much in advance for your help!
Hi. This looks like a different issue (the first one is supposedly an error within ray). Can you first try one of our tutorials to exclude it’s something with your specific data. There’s not much happening at setup_anndata. Do you get any message in the command line instance that runs the notebook? Could you check the code directly in command line to provide more insights?
Hi can, thanks so much for your reply. I tried with an adata from your tutorials
(adata_path = os.path.join(save_dir.name, "lung_atlas.h5ad")
adata = sc.read( adata_path, backup_url="https://figshare.com/ndownloader/files/24539942", )
I get the same kernel crash when I do: model_cls.setup_anndata(adata, categorical_covariate_keys = ['batch'],continuous_covariate_keys=['percent.mito'])
All I get in the terminal from the jupyter session is: WARNING:root:kernel 829f8cb5-256f-431c-84bf-e54521da75c0 restarted
Right now I unfortunately don't have time to test the code directly in the command line, will do this next though if the others infos aren't enough for now! Thank you so much!
Best Philipp
Ok update:
I tried again directly in the command line with the scvi tutorials lung atlas dataset and the error I get is: Segmentation fault (core dumped)
I have 100GB free Memory however in the interactive session and when I tried on a Jupyter Kernel Session with 512GB free Memory the kernel also crashed.
Thanks it’s a hard one to debug. Can you paste the output when enabling faulthandler: e.g. python3 -q -X faulthandler
Sure, here you go! Thanks a lot for your hard work!
`Thread 0x00007fa883d91640 (most recent call first): File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/threading.py", line 359 in wait File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/threading.py", line 655 in wait File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/threading.py", line 1073 in _bootstrap_inner File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/threading.py", line 1030 in _bootstrap
Current thread 0x00007fa98c397740 (most recent call first):
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 704 in register_plugin
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax_plugins/xla_cuda12/init.py", line 79 in initialize
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 623 in discover_pjrt_plugins
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 758 in _discover_and_register_pjrt_plugins
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 855 in backends
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1002 in _get_backend_uncached
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1023 in get_backend
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1089 in devices
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/_utils.py", line 272 in _check_nonnegative_integers
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/fields/_layer_field.py", line 87 in validate_field
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/fields/_base_field.py", line 64 in register_field
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/fields/_layer_field.py", line 114 in register_field
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/_manager.py", line 223 in _add_field
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/data/_manager.py", line 180 in register_fields
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/scvi/model/_scvi.py", line 219 in setup_anndata
File "
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, PIL._imaging, kiwisolver._cext, sklearn.utils._random, _brotli, zstandard.backend_c, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, gmpy2.gmpy2, jaxlib.cpu_feature_guard, msgpack._cmsgpack, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.utils._fast_dict, sklearn.cluster._hierarchical_fast, sklearn.cluster._k_means_common, sklearn.cluster._k_means_elkan, sklearn.cluster._k_means_lloyd, sklearn.cluster._k_means_minibatch, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, sklearn.cluster._dbscan_inner, sklearn.cluster._hdbscan._tree, sklearn.cluster._hdbscan._linkage, sklearn.cluster._hdbscan._reachability, sklearn._isotonic, sklearn.tree._utils, sklearn.tree._tree, sklearn.tree._splitter, sklearn.tree._criterion, sklearn.neighbors._quad_tree, sklearn.manifold._barnes_hut_tsne, sklearn.manifold._utils, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering (total: 272) Segmentation fault (core dumped)`
Thanks, we’re getting there. It looks like your JAX installation is broken. Can you try:
def selu(x, alpha=1.67, lmbda=1.05):
return lmbda * jnp.where(x > 0, x, alpha * jnp.exp(x) - alpha)
x = jnp.arange(5.0)
print(selu(x))
And:
from jax import jit
selu_jit = jit(selu)
_ = selu_jit(x) # compiles on first call
%timeit selu_jit(x).block_until_ready()
def selu(x, alpha=1.67, lmbda=1.05): ... return lmbda jnp.where(x > 0, x, alpha jnp.exp(x) - alpha) ...
x = jnp.arange(5.0) Fatal Python error: Segmentation fault
Current thread 0x00007f3597f0d740 (most recent call first):
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 704 in register_plugin
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax_plugins/xla_cuda12/init.py", line 79 in initialize
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 623 in discover_pjrt_plugins
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 758 in _discover_and_register_pjrt_plugins
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 855 in backends
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1002 in _get_backend_uncached
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1023 in get_backend
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/xla_bridge.py", line 1147 in local_devices
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/interpreters/pxla.py", line 1671 in _get_default_device
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/interpreters/pxla.py", line 1716 in _get_and_check_device_assignment
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/interpreters/pxla.py", line 1680 in call
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1365 in _resolve_in_shardings
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1443 in _resolve_and_lower
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1460 in _pjit_call_impl_python
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1534 in call_impl_cache_miss
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 1558 in _pjit_call_impl
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 879 in process_primitive
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 391 in bind_with_trace
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 2789 in bind
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 182 in _python_pjit_helper
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/pjit.py", line 305 in cache_miss
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/traceback_util.py", line 179 in reraise_with_filtered_traceback
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/dispatch.py", line 86 in apply_primitive
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 879 in process_primitive
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 391 in bind_with_trace
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/core.py", line 387 in bind
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/lax/lax.py", line 1292 in broadcasted_iota
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/lax/lax.py", line 1282 in iota
File "/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.12/site-packages/jax/_src/numpy/lax_numpy.py", line 2952 in arange
File "
Extension modules: h5py._errors, h5py.defs, h5py._objects, h5py.h5, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, PIL._imaging, kiwisolver._cext, sklearn.utils._random, _brotli, zstandard.backend_c, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, gmpy2.gmpy2, jaxlib.cpu_feature_guard, msgpack._cmsgpack, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.utils._fast_dict, sklearn.cluster._hierarchical_fast, sklearn.cluster._k_means_common, sklearn.cluster._k_means_elkan, sklearn.cluster._k_means_lloyd, sklearn.cluster._k_means_minibatch, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, sklearn.cluster._dbscan_inner, sklearn.cluster._hdbscan._tree, sklearn.cluster._hdbscan._linkage, sklearn.cluster._hdbscan._reachability, sklearn._isotonic, sklearn.tree._utils, sklearn.tree._tree, sklearn.tree._splitter, sklearn.tree._criterion, sklearn.neighbors._quad_tree, sklearn.manifold._barnes_hut_tsne, sklearn.manifold._utils, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering (total: 272) Segmentation fault (core dumped)
Seems like just the jnp.arrang() gives the mistake already!
I would recommend to set up a new conda environment and follow the installation instructions on the JAX homepage (CPU version will be more stable if you only need it for scVI). If this doesn’t fix it, can you create an issue there. Can you install scVI after installing and testing JAX?
Ok I will do that. I want to use scVI for an integration of datasets and wanted to perform a hyperparameter search before. Wouldn't it be better to do that on the gpu and thus also use JAX for gpu?
JAX is used for the MrVI model and JaxSCVI, all other models don't benefit from a GPU-enabled JAX (installing CUDA-enabled JAX is sometimes a bit of a pain). However, it should also work to use a GPU-enabled version of JAX. They have guides on how to install it on their webpage. Please follow this and verify your installation. Closing here as it is an issue not related to scvi-tools.
I did like you suggested and installed Jax first. The tests were then working however when I install scvi afterwards and then try to import scvi I get this error:
RuntimeError Traceback (most recent call last) Cell In[1], line 1 ----> 1 import scvi 2 from scvi.autotune import ModelTuner 3 from ray import tune
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/init.py:11 8 from ._settings import settings 10 # this import needs to come after prior imports to prevent circular import ---> 11 from . import data, model, external, utils 13 from importlib.metadata import version 15 package_name = "scvi-tools"
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/data/init.py:3 1 from anndata import read_csv, read_h5ad, read_loom, read_text ----> 3 from ._anntorchdataset import AnnTorchDataset 4 from ._datasets import ( 5 annotation_simulation, 6 brainlarge_dataset, (...) 22 synthetic_iid, 23 ) 24 from ._manager import AnnDataManager, AnnDataManagerValidationCheck
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/data/_anntorchdataset.py:23 20 from torch.utils.data import Dataset 22 from scvi._constants import REGISTRY_KEYS ---> 23 from scvi.utils._exceptions import InvalidParameterError 25 if TYPE_CHECKING: 26 from ._manager import AnnDataManager
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/utils/init.py:6 4 from ._docstrings import de_dsp, setup_anndata_dsp 5 from ._exceptions import InvalidParameterError ----> 6 from ._jax import device_selecting_PRNGKey 7 from ._track import track 9 all = [ 10 "track", 11 "setup_anndata_dsp", (...) 18 "dependencies", 19 ]
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/utils/_jax.py:3 1 from typing import Callable ----> 3 import jax 4 from jax import random 7 def device_selecting_PRNGKey(use_cpu: bool = True) -> Callable:
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/init.py:37
34 del _cloud_tpu_init
36 # Force early import, allowing use of jax.core
after importing jax
.
---> 37 import jax.core as _core
38 del _core
40 # Note: import
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/core.py:18
1 # Copyright 2022 The JAX Authors.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
15 # Note: import
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/core.py:39 35 from weakref import ref 37 import numpy as np ---> 39 from jax._src import dtypes 40 from jax._src import config 41 from jax._src import effects
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/dtypes.py:33 30 import ml_dtypes 31 import numpy as np ---> 33 from jax._src import config 34 from jax._src.typing import DType, DTypeLike 35 from jax._src.util import set_module, StrictABC
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/config.py:27 24 import threading 25 from typing import Any, Callable, Generic, NamedTuple, NoReturn, TypeVar, cast ---> 27 from jax._src import lib 28 from jax._src.lib import jax_jit 29 from jax._src.lib import transfer_guard_lib
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/lib/init.py:75 71 return _jaxlib_version 74 version_str = jaxlib.version.version ---> 75 version = check_jaxlib_version( 76 jax_version=jax.version.version, 77 jaxlib_version=jaxlib.version.version, 78 minimum_jaxlib_version=jax.version._minimum_jaxlib_version) 80 # Before importing any C compiled modules from jaxlib, first import the CPU 81 # feature guard module to verify that jaxlib was compiled in a way that only 82 # uses instructions that are present on this machine. 83 import jaxlib.cpu_feature_guard as cpu_feature_guard
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/jax/_src/lib/init.py:67, in check_jaxlib_version(jax_version, jaxlib_version, minimum_jaxlib_version) 64 raise RuntimeError(msg) 66 if _jaxlib_version > _jax_version: ---> 67 raise RuntimeError( 68 f'jaxlib version {jaxlib_version} is newer than and ' 69 f'incompatible with jax version {jax_version}. Please ' 70 'update your jax and/or jaxlib packages.') 71 return _jaxlib_version
RuntimeError: jaxlib version 0.4.31 is newer than and incompatible with jax version 0.4.27. Please update your jax and/or jaxlib packages.
It seems the Windows prebuilt are experimental for jax. You have to install jaxlib and jax in the same version pip install jaxlib==0.4.27
. It's not straightforward to track this in the pyproject.toml file as far as I'm aware.
Thank you, I managed to get everything installed and the setup.anndata() works now however later when I run
model_cls = scvi.model.SCVI
model_cls.setup_anndata(adata, categorical_covariate_keys = ['Sample'],continuous_covariate_keys=['pct_counts_mt', 'pct_counts_ribo'])
tuner = ModelTuner(model_cls)
results = tuner.fit(adata, metric="validation_loss",
resources = {'gpu': 1}, #have to specify gpu or might not use
search_space = search_space,
num_samples = 100,
max_epochs = 20)
I get this error:
---------------------------------------------------------------------------
DeprecationWarning Traceback (most recent call last)
Cell In[16], [line 1](vscode-notebook-cell:?execution_count=16&line=1)
----> [1](vscode-notebook-cell:?execution_count=16&line=1) results = tuner.fit(adata, metric="validation_loss",
[2](vscode-notebook-cell:?execution_count=16&line=2) resources = {'gpu': 1}, #have to specify gpu or might not use
[3](vscode-notebook-cell:?execution_count=16&line=3) search_space = search_space,
[4](vscode-notebook-cell:?execution_count=16&line=4) num_samples = 100,
[5](vscode-notebook-cell:?execution_count=16&line=5) max_epochs = 20)
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:115, in ModelTuner.fit(self, adata, **kwargs)
[33](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:33) def fit(self, adata: AnnOrMuData, **kwargs) -> None:
[34](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:34) """Run a specified hyperparameter sweep for the associated model class.
[35](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:35)
[36](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:36) Parameters
(...)
[113](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:113) A dataclass containing the results of the tuning experiment.
[114](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:114) """
--> [115](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:115) tuner, config = self._manager._get_tuner(adata, **kwargs)
[116](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:116) results = tuner.fit()
[117](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_tuner.py:117) return self._manager._get_analysis(results, config)
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:558, in TunerManager._get_tuner(self, adata, metric, additional_metrics, search_space, model_kwargs, train_kwargs, use_defaults, num_samples, max_epochs, scheduler, scheduler_kwargs, searcher, searcher_kwargs, reporter, resources, seed, experiment_name, logging_dir, monitor_device_stats)
[538](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:538) _trainable = self._get_trainable(
[539](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:539) adata,
[540](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:540) _metrics,
(...)
[550](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:550) monitor_device_stats=monitor_device_stats,
[551](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:551) )
[553](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:553) tune_config = tune.tune_config.TuneConfig(
[554](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:554) scheduler=_scheduler,
[555](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:555) search_alg=_searcher,
[556](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:556) num_samples=num_samples,
[557](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:557) )
--> [558](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:558) run_config = air.config.RunConfig(
[559](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:559) name=_experiment_name,
[560](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:560) local_dir=_logging_dir,
[561](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:561) progress_reporter=_reporter,
[562](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:562) log_to_file=True,
[563](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:563) verbose=1,
[564](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:564) )
[565](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:565) tuner = tune.Tuner(
[566](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:566) trainable=_trainable,
[567](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:567) param_space=_search_space,
[568](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:568) tune_config=tune_config,
[569](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:569) run_config=run_config,
[570](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:570) )
[571](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:571) config = {
[572](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:572) "metrics": _metrics,
[573](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:573) "search_space": _search_space,
[574](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/scvi/autotune/_manager.py:574) }
File <string>:15, in __init__(self, name, storage_path, storage_filesystem, failure_config, checkpoint_config, sync_config, verbose, stop, callbacks, progress_reporter, log_to_file, local_dir)
File /fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:665, in RunConfig.__post_init__(self)
[662](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:662) from ray.tune.experimental.output import AirVerbosity, get_air_verbosity
[664](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:664) if self.local_dir is not None:
--> [665](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:665) raise DeprecationWarning(
[666](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:666) "The `RunConfig(local_dir)` argument is deprecated. "
[667](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:667) "You should set the `RunConfig(storage_path)` instead."
[668](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:668) "See the docs: https://docs.ray.io/en/latest/train/user-guides/"
[669](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:669) "persistent-storage.html#setting-the-local-staging-directory"
[670](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:670) )
[672](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:672) if self.storage_path is None:
[673](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:673) # TODO(justinvyu): [Deprecated] Remove in 2.30
[674](https://vscode-remote+ssh-002dremote-002bmax-002dlogin-002emdc-002dberlin-002enet.vscode-resource.vscode-cdn.net/fast/AG_Metzger/philipp/bins/miniforge3/envs/scvi_ray/lib/python3.11/site-packages/ray/air/config.py:674) self.storage_path = DEFAULT_STORAGE_PATH
DeprecationWarning: The `RunConfig(local_dir)` argument is deprecated. You should set the `RunConfig(storage_path)` instead.See the docs: https://docs.ray.io/en/latest/train/user-guides/persistent-storage.html#setting-the-local-staging-directory
Great that it worked. This is a duplicate of #2908. Please see there for fixes.
Hi
The running code is as follows:
`model_cls = scvi.model.SCVI model_cls.setup_anndata(adata, categorical_covariate_keys = ['Sample'], continuous_covariate_keys=['pct_counts_mt'])
tuner = ModelTuner(model_cls)
search_space = { "n_hidden": tune.choice([92, 128, 192, 256]), "n_latent": tune.choice([10, 20, 30, 40, 50, 60,70,80]), "n_layers": tune.choice([1, 2, 3]), "lr": tune.loguniform(1e-4, 1e-2), "gene_likelihood": tune.choice(["nb", "zinb"])}
results = tuner.fit( adata, metric="validation_loss", search_space=search_space, num_samples=30, max_epochs=20, resources={"cpu": 10, "gpu": 1}, ) `
The following error message is displayed:
symbolize_win32.inc : 53] raw: syminitialize() failed: 87 fatal python error: aborted
For the above question, could you provide some debugging suggestions? Thank you for your valuable time and assistance. I sincerely look forward to your response!