ImportError: cannot import name 'packaging' from 'pkg_resources' for Model Tuning using Ray

[shahrozeabbas]

Hello,

I am trying to use the autotune module for a parameter search but am getting an error that packaging cannot be imported. I'm using a conda environment and pkg_resources is installed. I usedpip to install scvi with the autotune dependencies. Below is a copy of my script and the error message.

adata = scanpy.read_h5ad('anndata_object.h5ad')

scvi.data.poisson_gene_selection(adata, accelerator='gpu', n_top_genes=2000, subset=True)

adata.layers['counts'] = adata.X.copy()

model = scvi.model.SCVI
model.setup_anndata(adata, layer='counts')

tuner = autotune.ModelTuner(model)

search_space = {
    "n_latent": ray.tune.choice([10, 20, 30]),
    "n_layers": ray.tune.choice([1, 2]),
    "lr": ray.tune.loguniform(1e-4, 1e-2),
    "gene_likelihood": ray.tune.choice(['zinb', 'nb']),
    "lr_factor": ray.tune.loguniform(0.01, 0.05, 0.1)
}

ray.init(log_to_driver=False)

results = tuner.fit(
    adata,
    metric="validation_loss",
    search_space=search_space,
    num_samples=10,
    max_epochs=300,
    resources={"cpu": 2, "gpu": 1},
)

When initializing ray:

/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/subprocess.py:1770: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
  self.pid = _posixsubprocess.fork_exec(
2024-05-29 10:45:09,740 ERROR services.py:1329 -- Failed to start the dashboard 
2024-05-29 10:45:09,740 ERROR services.py:1354 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2024-05-29 10:45:09,740 ERROR services.py:1398 -- 
The last 20 lines of /tmp/ray/session_2024-05-29_10-44-48_483613_11785/logs/dashboard.log (it contains the error message from the dashboard): 
2024-05-29 10:45:05,782 INFO utils.py:123 -- Module ray.dashboard.modules.actor.actor_head cannot be loaded because we cannot import all dependencies. Install this module using `pip install 'ray[default]'` for the full dashboard functionality. Error: No module named 'opencensus'

2024-05-29 10:45:10,971 INFO worker.py:1724 -- Started a local Ray instance.
RayContext(dashboard_url=None, python_version='3.9.19', ray_version='2.9.3', ray_commit='62655e11ed76509b78654b60be67bc59f8f3460a', protocol_version=None)

It seems like there's some errors regarding a dashboard, but maybe I don't need that?

Then when running tuner.fit:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/scvi/autotune/_tuner.py", line 115, in fit
    tuner, config = self._manager._get_tuner(adata, **kwargs)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/scvi/autotune/_manager.py", line 538, in _get_tuner
    _trainable = self._get_trainable(
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/scvi/autotune/_manager.py", line 455, in _get_trainable
    _wrap_params = tune.with_parameters(
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/tune/trainable/util.py", line 109, in with_parameters
    parameter_registry.put(prefix + k, v)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/tune/registry.py", line 296, in put
    self.flush()
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/tune/registry.py", line 308, in flush
    self.references[k] = ray.put(v)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/_private/worker.py", line 2697, in put
    object_ref = worker.put_object(value, owner_address=serialize_owner_address)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/_private/worker.py", line 753, in put_object
    serialized_value = self.get_serialization_context().serialize(value)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/_private/worker.py", line 678, in get_serialization_context
    context_map[job_id] = serialization.SerializationContext(self)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/_private/serialization.py", line 153, in __init__
    serialization_addons.apply(self)
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/util/serialization_addons.py", line 29, in apply
    from ray._private.pydantic_compat import register_pydantic_serializers
  File "/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/ray/_private/pydantic_compat.py", line 2, in <module>
    from pkg_resources import packaging
ImportError: cannot import name 'packaging' from 'pkg_resources' (/data/abbass2/miniforge3/envs/scvi-tune/lib/python3.9/site-packages/pkg_resources/__init__.py)

I've also attached a copy of my environment file: scvi-tune.txt

Appreciate any advice on this, thanks!

[martinkim0]

Hi, could you try downgrading setuptools from 70.0.0 to 69.5.1 (following the pointers here):

pip install -U setuptools==69.5.1

[shahrozeabbas]

Success! Thank you

[shahrozeabbas]

autotune should work with other models like poissonvi right? Just to confirm.

[martinkim0]

Yes (sorry for the delay)!