search

raw-lab / mercat2

MerCat2: python code for versatile k-mer counting and diversity estimation for database independent property analysis for metaome data
https://github.com/raw-lab/mercat2/
BSD 3-Clause "New" or "Revised" License
11 stars 1 forks source link

Failed to start dashboard / cannot import name 'packaging' from 'pkg_resources' / Unable to register worker with raylet. No such file or directory #13

Closed kreil closed 4 months ago

kreil commented 4 months ago

I am trying to run mercat2 on SwissProt. In a little test invocation,

   mercat2.py -i uniprot_sprot.fasta -k1 -c1 -skipclean -o k1

I get the below errors.

I also tried starting a Ray cluster manually by

   ray start --head

but that gives me another error as below.

What am I doing wrong?

Many thanks, David.

Error messages from mercat2 alone:

Starting MerCat2 v1.4.1 with k-mer 1 and 48 threads

2024-05-23 19:52:54,205 ERROR services.py:1207 -- Failed to start the dashboard , return code 1
2024-05-23 19:52:54,206 ERROR services.py:1232 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2024-05-23 19:52:54,206 ERROR services.py:1276 -- 
The last 20 lines of /tmp/ray/session_2024-05-23_19-52-52_585467_40696/logs/dashboard.log (it contains the error message from the dashboard): 
  File "/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/site-packages/ray/dashboard/dashboard.py", line 75, in run
    await self.dashboard_head.run()
  File "/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/site-packages/ray/dashboard/head.py", line 322, in run
    modules = self._load_modules(self._modules_to_load)
  File "/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/site-packages/ray/dashboard/head.py", line 219, in _load_modules
    head_cls_list = dashboard_utils.get_all_modules(DashboardHeadModule)
  File "/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/site-packages/ray/dashboard/utils.py", line 121, in get_all_modules
    importlib.import_module(name)
  File "/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/site-packages/ray/dashboard/modules/dashboard_sdk.py", line 10, in <module>
    from pkg_resources import packaging
ImportError: cannot import name 'packaging' from 'pkg_resources' (/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/site-packages/pkg_resources/__init__.py)
2024-05-23 19:52:54,252 INFO worker.py:1621 -- Started a local Ray instance.
[2024-05-23 19:52:54,936 E 40696 40696] core_worker.cc:201: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

Error messages from mercat2 after first starting a Ray head node:

Starting MerCat2 v1.4.1 with k-mer 1 and 48 threads

2024-05-23 20:23:58,024 INFO worker.py:1431 -- Connecting to existing Ray cluster at address: 141.244.140.16:6379...
Traceback (most recent call last):
  File "/bi/home/dkreil/.conda/envs/mercat2/bin/mercat2.py", line 508, in <module>
    mercat_main()
  File "/bi/home/dkreil/.conda/envs/mercat2/bin/mercat2.py", line 217, in mercat_main
    ray.init(num_cpus=m_num_cores, log_to_driver=DEBUG)
  File "/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/bi/home/dkreil/.conda/envs/mercat2/lib/python3.10/site-packages/ray/_private/worker.py", line 1523, in init
    raise ValueError(
ValueError: When connecting to an existing cluster, num_cpus and num_gpus must not be provided.
raw-lab commented 4 months ago

You can't run a k-mer length of 1 on MerCat2. It has to be >1.

For viruses/phages: classically you can use 2. For Bacteria/archaea: classically you can use 4.

If you looking for signature differences I would use 31 k especially for bacteria/archaea.

Paper from Forest Rohwer on this: https://doi.org/10.1111/j.1462-2920.2009.01901.x

I am going to close this. As k-mer counting of 1 isn't common. Let us know if you have further questions?

Thank you for using MerCat2

raw937 commented 4 months ago

This is a ray issue. Which we have fixed. You need a manual install for now - until conda updates it. Which we will let you know. So do this.

This will delete the bug mercat2

conda remove -n mercat2 --all

Install manual

cd mercat2
bash install_mercat2.sh
conda activate mercat2-1.4.1