Bug report [Error with Ray while running pseudobulk_peak_calling]

[lila167]

Describe the bug Hi, I try to run export_pseudobulk and get the following error:

OSError: AF_UNIX path length cannot exceed 107 bytes: '/nfs/XXX/tmp/session_2023-09-15_10-04-33_759150_24462/sockets/plasma_store'

To Reproduce This is what I run:

from pycisTopic.pseudobulk_peak_calling import *
bw_paths, bed_paths = export_pseudobulk(input_data = cell_data,
                 variable = 'ann',
                 sample_id_col = 'sample',
                 chromsizes = chromsizes,
                 bed_path = outDir + 'cpc/pseudobulk_bed_files/',
                 bigwig_path = outDir + 'cpc/pseudobulk_bw_files/',
                 path_to_fragments = fragments_dict,
                 n_cpu = 5,
                 normalize_bigwig = True,
                 remove_duplicates = True,
                 _temp_dir = tmpDir,
                 split_pattern = '-',
                 use_polars = True) 

Error output

2023-09-15 09:59:58,391 cisTopic     INFO     Reading fragments from /nfs/XXX/atac_fragments.tsv.gz
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[40], line 2
      1 from pycisTopic.pseudobulk_peak_calling import *
----> 2 bw_paths, bed_paths = export_pseudobulk(input_data = cell_data,
      3                  variable = 'Hana_ann',
      4                  sample_id_col = 'sample',
      5                  chromsizes = chromsizes,
      6                  bed_path = outDir + 'cpc/pseudobulk_bed_files/',
      7                  bigwig_path = outDir + 'cpc/pseudobulk_bw_files/',
      8                  path_to_fragments = fragments_dict,
      9                  n_cpu = 5,
     10                  normalize_bigwig = True,
     11                  remove_duplicates = True,
     12                  _temp_dir = tmpDir,
     13                  split_pattern = '-',
     14                  use_polars = True) 

File /nfs/XXX/miniconda3/envs/data-an/lib/python3.9/site-packages/pycisTopic/pseudobulk_peak_calling.py:165, in export_pseudobulk(input_data, variable, chromsizes, bed_path, bigwig_path, path_to_fragments, sample_id_col, n_cpu, normalize_bigwig, remove_duplicates, split_pattern, use_polars, **kwargs)
    163 # Create pseudobulks
    164 if n_cpu > 1:
--> 165     ray.init(num_cpus=n_cpu, **kwargs)
    166     ray_handle = ray.wait(
    167         [
    168             export_pseudobulk_ray.remote(
   (...)
    182         num_returns=len(groups),
    183     )
    184     ray.shutdown()

File /nfs/XXX/miniconda3/envs/data-an/lib/python3.9/site-packages/ray/_private/client_mode_hook.py:103, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    101     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    102         return getattr(ray, func.__name__)(*args, **kwargs)
--> 103 return func(*args, **kwargs)

File /nfs/XXX/miniconda3/envs/data-an/lib/python3.9/site-packages/ray/_private/worker.py:1514, in init(address, num_cpus, num_gpus, resources, labels, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, log_to_driver, namespace, runtime_env, storage, **kwargs)
   1480     ray_params = ray._private.parameter.RayParams(
   1481         node_ip_address=node_ip_address,
   1482         raylet_ip_address=raylet_ip_address,
   (...)
   1508         node_name=_node_name,
   1509     )
   1510     # Start the Ray processes. We set shutdown_at_exit=False because we
   1511     # shutdown the node in the ray.shutdown call that happens in the atexit
   1512     # handler. We still spawn a reaper process in case the atexit handler
   1513     # isn't called.
-> 1514     _global_node = ray._private.node.Node(
   1515         head=True,
   1516         shutdown_at_exit=False,
   1517         spawn_reaper=True,
   1518         ray_params=ray_params,
   1519     )
   1520 else:
   1521     # In this case, we are connecting to an existing cluster.
   1522     if num_cpus is not None or num_gpus is not None:

File /nfs/XXX/miniconda3/envs/data-an/lib/python3.9/site-packages/ray/_private/node.py:248, in Node.__init__(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only, default_worker)
    245         self._ray_params.node_manager_port = node_info["node_manager_port"]
    246 else:
    247     # If the user specified a socket name, use it.
--> 248     self._plasma_store_socket_name = self._prepare_socket_file(
    249         self._ray_params.plasma_store_socket_name, default_prefix="plasma_store"
    250     )
    251     self._raylet_socket_name = self._prepare_socket_file(
    252         self._ray_params.raylet_socket_name, default_prefix="raylet"
    253     )
    255 self.metrics_agent_port = self._get_cached_port(
    256     "metrics_agent_port", default_port=ray_params.metrics_agent_port
    257 )

File /nfs/XXX/miniconda3/envs/data-an/lib/python3.9/site-packages/ray/_private/node.py:820, in Node._prepare_socket_file(self, socket_path, default_prefix)
    818     maxlen = (104 if is_mac else 108) - 1  # sockaddr_un->sun_path
    819     if len(result.split("://", 1)[-1].encode("utf-8")) > maxlen:
--> 820         raise OSError(
    821             f"AF_UNIX path length cannot exceed {maxlen} bytes: {result!r}"
    822         )
    823 return result

OSError: AF_UNIX path length cannot exceed 107 bytes: '/nfs/team292/XXX/tmp/session_2023-09-15_10-04-33_759150_24462/sockets/plasma_store'

Version (please complete the following information):

• Python: 3.9.16
• pycistopic: '1.0.3.dev18+ge563fb6'
• Ray: '2.6.3'

[lila167]

Also, running the following code gives the same error:

import ray
ray.init(num_cpus = 1, _temp_dir = os.path.join(tmp_dir + 'ray_spill'))
ray.shutdown()

[SeppeDeWinter]

Hi @lila167

The path of your tmp_dir is too long (it contains too many characters). This is a known issue with ray: https://github.com/ray-project/ray/issues/7724

This can be fixed by pointing the tmp_dir to a different directory (with a short path) or using a symlink.

Best,

Seppe

[lila167]

Thank Seppe for your quick response. I could fix that. But still I get many other errors with Ray. I'm not sure where the inconsistency comes from. I just deactivated Ray with n_cpus = 1 and it works now.

Thanks, Hana