search

aertslab / pySCENIC

pySCENIC is a lightning-fast python implementation of the SCENIC pipeline (Single-Cell rEgulatory Network Inference and Clustering) which enables biologists to infer transcription factors, gene regulatory networks and cell types from single-cell RNA-seq data.
http://scenic.aertslab.org
GNU General Public License v3.0
424 stars 179 forks source link

[BUG] A new bug of pysenic grn function #539

Closed HelloWorldLTY closed 5 months ago

HelloWorldLTY commented 5 months ago

Describe the bug Yesterday, my program still worked well. However, it seems that I suddenly meet an error when running

!pyscenic grn rna_wtperturb.loom tfs_wtperturb.txt \
    -o draft_grn_wt.csv --seed 0 --num_workers 2  \
    --cell_id_attribute cells --gene_attribute genes

All the inputs are correct, and I can successfully run it previously.

The error is very long, that is:

PermissionError: [Errno 13] Permission denied: '/tmp/dask-worker-space/worker-0yc3ftcz'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/asyncio/tasks.py", line 688, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/core.py", line 489, in start
    raise RuntimeError(f"{type(self).__name__} failed to start.") from exc
RuntimeError: Nanny failed to start.
Traceback (most recent call last):
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/bin/pyscenic", line 8, in <module>
    sys.exit(main())
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/pyscenic/cli/pyscenic.py", line 675, in main
    args.func(args)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/pyscenic/cli/pyscenic.py", line 86, in find_adjacencies_command
    client, shutdown_callback = _prepare_client(args.client_or_address, num_workers=args.num_workers)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/pyscenic/prune.py", line 63, in _prepare_client
    local_cluster = LocalCluster(n_workers=num_workers, threads_per_worker=1)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/deploy/local.py", line 238, in __init__
    super().__init__(
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/deploy/spec.py", line 277, in __init__
    self.sync(self._correct_state)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/utils.py", line 338, in sync
    return sync(
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/utils.py", line 405, in sync
    raise exc.with_traceback(tb)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/utils.py", line 378, in f
    result = yield future
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/deploy/spec.py", line 369, in _correct_state_internal
    await w  # for tornado gen.coroutine support
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/core.py", line 470, in start
    raise self.__startup_exc
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/core.py", line 481, in start
    await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/nanny.py", line 350, in start_unsafe
    response = await self.instantiate()
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/nanny.py", line 423, in instantiate
    result = await self.process.start()
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/nanny.py", line 671, in start
    msg = await self._wait_until_connected(uid)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/nanny.py", line 799, in _wait_until_connected
    raise msg["exception"]
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/nanny.py", line 829, in _run
    worker = Worker(**worker_kwargs)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/worker.py", line 579, in __init__
    self._workdir = self._workspace.new_work_dir(prefix="worker-")
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/diskutils.py", line 248, in new_work_dir
    return WorkDir(self, **kwargs)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/site-packages/distributed/diskutils.py", line 48, in __init__
    self.dir_path = tempfile.mkdtemp(prefix=prefix, dir=workspace.base_dir)
  File "/gpfs/gibbs/project/zhao/tl688/conda_envs/scglue/lib/python3.9/tempfile.py", line 363, in mkdtemp
    _os.mkdir(file, 0o700)
PermissionError: [Errno 13] Permission denied: '/tmp/dask-worker-space/worker-tejoo7z0'

Steps to reproduce the behavior

  1. Command run when the error occurred:

    ...

  2. Error encountered:

    ...

Expected behavior A clear and concise description of what you expected to happen.

The error is described above

I used the updated pySENIC. Thanks.

Please complete the following information:

HelloWorldLTY commented 5 months ago

Interestingly, when I open the same program after 8 hours, this problem was resolved... But I am still confused about the source of errors.

ghuls commented 5 months ago

Maybe there was another user on your node that was also running dask and had already created /tmp/dask-worker-space/. This can happen if /tmp is shared between users.

%%bash
Create a custom TMPDIR.
custom_tmp_dir='/tmp/HelloWorldLTY'
mkdir -p "${custom_tmp_dir}"

# Run pySCENIC with a custom TMPDIR.
TMPDIR="${custom_tmp_dir}" pyscenic grn rna_wtperturb.loom tfs_wtperturb.txt \
    -o draft_grn_wt.csv --seed 0 --num_workers 2  \
    --cell_id_attribute cells --gene_attribute genes
HelloWorldLTY commented 5 months ago

Thanks a lot, that approach looks better.