Bug report [BUG] - Githubissues

TomVuod commented 1 year ago

Hello, I have the following error when I try to run UMAP on cistopic object:

File "VCaP_scenic_plus.py", line 275, in <module>
    cistopic_obj = find_topics(adata, sample_names, paths_to_fragments, work_dir, tmp_dir, paths_to_peak_matrix, n_cpu)
  File "VCaP_scenic_plus.py", line 96, in find_topics
    run_umap(cistopic_obj, target  = 'cell', scale=True, low_memory = False, verbose = False)
  File "/xxxxxxx/home/xxxxxxx/.local/lib/python3.8/site-packages/pycisTopic/clust_vis.py", line 241, in run_umap
    embedding = reducer.fit_transform(data_mat)
  File "/xxxxxxx/home/xxxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/umap/umap_.py", line 2772, in fit_transform
    self.fit(X, y)
  File "/xxxxxxx/home/xxxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/umap/umap_.py", line 2516, in fit
    ) = nearest_neighbors(
  File "/xxxxxxx/home/xxxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/umap/umap_.py", line 328, in nearest_neighbors
    knn_search_index = NNDescent(
  File "/xxxxxxx/home/xxxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/pynndescent/pynndescent_.py", line 921, in __init__
    self._neighbor_graph = nn_descent(
  File "/xxxxxxx/home/xxxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/xxxxxxx/home/xxxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name 'print': Cannot determine Numba type of <class 'function'>

File "../../../home/xxxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/pynndescent/pynndescent_.py", line 252:
def nn_descent_internal_low_memory_parallel(
    <source elided>
        if verbose:
            print("\t", n + 1, " / ", n_iters)
            ^

During: resolving callee type: type(CPUDispatcher(<function nn_descent_internal_low_memory_parallel at 0x2aaad9903ca0>))
During: typing of call at /xxxxxxx/home/xxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/pynndescent/pynndescent_.py (358)

During: resolving callee type: type(CPUDispatcher(<function nn_descent_internal_low_memory_parallel at 0x2aaad9903ca0>))
During: typing of call at /xxxxxxx/home/xxxxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/pynndescent/pynndescent_.py (358)

File "../../../home/xxxxx/mambaforge/envs/scenic/lib/python3.8/site-packages/pynndescent/pynndescent_.py", line 358:
def nn_descent(
    <source elided>
    if low_memory:
        nn_descent_internal_low_memory_parallel(
        ^

Type of the print is recognized as <class 'function'> instead of <class 'builtin_function_or_method'>. Probably the logger overwrites the builtin function since the error does not appear when I save cistopic object, load it in a separate process and then run UMAP, but then I have the error from run_pycistarget:

2023-08-31 03:36:25,352 ERROR services.py:1195 -- Failed to start the dashboard: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/xxxxxxx/scratch/u/xxxxxxx/V/ray_spill/session_2023-08-31_03-36-17_692267_15574/logs/dashboard.log'
2023-08-31 03:36:25,353 ERROR services.py:1196 -- Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/xxxxxxx/scratch/u/xxxxxxx/V/ray_spill/session_2023-08-31_03-36-17_692267_15574/logs/dashboard.log'
Traceback (most recent call last):
  File "/xxxxxxx/home/xxxxxxx/anaconda3/envs/scenic_plus/lib/python3.8/site-packages/ray/_private/services.py", line 1167, in start_api_server
    with open(dashboard_log, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/xxxxxxx/scratch/u/xxxxxxx/V/ray_spill/session_2023-08-31_03-36-17_692267_15574/logs/dashboard.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/xxxxxxx/home/xxxxxxx/anaconda3/envs/scenic_plus/lib/python3.8/site-packages/ray/_private/services.py", line 1178, in start_api_server
    raise Exception(err_msg + f"\nFailed to read dashboard log: {e}")
Exception: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/xxxxxxx/scratch/u/xxxxxxx/V/ray_spill/session_2023-08-31_03-36-17_692267_15574/logs/dashboard.log'
2023-08-31 03:36:26,844 INFO worker.py:1538 -- Started a local Ray instance.
[2023-08-31 03:36:55,501 E 15574 15574] core_worker.cc:179: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

python==3.8.16 pycisTopic==1.0.3.dev18+ge563fb6 numba==0.57.1

SeppeDeWinter commented 1 year ago

Hi @TomVuod

For the ray related error. Could you check wether the temporary directory (_temp_dir) exists?

You could try troubleshooting using:


import ray

ray.init(
 num_cpus = 5,
  _temp_dir = os.path.join(tmp_dir + 'ray_spill'))

Best,

Seppe

TomVuod commented 1 year ago

Thanks @SeppeDeWinter fot the response. The temporary directory exists. I also found that downgrading ray to 2.5.0 solves the problem.

aertslab / pycisTopic

Bug report [BUG] #91