Error when creating the data universe (10.1164/rccm.201712-2410OC)

[alex-d13]

Describe the bug I am getting an error message when I try to create the data universe:

Error in checkForRemoteErrors(lapply(cl, recvResult)) : 
  one node produced an error: ValueError: '53-year-old human stage' is not a valid entry for development_stage in data set ['10.1164/rccm.201712-2410OC'].

Detailed traceback:
  File "/home/.cache/R/basilisk/1.4.0/SimBu/0.99.0/sfaira_env2/lib/python3.9/site-packages/sfaira/data/dataloaders/super_group.py", line 31, in __init__
    DatasetSuperGroupLoaders(
  File "/home/.cache/R/basilisk/1.4.0/SimBu/0.99.0/sfaira_env2/lib/python3.9/site-packages/sfaira/data/dataloaders/loaders/super_group.py", line 39, in __init__
    dsg = DatasetGroupDirectoryOriented(
  File "/home/.cache/R/basilisk/1.4.0/SimBu/0.99.0/sfaira_env2/lib/python3.9/site-packages/sfaira/data/dataloaders/base/dataset_group.py", line 773, in __init__
    DatasetBase(
  File "/home/.cache/R/basilisk/1.4.0/SimBu/0.99.0/sfaira_env2/lib/python3.9/site-packages/sfaira/data/dataloaders/base/dataset.py", line 310, in __init__
    set

To Reproduce I am using sfaira in R via reticulate and a conda environment where only sfaira and its dependencies are installed. This is how I am calling sfaira in R (though this error message does not seem to be R related I feel like..):

sfaira <- reticulate::import("sfaira")  
ds <- sfaira$data$Universe(data_path = paste0(tempdir(), "/raw"),
                             meta_path = paste0(tempdir(), "/meta"),
                             cache_path = paste0(tempdir(), "/cache"))

I had sfaira working previously with this setup when I was using the development version 0.3.9+4.gdd849d5 and wanted to upgrade to 0.3.12.

System [please complete the following information]:

• OS: Ubuntu 20.04.4
• Language Version: Python 3.9
• Virtual environment: Conda
• Sfaira version: 0.3.12

Cheers, Alex

[davidsebfischer]

Hi Alex! My first guess would be that your ontology cache is outdated, could you check in your cache directory (ie most likely ~/.cache/sfaira/ontologies/hsapdv) and delete the entries under hsapdv? It ll automatically recache then. This ontology is hard to version unfortunately which causes this issue.

[alex-d13]

Hi, I tried that, but get the same error. It also did not recache the files in this directory, it remains empty. During which step is the recaching usually taking place?

[davidsebfischer]

This would happen upon first usage of the ontology, which is before the error above, is it potentially in the sfaira repo under ./.cache?

[alex-d13]

I solved the error above: I created a new conda environment after deleting the mentioned ontology, that created the new version of this file. Unfortunately, I now have a new error:

Error in checkForRemoteErrors(lapply(cl, recvResult)) : 
  one node produced an error: ErrorDuringImport: problem in sfaira.data.dataloaders.loaders.dno_doi_10x_genomics.human_blood_2019_10xsequencing_10xgenomics_001 - ModuleNotFoundError: No module named 'tables'

Detailed traceback:
  File "/home/.cache/R/basilisk/1.4.0/SimBu/0.99.0/sfaira_env_12/lib/python3.9/site-packages/sfaira/data/dataloaders/super_group.py", line 31, in __init__
    DatasetSuperGroupLoaders(
  File "/home/.cache/R/basilisk/1.4.0/SimBu/0.99.0/sfaira_env_12/lib/python3.9/site-packages/sfaira/data/dataloaders/loaders/super_group.py", line 39, in __init__
    dsg = DatasetGroupDirectoryOriented(
  File "/home/.cache/R/basilisk/1.4.0/SimBu/0.99.0/sfaira_env_12/lib/python3.9/site-packages/sfaira/data/dataloaders/base/dataset_group.py", line 740, in __init__
    DatasetFound = pydoc.locate(loader_pydoc_path + collection_id + "." + file_module + ".Dataset")
  File "/home/.cache

[davidsebfischer]

• [x] We ll add tables to the requirements, for now try installing that package into your environment manually?

[alex-d13]

Yep, thats what I tried and it just finished without any errors :) I used tables==3.7.0.

Thank your for the fast responses!