search

snap-stanford / UCE

UCE is a zero-shot foundation model for single-cell gene expression data
MIT License
136 stars 21 forks source link

`Unexpected key(s) in state_dict: "transformer_encoder.layers...` #41

Closed bschilder closed 3 months ago

bschilder commented 3 months ago

Hi @Yanay1 ,

I seem to be running into some issues with the 33-layer model that I hadn't observed with the 4-layer model.

It may be due to some versioning conflict. If so, would you mind sharing the exact versions of the packages you used? (especially the ones listed here)

Command

!python eval_single_anndata.py --adata_path data/ot_l2g.h5ad --dir output/ot/ --species human --model_loc model_files/33l_8ep_1024t_1280.torch --batch_size 25 --filter False --skip False

Output

Proccessing ot_l2g
944.4519
ot_l2g (11963, 18160)
Wrote Shapes Dict
18160
Max Code: 612
Traceback (most recent call last):
  File "/shared/bms20/projects/UCE/eval_single_anndata.py", line 155, in <module>
    main(args, accelerator)
  File "/shared/bms20/projects/UCE/eval_single_anndata.py", line 85, in main
    processor.run_evaluation()
  File "/shared/bms20/projects/UCE/evaluate.py", line 145, in run_evaluation
    run_eval(self.adata, self.name, self.pe_idx_path, self.chroms_path,
  File "/shared/bms20/projects/UCE/evaluate.py", line 203, in run_eval
    model.load_state_dict(torch.load(args.model_loc, map_location="cpu"),
  File "/home/bms20/anaconda3/envs/uce/lib/python3.12/site-packages/torch/nn/modules/module.py", line 2189, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for TransformerModel:
    Unexpected key(s) in state_dict: "transformer_encoder.layers.4.self_attn.in_proj_weight", "transformer_encoder.layers.4.self_attn.in_proj_bias", "transformer_encoder.layers.4.self_attn.out_proj.weight", "transformer_encoder.layers.4.self_attn.out_proj.bias", "transformer_encoder.layers.4.linear1.weight", "transformer_encoder.layers.4.linear1.bias", "transformer_encoder.layers.4.linear2.weight", "transformer_encoder.layers.4.linear2.bias", "transformer_encoder.layers.4.norm1.weight", "transformer_encoder.layers.4.norm1.bias", "transformer_encoder.layers.4.norm2.weight", "transformer_encoder.layers.4.norm2.bias", "transformer_encoder.layers.5.self_attn.in_proj_weight", "transformer_encoder.layers.5.self_attn.in_proj_bias", "transformer_encoder.layers.5.self_attn.out_proj.weight", "transformer_encoder.layers.5.self_attn.out_proj.bias", "transformer_encoder.layers.5.linear1.weight", "transformer_encoder.layers.5.linear1.bias", "transformer_encoder.layers.5.linear2.weight", "transformer_encoder.layers.5.linear2.bias", "transformer_encoder.layers.5.norm1.weight", "transformer_encoder.layers.5.norm1.bias", "transformer_encoder.layers.5.norm2.weight", "transformer_encoder.layers.5.norm2.bias", "transformer_encoder.layers.6.self_attn.in_proj_weight", "transformer_encoder.layers.6.self_attn.in_proj_bias", "transformer_encoder.layers.6.self_attn.out_proj.weight", "transformer_encoder.layers.6.self_attn.out_proj.bias", "transformer_encoder.layers.6.linear1.weight", "transformer_encoder.layers.6.linear1.bias", "transformer_encoder.layers.6.linear2.weight", "transformer_encoder.layers.6.linear2.bias", "transformer_encoder.layers.6.norm1.weight", "transformer_encoder.layers.6.norm1.bias", "transformer_encoder.layers.6.norm2.weight", "transformer_encoder.layers.6.norm2.bias", "transformer_encoder.layers.7.self_attn.in_proj_weight", "transformer_encoder.layers.7.self_attn.in_proj_bias", "transformer_encoder.layers.7.self_attn.out_proj.weight", "transformer_encoder.layers.7.self_attn.out_proj.bias", "transformer_encoder.layers.7.linear1.weight", "transformer_encoder.layers.7.linear1.bias", "transformer_encoder.layers.7.linear2.weight", "transformer_encoder.layers.7.linear2.bias", "transformer_encoder.layers.7.norm1.weight", "transformer_encoder.layers.7.norm1.bias", "transformer_encoder.layers.7.norm2.weight", "transformer_encoder.layers.7.norm2.bias", "transformer_encoder.layers.8.self_attn.in_proj_weight", "transformer_encoder.layers.8.self_attn.in_proj_bias", "transformer_encoder.layers.8.self_attn.out_proj.weight", "transformer_encoder.layers.8.self_attn.out_proj.bias", "transformer_encoder.layers.8.linear1.weight", "transformer_encoder.layers.8.linear1.bias", "transformer_encoder.layers.8.linear2.weight", "transformer_encoder.layers.8.linear2.bias", "transformer_encoder.layers.8.norm1.weight", "transformer_encoder.layers.8.norm1.bias", "transformer_encoder.layers.8.norm2.weight", "transformer_encoder.layers.8.norm2.bias", "transformer_encoder.layers.9.self_attn.in_proj_weight", "transformer_encoder.layers.9.self_attn.in_proj_bias", "transformer_encoder.layers.9.self_attn.out_proj.weight", "transformer_encoder.layers.9.self_attn.out_proj.bias", "transformer_encoder.layers.9.linear1.weight", "transformer_encoder.layers.9.linear1.bias", "transformer_encoder.layers.9.linear2.weight", "transformer_encoder.layers.9.linear2.bias", "transformer_encoder.layers.9.norm1.weight", "transformer_encoder.layers.9.norm1.bias", "transformer_encoder.layers.9.norm2.weight", "transformer_encoder.layers.9.norm2.bias", "transformer_encoder.layers.10.self_attn.in_proj_weight", "transformer_encoder.layers.10.self_attn.in_proj_bias", "transformer_encoder.layers.10.self_attn.out_proj.weight", "transformer_encoder.layers.10.self_attn.out_proj.bias", "transformer_encoder.layers.10.linear1.weight", "transformer_encoder.layers.10.linear1.bias", "transformer_encoder.layers.10.linear2.weight", "transformer_encoder.layers.10.linear2.bias", "transformer_encoder.layers.10.norm1.weight", "transformer_encoder.layers.10.norm1.bias", "transformer_encoder.layers.10.norm2.weight", "transformer_encoder.layers.10.norm2.bias", "transformer_encoder.layers.11.self_attn.in_proj_weight", "transformer_encoder.layers.11.self_attn.in_proj_bias", "transformer_encoder.layers.11.self_attn.out_proj.weight", "transformer_encoder.layers.11.self_attn.out_proj.bias", "transformer_encoder.layers.11.linear1.weight", "transformer_encoder.layers.11.linear1.bias", "transformer_encoder.layers.11.linear2.weight", "transformer_encoder.layers.11.linear2.bias", "transformer_encoder.layers.11.norm1.weight", "transformer_encoder.layers.11.norm1.bias", "transformer_encoder.layers.11.norm2.weight", "transformer_encoder.layers.11.norm2.bias", "transformer_encoder.layers.12.self_attn.in_proj_weight", "transformer_encoder.layers.12.self_attn.in_proj_bias", "transformer_encoder.layers.12.self_attn.out_proj.weight", "transformer_encoder.layers.12.self_attn.out_proj.bias", "transformer_encoder.layers.12.linear1.weight", "transformer_encoder.layers.12.linear1.bias", "transformer_encoder.layers.12.linear2.weight", "transformer_encoder.layers.12.linear2.bias", "transformer_encoder.layers.12.norm1.weight", "transformer_encoder.layers.12.norm1.bias", "transformer_encoder.layers.12.norm2.weight", "transformer_encoder.layers.12.norm2.bias", "transformer_encoder.layers.13.self_attn.in_proj_weight", "transformer_encoder.layers.13.self_attn.in_proj_bias", "transformer_encoder.layers.13.self_attn.out_proj.weight", "transformer_encoder.layers.13.self_attn.out_proj.bias", "transformer_encoder.layers.13.linear1.weight", "transformer_encoder.layers.13.linear1.bias", "transformer_encoder.layers.13.linear2.weight", "transformer_encoder.layers.13.linear2.bias", "transformer_encoder.layers.13.norm1.weight", "transformer_encoder.layers.13.norm1.bias", "transformer_encoder.layers.13.norm2.weight", "transformer_encoder.layers.13.norm2.bias", "transformer_encoder.layers.14.self_attn.in_proj_weight", "transformer_encoder.layers.14.self_attn.in_proj_bias", "transformer_encoder.layers.14.self_attn.out_proj.weight", "transformer_encoder.layers.14.self_attn.out_proj.bias", "transformer_encoder.layers.14.linear1.weight", "transformer_encoder.layers.14.linear1.bias", "transformer_encoder.layers.14.linear2.weight", "transformer_encoder.layers.14.linear2.bias", "transformer_encoder.layers.14.norm1.weight", "transformer_encoder.layers.14.norm1.bias", "transformer_encoder.layers.14.norm2.weight", "transformer_encoder.layers.14.norm2.bias", "transformer_encoder.layers.15.self_attn.in_proj_weight", "transformer_encoder.layers.15.self_attn.in_proj_bias", "transformer_encoder.layers.15.self_attn.out_proj.weight", "transformer_encoder.layers.15.self_attn.out_proj.bias", "transformer_encoder.layers.15.linear1.weight", "transformer_encoder.layers.15.linear1.bias", "transformer_encoder.layers.15.linear2.weight", "transformer_encoder.layers.15.linear2.bias", "transformer_encoder.layers.15.norm1.weight", "transformer_encoder.layers.15.norm1.bias", "transformer_encoder.layers.15.norm2.weight", "transformer_encoder.layers.15.norm2.bias", "transformer_encoder.layers.16.self_attn.in_proj_weight", "transformer_encoder.layers.16.self_attn.in_proj_bias", "transformer_encoder.layers.16.self_attn.out_proj.weight", "transformer_encoder.layers.16.self_attn.out_proj.bias", "transformer_encoder.layers.16.linear1.weight", "transformer_encoder.layers.16.linear1.bias", "transformer_encoder.layers.16.linear2.weight", "transformer_encoder.layers.16.linear2.bias", "transformer_encoder.layers.16.norm1.weight", "transformer_encoder.layers.16.norm1.bias", "transformer_encoder.layers.16.norm2.weight", "transformer_encoder.layers.16.norm2.bias", "transformer_encoder.layers.17.self_attn.in_proj_weight", "transformer_encoder.layers.17.self_attn.in_proj_bias", "transformer_encoder.layers.17.self_attn.out_proj.weight", "transformer_encoder.layers.17.self_attn.out_proj.bias", "transformer_encoder.layers.17.linear1.weight", "transformer_encoder.layers.17.linear1.bias", "transformer_encoder.layers.17.linear2.weight", "transformer_encoder.layers.17.linear2.bias", "transformer_encoder.layers.17.norm1.weight", "transformer_encoder.layers.17.norm1.bias", "transformer_encoder.layers.17.norm2.weight", "transformer_encoder.layers.17.norm2.bias", "transformer_encoder.layers.18.self_attn.in_proj_weight", "transformer_encoder.layers.18.self_attn.in_proj_bias", "transformer_encoder.layers.18.self_attn.out_proj.weight", "transformer_encoder.layers.18.self_attn.out_proj.bias", "transformer_encoder.layers.18.linear1.weight", "transformer_encoder.layers.18.linear1.bias", "transformer_encoder.layers.18.linear2.weight", "transformer_encoder.layers.18.linear2.bias", "transformer_encoder.layers.18.norm1.weight", "transformer_encoder.layers.18.norm1.bias", "transformer_encoder.layers.18.norm2.weight", "transformer_encoder.layers.18.norm2.bias", "transformer_encoder.layers.19.self_attn.in_proj_weight", "transformer_encoder.layers.19.self_attn.in_proj_bias", "transformer_encoder.layers.19.self_attn.out_proj.weight", "transformer_encoder.layers.19.self_attn.out_proj.bias", "transformer_encoder.layers.19.linear1.weight", "transformer_encoder.layers.19.linear1.bias", "transformer_encoder.layers.19.linear2.weight", "transformer_encoder.layers.19.linear2.bias", "transformer_encoder.layers.19.norm1.weight", "transformer_encoder.layers.19.norm1.bias", "transformer_encoder.layers.19.norm2.weight", "transformer_encoder.layers.19.norm2.bias", "transformer_encoder.layers.20.self_attn.in_proj_weight", "transformer_encoder.layers.20.self_attn.in_proj_bias", "transformer_encoder.layers.20.self_attn.out_proj.weight", "transformer_encoder.layers.20.self_attn.out_proj.bias", "transformer_encoder.layers.20.linear1.weight", "transformer_encoder.layers.20.linear1.bias", "transformer_encoder.layers.20.linear2.weight", "transformer_encoder.layers.20.linear2.bias", "transformer_encoder.layers.20.norm1.weight", "transformer_encoder.layers.20.norm1.bias", "transformer_encoder.layers.20.norm2.weight", "transformer_encoder.layers.20.norm2.bias", "transformer_encoder.layers.21.self_attn.in_proj_weight", "transformer_encoder.layers.21.self_attn.in_proj_bias", "transformer_encoder.layers.21.self_attn.out_proj.weight", "transformer_encoder.layers.21.self_attn.out_proj.bias", "transformer_encoder.layers.21.linear1.weight", "transformer_encoder.layers.21.linear1.bias", "transformer_encoder.layers.21.linear2.weight", "transformer_encoder.layers.21.linear2.bias", "transformer_encoder.layers.21.norm1.weight", "transformer_encoder.layers.21.norm1.bias", "transformer_encoder.layers.21.norm2.weight", "transformer_encoder.layers.21.norm2.bias", "transformer_encoder.layers.22.self_attn.in_proj_weight", "transformer_encoder.layers.22.self_attn.in_proj_bias", "transformer_encoder.layers.22.self_attn.out_proj.weight", "transformer_encoder.layers.22.self_attn.out_proj.bias", "transformer_encoder.layers.22.linear1.weight", "transformer_encoder.layers.22.linear1.bias", "transformer_encoder.layers.22.linear2.weight", "transformer_encoder.layers.22.linear2.bias", "transformer_encoder.layers.22.norm1.weight", "transformer_encoder.layers.22.norm1.bias", "transformer_encoder.layers.22.norm2.weight", "transformer_encoder.layers.22.norm2.bias", "transformer_encoder.layers.23.self_attn.in_proj_weight", "transformer_encoder.layers.23.self_attn.in_proj_bias", "transformer_encoder.layers.23.self_attn.out_proj.weight", "transformer_encoder.layers.23.self_attn.out_proj.bias", "transformer_encoder.layers.23.linear1.weight", "transformer_encoder.layers.23.linear1.bias", "transformer_encoder.layers.23.linear2.weight", "transformer_encoder.layers.23.linear2.bias", "transformer_encoder.layers.23.norm1.weight", "transformer_encoder.layers.23.norm1.bias", "transformer_encoder.layers.23.norm2.weight", "transformer_encoder.layers.23.norm2.bias", "transformer_encoder.layers.24.self_attn.in_proj_weight", "transformer_encoder.layers.24.self_attn.in_proj_bias", "transformer_encoder.layers.24.self_attn.out_proj.weight", "transformer_encoder.layers.24.self_attn.out_proj.bias", "transformer_encoder.layers.24.linear1.weight", "transformer_encoder.layers.24.linear1.bias", "transformer_encoder.layers.24.linear2.weight", "transformer_encoder.layers.24.linear2.bias", "transformer_encoder.layers.24.norm1.weight", "transformer_encoder.layers.24.norm1.bias", "transformer_encoder.layers.24.norm2.weight", "transformer_encoder.layers.24.norm2.bias", "transformer_encoder.layers.25.self_attn.in_proj_weight", "transformer_encoder.layers.25.self_attn.in_proj_bias", "transformer_encoder.layers.25.self_attn.out_proj.weight", "transformer_encoder.layers.25.self_attn.out_proj.bias", "transformer_encoder.layers.25.linear1.weight", "transformer_encoder.layers.25.linear1.bias", "transformer_encoder.layers.25.linear2.weight", "transformer_encoder.layers.25.linear2.bias", "transformer_encoder.layers.25.norm1.weight", "transformer_encoder.layers.25.norm1.bias", "transformer_encoder.layers.25.norm2.weight", "transformer_encoder.layers.25.norm2.bias", "transformer_encoder.layers.26.self_attn.in_proj_weight", "transformer_encoder.layers.26.self_attn.in_proj_bias", "transformer_encoder.layers.26.self_attn.out_proj.weight", "transformer_encoder.layers.26.self_attn.out_proj.bias", "transformer_encoder.layers.26.linear1.weight", "transformer_encoder.layers.26.linear1.bias", "transformer_encoder.layers.26.linear2.weight", "transformer_encoder.layers.26.linear2.bias", "transformer_encoder.layers.26.norm1.weight", "transformer_encoder.layers.26.norm1.bias", "transformer_encoder.layers.26.norm2.weight", "transformer_encoder.layers.26.norm2.bias", "transformer_encoder.layers.27.self_attn.in_proj_weight", "transformer_encoder.layers.27.self_attn.in_proj_bias", "transformer_encoder.layers.27.self_attn.out_proj.weight", "transformer_encoder.layers.27.self_attn.out_proj.bias", "transformer_encoder.layers.27.linear1.weight", "transformer_encoder.layers.27.linear1.bias", "transformer_encoder.layers.27.linear2.weight", "transformer_encoder.layers.27.linear2.bias", "transformer_encoder.layers.27.norm1.weight", "transformer_encoder.layers.27.norm1.bias", "transformer_encoder.layers.27.norm2.weight", "transformer_encoder.layers.27.norm2.bias", "transformer_encoder.layers.28.self_attn.in_proj_weight", "transformer_encoder.layers.28.self_attn.in_proj_bias", "transformer_encoder.layers.28.self_attn.out_proj.weight", "transformer_encoder.layers.28.self_attn.out_proj.bias", "transformer_encoder.layers.28.linear1.weight", "transformer_encoder.layers.28.linear1.bias", "transformer_encoder.layers.28.linear2.weight", "transformer_encoder.layers.28.linear2.bias", "transformer_encoder.layers.28.norm1.weight", "transformer_encoder.layers.28.norm1.bias", "transformer_encoder.layers.28.norm2.weight", "transformer_encoder.layers.28.norm2.bias", "transformer_encoder.layers.29.self_attn.in_proj_weight", "transformer_encoder.layers.29.self_attn.in_proj_bias", "transformer_encoder.layers.29.self_attn.out_proj.weight", "transformer_encoder.layers.29.self_attn.out_proj.bias", "transformer_encoder.layers.29.linear1.weight", "transformer_encoder.layers.29.linear1.bias", "transformer_encoder.layers.29.linear2.weight", "transformer_encoder.layers.29.linear2.bias", "transformer_encoder.layers.29.norm1.weight", "transformer_encoder.layers.29.norm1.bias", "transformer_encoder.layers.29.norm2.weight", "transformer_encoder.layers.29.norm2.bias", "transformer_encoder.layers.30.self_attn.in_proj_weight", "transformer_encoder.layers.30.self_attn.in_proj_bias", "transformer_encoder.layers.30.self_attn.out_proj.weight", "transformer_encoder.layers.30.self_attn.out_proj.bias", "transformer_encoder.layers.30.linear1.weight", "transformer_encoder.layers.30.linear1.bias", "transformer_encoder.layers.30.linear2.weight", "transformer_encoder.layers.30.linear2.bias", "transformer_encoder.layers.30.norm1.weight", "transformer_encoder.layers.30.norm1.bias", "transformer_encoder.layers.30.norm2.weight", "transformer_encoder.layers.30.norm2.bias", "transformer_encoder.layers.31.self_attn.in_proj_weight", "transformer_encoder.layers.31.self_attn.in_proj_bias", "transformer_encoder.layers.31.self_attn.out_proj.weight", "transformer_encoder.layers.31.self_attn.out_proj.bias", "transformer_encoder.layers.31.linear1.weight", "transformer_encoder.layers.31.linear1.bias", "transformer_encoder.layers.31.linear2.weight", "transformer_encoder.layers.31.linear2.bias", "transformer_encoder.layers.31.norm1.weight", "transformer_encoder.layers.31.norm1.bias", "transformer_encoder.layers.31.norm2.weight", "transformer_encoder.layers.31.norm2.bias", "transformer_encoder.layers.32.self_attn.in_proj_weight", "transformer_encoder.layers.32.self_attn.in_proj_bias", "transformer_encoder.layers.32.self_attn.out_proj.weight", "transformer_encoder.layers.32.self_attn.out_proj.bias", "transformer_encoder.layers.32.linear1.weight", "transformer_encoder.layers.32.linear1.bias", "transformer_encoder.layers.32.linear2.weight", "transformer_encoder.layers.32.linear2.bias", "transformer_encoder.layers.32.norm1.weight", "transformer_encoder.layers.32.norm1.bias", "transformer_encoder.layers.32.norm2.weight", "transformer_encoder.layers.32.norm2.bias".

Conda

yaml

I'm using a conda env I constructed on a Linux machine with an NVIDIA GPU as follows:

name: uce
channels:
  - conda-forge
  - pytorch
  - nvidia
  - defaults
dependencies:
  - python
  - pandas
  - scikit-learn
  - numpy
  - tqdm
  - pytorch>=2.0.0
  - pytorch-gpu
  - torchvision
  - torchaudio
  - cuda-toolkit>=12.0.0
  - accelerate
  - requests
  - urllib3==1.26.6  
  - scanpy
  # - igraph-python
  # - leidenalg
  - ipykernel 
  - pip 
  - pip:
    - gdown

versions

``` (uce) bms20@a30:~$ conda list # packages in environment at /home/bms20/anaconda3/envs/uce: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_kmp_llvm conda-forge accelerate 0.30.1 pyhd8ed1ab_0 conda-forge alsa-lib 1.2.11 hd590300_1 conda-forge anndata 0.10.7 pyhd8ed1ab_0 conda-forge array-api-compat 1.7.1 pyhd8ed1ab_0 conda-forge asttokens 2.4.1 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.12.3 pypi_0 pypi brotli 1.1.0 hd590300_1 conda-forge brotli-bin 1.1.0 hd590300_1 conda-forge brotlipy 0.7.0 py312h98912ed_1006 conda-forge bzip2 1.0.8 hd590300_5 conda-forge c-ares 1.28.1 hd590300_0 conda-forge ca-certificates 2024.3.11 h06a4308_0 cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cccl 2.4.0 h7ab4013_0 conda-forge certifi 2024.2.2 py312h06a4308_0 cffi 1.16.0 py312hf06ca03_0 conda-forge charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.2.2 pyhd8ed1ab_0 conda-forge contourpy 1.2.1 py312h8572e83_0 conda-forge cryptography 42.0.7 py312hbcc2302_0 conda-forge cuda-cccl 12.5.39 ha770c72_0 conda-forge cuda-cccl_linux-64 12.5.39 ha770c72_0 conda-forge cuda-command-line-tools 12.5.0 ha770c72_0 conda-forge cuda-compiler 12.5.0 0 nvidia cuda-cudart 12.5.39 he02047a_0 conda-forge cuda-cudart-dev 12.5.39 he02047a_0 conda-forge cuda-cudart-dev_linux-64 12.5.39 h85509e4_0 conda-forge cuda-cudart-static 12.5.39 he02047a_0 conda-forge cuda-cudart-static_linux-64 12.5.39 h85509e4_0 conda-forge cuda-cudart_linux-64 12.5.39 h85509e4_0 conda-forge cuda-cuobjdump 12.5.39 he02047a_0 conda-forge cuda-cupti 12.5.39 he02047a_0 conda-forge cuda-cupti-dev 12.5.39 he02047a_0 conda-forge cuda-cuxxfilt 12.5.39 he02047a_0 conda-forge cuda-driver-dev 12.5.39 he02047a_0 conda-forge cuda-driver-dev_linux-64 12.5.39 h85509e4_0 conda-forge cuda-gdb 12.5.39 hda18ab6_0 conda-forge cuda-libraries 12.5.0 ha770c72_0 conda-forge cuda-libraries-dev 12.5.0 ha770c72_0 conda-forge cuda-nsight 12.5.39 ha770c72_0 conda-forge cuda-nvcc 12.4.131 0 nvidia cuda-nvdisasm 12.5.39 he02047a_0 conda-forge cuda-nvml-dev 12.5.39 he02047a_0 conda-forge cuda-nvprof 12.5.39 he02047a_0 conda-forge cuda-nvprune 12.5.39 he02047a_0 conda-forge cuda-nvrtc 12.5.40 he02047a_0 conda-forge cuda-nvrtc-dev 12.5.40 he02047a_0 conda-forge cuda-nvtx 12.5.39 he02047a_0 conda-forge cuda-nvvp 12.5.39 he02047a_0 conda-forge cuda-opencl 12.5.39 he02047a_0 conda-forge cuda-opencl-dev 12.5.39 he02047a_0 conda-forge cuda-profiler-api 12.5.39 ha770c72_0 conda-forge cuda-sanitizer-api 12.5.39 he02047a_0 conda-forge cuda-toolkit 12.5.0 ha804496_0 conda-forge cuda-tools 12.5.0 ha770c72_0 conda-forge cuda-version 12.5 hd4f0392_3 conda-forge cuda-visual-tools 12.5.0 ha770c72_0 conda-forge cudnn 8.9.7.29 h092f7fd_3 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dbus 1.13.6 h5008d03_3 conda-forge debugpy 1.8.1 py312h30efb56_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge expat 2.6.2 h59595ed_0 conda-forge filelock 3.14.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_2 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.53.0 py312h9a8786e_0 conda-forge freetype 2.12.1 h267a509_2 conda-forge fsspec 2024.5.0 pyhff2d567_0 conda-forge gdown 5.2.0 pypi_0 pypi gds-tools 1.10.0.4 he02047a_0 conda-forge get-annotations 0.1.2 pyhd8ed1ab_0 conda-forge gmp 6.3.0 h59595ed_1 conda-forge gmpy2 2.1.5 py312h1d5cde6_1 conda-forge h5py 3.11.0 nompi_py312hb7ab980_101 conda-forge hdf5 1.14.3 nompi_hdf9ad27_102 conda-forge huggingface_hub 0.23.2 pyhd8ed1ab_0 conda-forge icu 73.2 h59595ed_0 conda-forge idna 3.7 pyhd8ed1ab_0 conda-forge importlib-metadata 7.1.0 pyha770c72_0 conda-forge importlib_metadata 7.1.0 hd8ed1ab_0 conda-forge ipykernel 6.29.3 pyhd33586a_0 conda-forge ipython 8.25.0 pyh707e725_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jinja2 3.1.4 pyhd8ed1ab_0 conda-forge joblib 1.4.2 pyhd8ed1ab_0 conda-forge jupyter_client 8.6.2 pyhd8ed1ab_0 conda-forge jupyter_core 5.7.2 py312h7900ff3_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.5 py312h8572e83_1 conda-forge krb5 1.21.2 h659d440_0 conda-forge lcms2 2.16 hb7c19ff_0 conda-forge ld_impl_linux-64 2.40 hf3520f5_1 conda-forge legacy-api-wrap 1.4 pyhd8ed1ab_1 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libabseil 20240116.2 cxx17_h59595ed_0 conda-forge libaec 1.1.3 h59595ed_0 conda-forge libblas 3.9.0 22_linux64_openblas conda-forge libbrotlicommon 1.1.0 hd590300_1 conda-forge libbrotlidec 1.1.0 hd590300_1 conda-forge libbrotlienc 1.1.0 hd590300_1 conda-forge libcblas 3.9.0 22_linux64_openblas conda-forge libcublas 12.5.2.13 he02047a_0 conda-forge libcublas-dev 12.5.2.13 he02047a_0 conda-forge libcufft 11.2.3.18 he02047a_0 conda-forge libcufft-dev 11.2.3.18 he02047a_0 conda-forge libcufile 1.10.0.4 he02047a_0 conda-forge libcufile-dev 1.10.0.4 he02047a_0 conda-forge libcurand 10.3.6.39 he02047a_0 conda-forge libcurand-dev 10.3.6.39 he02047a_0 conda-forge libcurl 8.8.0 hca28451_0 conda-forge libcusolver 11.6.2.40 he02047a_0 conda-forge libcusolver-dev 11.6.2.40 he02047a_0 conda-forge libcusparse 12.4.1.24 he02047a_0 conda-forge libcusparse-dev 12.4.1.24 he02047a_0 conda-forge libdeflate 1.20 hd590300_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 hd590300_2 conda-forge libexpat 2.6.2 h59595ed_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 13.2.0 h77fa898_7 conda-forge libgfortran-ng 13.2.0 h69a702a_7 conda-forge libgfortran5 13.2.0 hca663fb_7 conda-forge libglib 2.80.2 hf974151_0 conda-forge libhwloc 2.10.0 default_h5622ce7_1001 conda-forge libiconv 1.17 hd590300_2 conda-forge libjpeg-turbo 3.0.0 hd590300_1 conda-forge liblapack 3.9.0 22_linux64_openblas conda-forge libllvm14 14.0.6 hcd5def8_4 conda-forge libmagma 2.7.2 h173bb3b_2 conda-forge libmagma_sparse 2.7.2 h173bb3b_3 conda-forge libnghttp2 1.58.0 h47da74e_1 conda-forge libnpp 12.3.0.116 he02047a_0 conda-forge libnpp-dev 12.3.0.116 he02047a_0 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libnvfatbin 12.5.39 he02047a_0 conda-forge libnvfatbin-dev 12.5.39 he02047a_0 conda-forge libnvjitlink 12.5.40 he02047a_0 conda-forge libnvjitlink-dev 12.5.40 he02047a_0 conda-forge libnvjpeg 12.3.2.38 he02047a_0 conda-forge libnvjpeg-dev 12.3.2.38 ha770c72_0 conda-forge libopenblas 0.3.27 pthreads_h413a1c8_0 conda-forge libpng 1.6.43 h2797004_0 conda-forge libprotobuf 4.25.3 h08a7969_0 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsqlite 3.45.3 h2797004_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.2.0 hc0a3c3a_7 conda-forge libtiff 4.6.0 h1dd3fc0_3 conda-forge libtorch 2.3.0 cuda120_h2b0da52_301 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libuv 1.48.0 hd590300_0 conda-forge libwebp-base 1.4.0 hd590300_0 conda-forge libxcb 1.15 h0b41bf4_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libxkbcommon 1.7.0 h662e7e4_0 conda-forge libxkbfile 1.1.0 h166bdaf_1 conda-forge libxml2 2.12.7 hc051c1a_0 conda-forge libzlib 1.2.13 h4ab18f5_6 conda-forge llvm-openmp 18.1.6 ha31de31_0 conda-forge llvmlite 0.42.0 py312hb06c811_1 conda-forge markupsafe 2.1.5 py312h98912ed_0 conda-forge matplotlib-base 3.8.4 py312h20ab3a6_2 conda-forge matplotlib-inline 0.1.7 pyhd8ed1ab_0 conda-forge mkl 2023.2.0 h84fe81f_50496 conda-forge mpc 1.3.1 hfe3b2da_0 conda-forge mpfr 4.2.1 h9458935_1 conda-forge mpmath 1.3.0 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge natsort 8.4.0 pyhd8ed1ab_0 conda-forge nccl 2.21.5.1 h3a97aeb_0 conda-forge ncurses 6.5 h59595ed_0 conda-forge nest-asyncio 1.6.0 pyhd8ed1ab_0 conda-forge networkx 3.3 pyhd8ed1ab_1 conda-forge nsight-compute 2024.2.0.16 h54680a5_0 conda-forge nspr 4.35 h27087fc_0 conda-forge nss 3.100 hca3bf56_0 conda-forge numba 0.59.1 py312hacefee8_0 conda-forge numpy 1.26.4 py312heda63a1_0 conda-forge ocl-icd 2.3.2 hd590300_1 conda-forge openjpeg 2.5.2 h488ebb8_0 conda-forge openssl 3.3.0 h4ab18f5_3 conda-forge owlready2 0.46 py312h98912ed_0 conda-forge packaging 24.0 pyhd8ed1ab_0 conda-forge pandas 2.2.2 py312h1d6d2e6_1 conda-forge parso 0.8.4 pyhd8ed1ab_0 conda-forge patsy 0.5.6 pyhd8ed1ab_0 conda-forge pcre2 10.43 hcad00b1_0 conda-forge pexpect 4.9.0 pyhd8ed1ab_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.3.0 py312hdcec9eb_0 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge platformdirs 4.2.2 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.42 pyha770c72_0 conda-forge psutil 5.9.8 py312h98912ed_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pycparser 2.22 pyhd8ed1ab_0 conda-forge pygments 2.18.0 pyhd8ed1ab_0 conda-forge pynndescent 0.5.12 pyhca7485f_0 conda-forge pyopenssl 24.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.12.3 hab00c5b_0_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge python_abi 3.12 4_cp312 conda-forge pytorch 2.3.0 cuda120_py312h26b3cf7_301 conda-forge pytorch-gpu 2.3.0 cuda120py312hf72234c_301 conda-forge pytorch-mutex 1.0 cpu pytorch pytz 2024.1 pyhd8ed1ab_0 conda-forge pyyaml 6.0.1 py312h98912ed_1 conda-forge pyzmq 26.0.3 py312h8fd38d8_0 conda-forge readline 8.2 h8228510_1 conda-forge requests 2.32.3 pyhd8ed1ab_0 conda-forge safetensors 0.4.3 py312h4b3b743_0 conda-forge scanpy 1.10.1 pyhd8ed1ab_0 conda-forge scikit-learn 1.5.0 py312h1fcc3ea_1 conda-forge scipy 1.13.1 py312hc2bc53b_0 conda-forge seaborn 0.13.2 hd8ed1ab_2 conda-forge seaborn-base 0.13.2 pyhd8ed1ab_2 conda-forge session-info 1.0.0 pyhd8ed1ab_0 conda-forge setuptools 70.0.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sleef 3.5.1 h9b69904_2 conda-forge soupsieve 2.5 pypi_0 pypi stack_data 0.6.2 pyhd8ed1ab_0 conda-forge statsmodels 0.14.2 py312h085067d_0 conda-forge stdlib-list 0.10.0 pyhd8ed1ab_0 conda-forge sympy 1.12 pypyh9d50eac_103 conda-forge tbb 2021.12.0 h297d8ca_1 conda-forge threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge tk 8.6.13 noxft_h4845f30_101 conda-forge torchaudio 2.3.0 py312_cpu pytorch torchvision 0.18.0 cuda120py312h85ee0b7_0 conda-forge tornado 6.4 py312h98912ed_0 conda-forge tqdm 4.66.4 pyhd8ed1ab_0 conda-forge traitlets 5.14.3 pyhd8ed1ab_0 conda-forge typing-extensions 4.11.0 hd8ed1ab_0 conda-forge typing_extensions 4.11.0 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge umap-learn 0.5.5 py312h7900ff3_1 conda-forge urllib3 1.26.6 pyhd8ed1ab_0 conda-forge wayland 1.23.0 h5291e77_0 conda-forge wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge xcb-util 0.4.0 hd590300_1 conda-forge xcb-util-cursor 0.1.4 hd590300_1 conda-forge xcb-util-image 0.4.0 h8ee46fc_1 conda-forge xcb-util-keysyms 0.4.0 h8ee46fc_1 conda-forge xcb-util-renderutil 0.3.9 hd590300_1 conda-forge xcb-util-wm 0.4.1 h8ee46fc_1 conda-forge xkeyboard-config 2.41 hd590300_0 conda-forge xorg-compositeproto 0.4.2 h7f98852_1001 conda-forge xorg-damageproto 1.2.1 h7f98852_1002 conda-forge xorg-fixesproto 5.0 h7f98852_1002 conda-forge xorg-inputproto 2.3.2 h7f98852_1002 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.1.1 hd590300_0 conda-forge xorg-libsm 1.2.4 h7391055_0 conda-forge xorg-libx11 1.8.9 h8ee46fc_0 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxcomposite 0.4.6 h0b41bf4_1 conda-forge xorg-libxdamage 1.1.5 h7f98852_1 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge xorg-libxi 1.7.10 h7f98852_0 conda-forge xorg-libxrandr 1.5.2 h7f98852_1 conda-forge xorg-libxrender 0.9.11 hd590300_0 conda-forge xorg-libxtst 1.2.3 h7f98852_1002 conda-forge xorg-randrproto 1.5.0 h7f98852_1001 conda-forge xorg-recordproto 1.14.2 h7f98852_1002 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-util-macros 1.19.3 h7f98852_0 conda-forge xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge zeromq 4.3.5 h75354e8_4 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zstd 1.5.6 ha6fb4c9_0 conda-forge ```
bschilder commented 3 months ago

GPU specifications

Mon Jun  3 08:41:31 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A30                     Off | 00000000:00:07.0 Off |                    0 |
| N/A   34C    P0              46W / 165W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
bschilder commented 3 months ago

ahhh, actually the issue was I had forgotten to add --nlayers 33 This works now!

!python eval_single_anndata.py --adata_path data/ot_l2g.h5ad --dir output/ot/ --species human --nlayers 33 --model_loc model_files/33l_8ep_1024t_1280.torch --batch_size 25 --filter False --skip False

That said, if you'd find it helpful I'd be happy to make a PR with the conda env. It took me a couple weeks to get a version of it that worked and it could save others quite a bit of time.

Yanay1 commented 3 months ago

Hi Brian which packages had issues? I think the up-to date versions of the needed libraries all work.

bschilder commented 3 months ago

cuda drivers, as always 😆

the trick was using cuda-toolkit (which has versions >12.0), NOT cudatoolkit (which only has versions <12.0). Exactly what is "up-to-date" changes over time, and which versions are the latest will depend on the distributor. For those reasons I always find it helpful to have some minimum requirements set in the yaml, along with the conda channels (or pip) to install them from.

Also, it seems you used pytorch >2.0, inferred from the fact that my other envs with earlier versions of pytorch didn't work.

bschilder commented 3 months ago

Made PR here #42