haimasree
commented
2 years ago Hi @bernardo-de-almeida May I ask where exactly you are getting this error? We have nightly tests in place that creates a model specific conda environment from scratch and runs kipoi test <model-name> --source=kipoi and in DeepSTARR's case also tests the resulting values against the predictions stored in here. An example run would be from yesterday's nightly test. If you dont have a circleci account this is what it says -
4/8 - model: DeepSTARR
--------------------
INFO [kipoi.cli.env] Writing environment file: /tmp/kipoi/envfiles/09950047
INFO [kipoi.cli.env] Loading model: DeepSTARR description
INFO [kipoi.sources] Update /home/circleci/repo/
Already up to date.
INFO [kipoi.cli.env] kipoiseq not installed. Using default kipoiseq dependencies for the dataloader: kipoiseq.dataloaders.SeqIntervalDl
INFO [kipoi.cli.env] Environment name: test-kipoi-DeepSTARR
INFO [kipoi.cli.env] Output env file: /tmp/kipoi/envfiles/09950047/test-kipoi-DeepSTARR.yaml
INFO [kipoi.cli.env] Done writing the environment file!
INFO [kipoi.cli.env] Creating conda env from file: /tmp/kipoi/envfiles/09950047/test-kipoi-DeepSTARR.yaml
INFO [kipoi.cli.env] Done!
Activate the environment via:
conda activate test-kipoi-DeepSTARR
INFO [kipoi.sources] Update /home/circleci/repo/
Already up to date.
INFO [kipoi.model] Downloading model arguments arch from https://zenodo.org/record/5502060/files/DeepSTARR.model.json?download=1
Downloading https://zenodo.org/record/5502060/files/DeepSTARR.model.json?download=1 to /home/circleci/repo/DeepSTARR/downloaded/model_files/arch/9b796f79441e53dc75dd79b911fff872
16.4kB [00:01, 9.88kB/s]
INFO [kipoi.model] Downloading model arguments weights from https://zenodo.org/record/5502060/files/DeepSTARR.model.h5?download=1
Downloading https://zenodo.org/record/5502060/files/DeepSTARR.model.h5?download=1 to /home/circleci/repo/DeepSTARR/downloaded/model_files/weights/7e53a9351b2520a4713a5ffdb5f1566c
2.56MB [00:02, 1.05MB/s]
2022-05-27 09:43:17.209117: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-05-27 09:43:17.209148: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-05-27 09:43:18.162728: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2022-05-27 09:43:18.162760: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-05-27 09:43:18.162783: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ip-172-28-27-83): /proc/driver/nvidia/version does not exist
2022-05-27 09:43:18.163021: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO [kipoi.model] successfully loaded model architecture from <_io.TextIOWrapper name='/home/circleci/repo/DeepSTARR/downloaded/model_files/arch/9b796f79441e53dc75dd79b911fff872' mode='r' encoding='utf-8'>
2022-05-27 09:43:18.261243: W tensorflow/core/util/tensor_slice_reader.cc:96] Could not open /home/circleci/repo/DeepSTARR/downloaded/model_files/weights/7e53a9351b2520a4713a5ffdb5f1566c: DATA_LOSS: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
INFO [kipoi.model] successfully loaded model weights from /home/circleci/repo/DeepSTARR/downloaded/model_files/weights/7e53a9351b2520a4713a5ffdb5f1566c
INFO [kipoi.pipeline] dataloader.output_schema is compatible with model.schema
INFO [kipoi.pipeline] Initialized data generator. Running batches...
Downloading https://raw.githubusercontent.com/kipoi/kipoiseq/master/tests/data/intervals_51bp.tsv to /home/circleci/repo/DeepSTARR/downloaded/example_files/intervals_file
8.19kB [00:00, 519kB/s]
Downloading https://raw.githubusercontent.com/kipoi/kipoiseq/master/tests/data/hg38_chr22_32000000_32300000.fa to /home/circleci/repo/DeepSTARR/downloaded/example_files/fasta_file
303kB [00:00, 20.2MB/s]
INFO [kipoi.pipeline] Returned data schema correct
100% 3/3 [00:00<00:00, 4.77it/s]
INFO [kipoi.pipeline] predict_example done!
Downloading https://zenodo.org/record/6553385/files/DeepSTARR.predictions.h5?download=1 to /home/circleci/repo/DeepSTARR/downloaded/model_files/test.expect.h5
98.3kB [00:01, 62.5kB/s]
INFO [kipoi.cli.main] Testing if the predictions match the expected ones in the file: /home/circleci/repo/DeepSTARR/downloaded/model_files/test.expect.h5
INFO [kipoi.cli.main] Desired precision (number of matching decimal places): 4
3it [00:00, 5.64it/s]
INFO [kipoi.cli.main] All predictions match
INFO [kipoi.cli.main] Successfully ran test_predict
On the other hand, if I change input shape to what you suggested I end up getting this warning although it fnishes succesfully -
ArraySchema mismatch
Array shapes don't match for the fields:
--
Model
--
shape: (249, 4)
doc: DNA sequence
--
Dataloader
--
shape: (249, 4, 1)
doc: One-hot encoded DNA sequence
name: seq
special_type: DNASeq
associated_metadata:
- ranges
--
Provided shape (without the batch axis): (249, 4, 1), expected shape: (249, 4) Please write back with a test case where you are getting this error and I will look into it. I will look into why test_new_models is failing later as well.
Thanks!
bernardo-de-almeida
The input shape is incorrect since the model uses (249, 4) - and it's showing the following error "ValueError: Error when checking input: expected input_12 to have 3 dimensions, but got array with shape (4, 249, 4, 1)"