Issue with saving the reference signature results

[rue1996]

Hi

Thanks for being patient with me. I am running cell2location with new tutorial as suggested, however the trained model is not being saved. The model gets created and trained until is Epoch 250/250: 100%. Then it gets stuck after reaching 100% sampling global variables, sample, this is the stage where it needs to start saving the model but it doesn't get saved. It just remains on that spot. I also noticed that it does not run the sampling global variables, batch to 100%.

Thank you

[vitkl]

Please post the output of scvi.data.view_anndata_setup(adata_ref). And the screenshot of your notebook cells output when the models is not saved.

Do you have enough RAM and GPU memory?

When I had such issues in the past, the solutions were restarting the notebook, and if that does not help, reinstalling the conda environemt.

[rue1996]

Hi @vitkl

Attached is the outputs.

I am using HPC and i believe i have enough RAM and GPU memory.

This is the screenshot of the output

it just get stuck there and does not run the rest of the code.

[vitkl]

Can you please point me to the data that you are using? I've used Allen Institute Smart Seq dataset and the number of cells in your analysis is exactly the same as that Smart Seq data https://portal.brain-map.org/atlases-and-data/rnaseq/mouse-whole-cortex-and-hippocampus-smart-seq.

[rue1996]

This is the link to the data: https://portal.brain-map.org/atlases-and-data/rnaseq/human-m1-10x

[rue1996]

Hi @vitkl

I have tried the option of installing a new environment and i am still running into the same issue. I realized that l do not have GPU. Do you think that might be the reason why it is getting stuck. Do i need GPU to run this?

[vitkl]

GPU is very useful but it should not be essential.

I think the batch_key looks like LKTX_190130_01_H01 in cells names TTACGTTAGGTTGCCC-LKTX_190130_01_H01. You can extract that removing the first letters from the string:

adata.obs['batch_col'] = [i[0:17] for i in adata.obs_names] 

it just get stuck there and does not run the rest of the code.

Does the notebook crash? This step takes quite a lot of RAM (essentially creates a dense matrix equivalent to 1000*127 cells by 17.8k genes, we are working on a solution) - you can try reducing the number of samples from 1000 to 100-200.

[rue1996]

No the notebook does not crash. I am really not sure what could be stopping it from saving the model. I tried reducing samples to 100 but still it is not saving. When l run the lymph node tutorial, the model saves. However it doesn't save the model from the Allen brains data.

[vitkl]

It doesn't save the model or doesn't finish posterior sampling? You can try exporting the median as shown in the advanced use section - which should use much less RAM.

[rue1996]

It doesn't save the model. It reaches 100% in posterior sampling then it doesn't move to the next step. I have left it run for days hoping it would save but still didn't.

I will try the median as well as use GPU. Then give you feedback.