ValueError with NaN tensors in gimVI training

[sunericd]

When training the gimVI model, I am running into a ValueError in the first epoch for some datasets and not others. In all cases, the inputs are AnnData objects with raw counts for both the RNAseq and for the spatial data (dtype=float64). I have tried filtering out cells with zero counts and/or normalizing the RNAseq data but am still running into the same error. Strangely, gimVI seems to be able to train successfully on one dataset but when I remove 4 genes from the spatial data (32 -> 28 genes), it fails on that dataset. Happy to share data if that would be helpful and if there are suggestions for doing so (screenshots of basic data info below).

    import scvi
    from scvi.external import GIMVI

    # preprocessing of data
    spatial_adata = spatial_adata[:, spatial_adata.var_names.isin(RNAseq_adata.var_names)]

    # indices for filtering out zero-expression cells
    filtered_cells_spatial = (spatial_adata.X.sum(axis=1) > 1)
    filtered_cells_RNAseq = (RNAseq_adata.X.sum(axis=1) > 1)

    # make copies of subsets
    spatial_adata = spatial_adata[filtered_cells_spatial,:].copy()
    RNAseq_adata = RNAseq_adata[filtered_cells_RNAseq,:].copy()

    # setup anndata for scvi
    GIMVI.setup_anndata(spatial_adata)
    GIMVI.setup_anndata(RNAseq_adata)

    # train gimVI model
    model = GIMVI(RNAseq_adata, spatial_adata, **kwargs)
    model.train(200)

ValueError: Expected parameter loc (Tensor of shape (128, 10)) of distribution Normal(loc: torch.Size([128, 10]), scale: torch.Size([128, 10])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0')

Versions:

0.20.3

[martinkim0]

Hi, thanks for bringing up this up and sorry you're running into this issue.

One possible source for these NaN errors are the exponential activations in the model - we've had similar issues before with other models depending on the dataset used as well as any non-default arguments that are passed during model initialization. Unfortunately, there's no straightforward way to modify these at the moment, and we cannot change them on our end because of reproducibility with the original manuscript.

Would you be able to share the dataset as well as a reproducible notebook?

[sunericd]

Thanks for the fast response. Here is a link to a minimal notebook and data files for reproducing the errors (when I tested it today, gimVI seems to fail on the 32 gene data and succeeds on the 28 gene data, so maybe there are some variable behaviors there):

https://www.dropbox.com/sh/5smbltpudpntmh4/AABrxiGL8jlNxdq3dvHOx3bga?dl=0

[martinkim0]

Thanks! I'll try to get this running and get back to you

[izabellaleahz]

Hello, I am also running into the same issue. Is there a way steps to modifying could be publicly posted?

[james-cranley]

any news on this? I am also experiencing the same error (mine is when running mvi.train())

[james-cranley]

PS I tried with the same code and same object using an old environment (with scvi 0.2.3 installed) and it ran. the env where I was getting this error had 1.0.3 installed

[james-cranley]

with the environment which previously gave this issue activated, I ran:

pip uninstall scvi-tools
pip install scvi-tools==0.20.3

then restarted the kernel and the notebook ran without error.

[lila167]

With scvi-tools==1.0.3 installed, I got a similar error while training MultiVI and could fix the problem by installing scvi-tools==0.20.3.

[Accelerator-thu]

One possible cause of this issue is invalid data input, I guess the input needs to be a non-negative matrix.

BTW, it would be reasonable to add an assert line at the outer modules, avoiding misdirections like this.

[sunericd]

Has there been any updates on this? I have tried running different versions (0.19.0, 0.20.3, and most recently 1.0.4) but none of them consistently work for the test data and notebook. Using scanpy==1.9.6 and scvi-tools==1.0.4, I did notice that if I restart the kernel a few times, sometimes it is able to complete the model.train(200) call successfully so perhaps it is due to some stochastic part of the model?

[Dana162001]

Hi, I would also like to hear if someone solved the problem, in my case model.train() function runs with 2 epochs but everything more than that and I am getting the same value error: ValueError: Expected parameter loc (Tensor of shape (128, 10)) of distribution Normal(loc: torch.Size([128, 10]), scale: torch.Size([128, 10])) to satisfy the constraint Real(), but found invalid values: tensor([[nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], ..., [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan]], device='cuda:0',

[ktpolanski]

Same story here - downgraded from 1.0.4 to 0.20.3 and MultiVI ran fine. Dropping a comment mostly to be notified if something budges on this front.

[adamdingliang]

Running into the same issue, there seems to be some randomness depending on the number of genes or cells, as mentioned above. Tried different version of scvi-tools, is there any workaround?

[canergen]

A lot of those issues should be fixed in the recent release in 1.2.0. Do you use the recent version? Have you removed cells with less than 20 counts (these add some instability). Adding drop_last=True to the datasplitter_kwargs might help additionally.

[adamdingliang]

We are using version 1.1.6.post2, we haven't tried 1.2.0. Yes, for scRNAseq, we removed cells with less than 100 counts; for spatial data, we removed cells with less than 20 counts. Thanks for the suggestions.