Softmax instability?

[vals]

Reported by David Cook

[dpcook]

Just to record some context here in writing, this occurred when trying to use AEH to find expression patterns along a 2D diffusion map of a ~10,000 scRNA-seq dataset.

Notebook In the notebook I grab the embedding from an AnnData object. Here, I've just included a csv file with the columns X_diffmap1 and X_diffmap2

Data for reproducing the error

1. Expression matrix (~2Gb; csv format)
2. Cell metadata (including diffusion map coordinates)
3. SpatialDE Output
4. AEH Results (pattern assignment and membership value)
5. AEH Pattern values

Let me know if there are any issues accessing everything!

[vals]

Wow it's pretty slow with that many cells it turns out!

I can reproduce the problem with a smaller subsample of the data though, so I'll get to looking into that.

[vals]

I think I fixed the problem in a stable way. I'm just rerunning your clustering now to make sure the results didn't change too much.

[vals]

Hey @dpcook I reran this over the day, now there's no division by 0 error, and the delta_ELBO is properly decreasing.

You would want to run this until delta_ELBO becomes close to 0 though, which will probably take quite some time with this dataset!

I think the patterns I get after 3 iterations are roughly similar to yours though, so I'll push this change to PyPI.

Thanks for the nice report and data needed to debug this!

/Valentine

[dpcook]

Thanks for this @vals! Sorry for delayed reply--just got back from vacation after SCB18.

I'll start running it today and will let you know if there are any issues!

[dpcook]

Glad to report that after runner for a couple days, it seems to have converged after 99 iterations.

[vals]

Wow! Thanks for checking.