Cannot replicate the result in the multimodal_tutorials

[dyxmvp]

Hi,

We tried to replicate the results in the multimodal_tutorials, but the spatial clustering result was not as good as the one in the tutorial. Would you provide some suggestions? Please find the model evaluation below. Thanks!

The clustering results from our end,

Result from the tutorial,

Training outputs,

Stopping early: metric has not improved more than 0.0 in the last 8 epochs. If the early stopping criterion is too strong, please instantiate it with different parameters in the train method. Model training finished after 96 min 38 sec. Using best model state, which was in epoch 38.

--- MODEL EVALUATION --- val AUROC score: 0.9870 val AUPRC score: 0.9787 val best accuracy score: 0.9578 val best F1 score: 0.9589 val target rna MSE score: 6.1840 val source rna MSE score: 1.6078 val target atac MSE score: 0.3834 val source atac MSE score: 0.1222

[sebastianbirk]

Hi @dyxmvp,

Did you use the same hyperparameters as in the tutorial? Looking at your evaluation, it looks to me that the model finished quite early (38 epochs) and the MSE scores seem quite high for this dataset. Maybe the model ran into an unlucky optimization.

[dyxmvp]

Yes, I used the notebook provided in the tutorial, and the same hyperparameters as in the tutorial. But I could not get the same results.

[sebastianbirk]

This was related to a bug in the random seed configuration. We fixed it now.