Deal with apparent NaNs during eval

theislab / chemCPA

Code for "Predicting Cellular Responses to Novel Drug Perturbations at a Single-Cell Resolution", NeurIPS 2022.

https://arxiv.org/abs/2204.13545

MIT License

88 stars 23 forks source link

Deal with apparent NaNs during eval #47

Closed siboehm closed 2 years ago

siboehm commented 2 years ago

On LINCS we have some problems with evaluations resulting in strange [0.0, 0.0, 0.0] results. There is this line:

https://github.com/theislab/chemical_CPA/blob/main/compert/train.py#L68

So probably the 0.0 come from some NaNs in the evaluation. We should remove this line. Now that the evaluation is much faster (17min without disentangle on LINCS) we can just run it more often and save intermediate checkpoints.

siboehm commented 2 years ago

Probably we should calculate the test loss (during eval) and if it's NaN, we should stop the training run and retain the last checkpoint,

MxMstrmn commented 2 years ago

Good idea! Is this already somewhere implemented?

siboehm commented 2 years ago