Reduce time spent on evaluation during training

theislab / chemCPA

Code for "Predicting Cellular Responses to Novel Drug Perturbations at a Single-Cell Resolution", NeurIPS 2022.

https://arxiv.org/abs/2204.13545

MIT License

99 stars 24 forks source link

Reduce time spent on evaluation during training #17

Closed siboehm closed 2 years ago

siboehm commented 2 years ago

Measures

[x] Change training such that only one evaluation is computed during run
[x] Rely on early stopping
[x] Only save on checkpoint

Potentially, investigate whether the evaluation is computed on the CPU and move it the GPU @siboehm

MxMstrmn commented 2 years ago

@siboehm Would be great if you can check the commit

siboehm commented 2 years ago

It's easier to review code if it's a pull request.

I don't understand the score == best_score line. What is it trying to do? Why are two floats being compared via == not via something like math.equals?
https://github.com/theislab/chemical_CPA/commit/d2b1896001f6e2591386be0d44ed7e3efc8222a3#diff-3c40d8046d2c391899dab9b72d6c73642e35dcd2d5a3da76ce98f3059811348dR220-R221

Else looks good, this should make things much faster.

MxMstrmn commented 2 years ago

I think this is fine since the early_stopping function sets the best_score attribute based on the provided score, cf. L465 here: https://github.com/theislab/chemical_CPA/blob/ad8b5c2cd6229753dcf7fb499f070cb2ecffd1af/compert/model.py#L455-L470

siboehm commented 2 years ago

Thanks, that clears it up! You could save the result of self.autoencoder.early_stopping(score) in a variable, then it'd be more readable, but it look like it works.