search

mist-medical / MIST

MIST: A simple, scalable, and end-to-end framework for 3D medical imaging segmentation.
Apache License 2.0
21 stars 12 forks source link

Added check for overwriting previous runs #23

Closed evanmlim closed 2 months ago

evanmlim commented 2 months ago

Added option to enable validation loss graph. Graph data is also saved to file (format can be changed). Added assert statement at start of main.py to check if the specified results folder contains a previous run to prevent accidentally overwriting it.

aecelaya commented 2 months ago

The training and validation loss curves are already recorded using Tensorboard in the ./results/logs folder. Are you not able to access them?

For the assert statement for the results.csv file, I think that it would make more sense to append the date/time stamp to the results.csv file so that it's something like results_07-16-2024-2328. Otherwise, I think that this assertion might cause more problems than it would solve. For example, what happens if you do a test run and then want to go back and run the same experiment for longer? It seems a little cumbersome to have users go back and delete the results.csv file every time.

evanmlim commented 2 months ago

Oh. I didn't know that the loss curves were stored somewhere. I'll start using that. (I'm going to be honest, I somehow never touched the logs folder)

For the assert statement, I've had many cases where I forget to change the results location and I end up overwriting a previous run which had data I still needed. I don't think just renaming the results csv file will work because there is other important data that is also save-worthy (Like if you want to compare predictions from an old run to a newer run). Maybe instead prompt the user that there is a previous run like: "There is a previous run in the results folder. Overwrite? (y/n)". We can also add an option to disable this prompt.

aecelaya commented 2 months ago

Hmm how about we add a flag like --overwrite and set the default value to False? In that case, it throws the assertion error you implemented. Otherwise it overwrites the results. What do you think?

evanmlim commented 2 months ago

That sounds good! I'll add the flag and revert the validation graph changes.

aecelaya commented 2 months ago

Done! There are a few more changes that I'm working on right now (not related to this PR) that I want to add soon. Once those are committed, I'll release a new version with this change.