Open hgaiser opened 1 year ago
@hgaiser, thanks for you patience! Looks like this is indeed a problem when there is no validation set. Thanks for reporting this!
The reason for changing the hook to validation_start
was because the hook order has changed on the Lightning side. For more details, you could refer to this:
https://lightning.ai/docs/pytorch/stable/common/lightning_module.html#hooks
When we change the hook to on_train_epoch_end
, it completes the training and the export, but fails to do trainer.test()
. It is probably because ModelCheckpoint does not handle the loading the model properly and memory_bank
is not being loaded properly.
We will investigate this and provided an update here
@samet-akcay is there any update on this issue?
@hgaiser, there has been a bit of change in these files as part of the v1 development. Tbh, I haven't checked this for a while. Let me check..
Describe the bug
Most algorithms seem to assume that there is data available for automatically computing the threshold. In my case I only want to train an anomaly model, without validation or evaluation. This is as far as I can tell theoretically possible, however the way some post-epoch values are computed makes this impossible.
Dataset
Folder
Model
PatchCore
Steps to reproduce the behavior
main
branch (commit ) but I would expect it to be the same on the latest released version.val_split_mode: none
.export_mode: onnx
.This will produce the following error:
Which comes from this line, because
self.memory_back
is an empty tensor at that point. This is because this never gets called, because there's no validation being performed.Changing that function to
def on_train_epoch_end(self) -> None:
resolves the issue, and the process seems to complete successfully.This change was introduced here. Maybe @samet-akcay can shed some light on this?
OS information
OS information:
Expected behavior
I expect training and exporting to complete, regardless of validation being performed or not.
Screenshots
No response
Pip/GitHub
GitHub
What version/branch did you use?
main
Configuration YAML
Logs