JoannaCCJH
commented
4 months ago Hi! I would like to work on this issue!
Open samet-akcay opened 6 months ago
JoannaCCJH
commented
4 months ago Hi! I would like to work on this issue!
samet-akcay
commented
4 months ago Sure, thanks for your interest!
JoannaCCJH
commented
4 months ago @samet-akcay @blaz-r Hi! I just wanna make sure I get the point right. I need to write code to check whether the validation and tests set contain samples above the following code segment. Here are the steps I need to take:
self.trainer.validate() or self.trainer.test() ? self._setup_transform(model, datamodule=datamodule, ckpt_path=ckpt_path)
if model.learning_type in [LearningType.ZERO_SHOT, LearningType.FEW_SHOT]:
# if the model is zero-shot or few-shot, we only need to run validate for normalization and thresholding
self.trainer.validate(model, val_dataloaders, None, verbose=False, datamodule=datamodule)
else:
self.trainer.fit(model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
self.trainer.test(model, test_dataloaders, ckpt_path=ckpt_path, datamodule=datamodule)@samet-akcay @blaz-r
I added a few lines to check whether val_dataloaders or datamodule contain validation images and test images. Am I going on the right track, please?
has_val_samples = True
if not val_dataloaders and not hasattr(datamodule, "val_data"):
has_val_samples = False
has_test_samples = True
if not test_dataloaders and not hasattr(datamodule, "test_data"):
has_test_samples = False
if model.learning_type in [LearningType.ZERO_SHOT, LearningType.FEW_SHOT]:
# if the model is zero-shot or few-shot, we only need to run validate for normalization and thresholding
self.trainer.validate(model, val_dataloaders, None, verbose=False, datamodule=datamodule)
else:
self.trainer.fit(model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
self.trainer.test(model, test_dataloaders, ckpt_path=ckpt_path, datamodule=datamodule)
blaz-r
commented
3 months ago @JoannaCCJH I think this is going into the right direction.
JoannaCCJH
commented
3 months ago Hi! @samet-akcay @blaz-r I've made changes to the code provided, and the modified code is attached below. While the code works for the given example, I encountered some problems during the process.
When calling the train method in the command line (anomalib train --model Padim --data normal.yaml), the data module (datamodule) wasn't being set up automatically. Adding the following lines resolved the error where 'Folder' object has no attribute 'train_data':
if datamodule is not None:
datamodule.setup()Question: I'm thinking whether setting up the data module at this point is appropriate or if there's a better place for it.
Without adding the image_size: (256,256) argument to normal.yaml, the following error occurred:
RuntimeError: stack expects each tensor to be equal size, but got [3, 512, 512] at entry 0 and [3, 522, 512] at entry
21Question: I also saw people having this problem. Should this be fixed?
I'm unsure if my implementation is sufficiently clear and tidy.
def train():
.....
if datamodule is not None:
datamodule.setup()
has_val_samples = True
if not val_dataloaders and not hasattr(datamodule, "val_data"):
has_val_samples = False
has_test_samples = True
if not test_dataloaders and not hasattr(datamodule, "test_data"):
has_test_samples = False
if model.learning_type in [LearningType.ZERO_SHOT, LearningType.FEW_SHOT] and has_val_samples:
# if the model is zero-shot or few-shot, we only need to run validate for normalization and thresholding
self.trainer.validate(model, val_dataloaders, None, verbose=False, datamodule=datamodule)
else:
if has_val_samples:
self.trainer.fit(model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
else:
train_dataloader = train_dataloaders if train_dataloaders is not None else datamodule.train_dataloader()
self.trainer.fit(model, train_dataloaders=train_dataloader, ckpt_path=ckpt_path)
if has_test_samples:
self.trainer.test(model, test_dataloaders, ckpt_path=ckpt_path, datamodule=datamodule)
@djdameln, can you provide your insight here?
Describe the bug
Engine's training entrypoint runs validation and test without checking whether validation and tests contain samples.
Dataset
Folder
Model
PADiM
Steps to reproduce the behavior
normal.yaml.OS information
OS information:
Expected behavior
Ideally, the engine should check if the validation and tests sets contain any images before running the validation and test entrypoints.
Screenshots
No response
Pip/GitHub
GitHub
What version/branch did you use?
No response
Configuration YAML
Logs
Code of Conduct