Closed KartikeyKansal1 closed 9 months ago
Hi @KartikeyKansal1! Are you able to post the compiled configuration file (found in the .hydra folder as config.yaml
)?
Hi @KartikeyKansal1! Are you able to post the compiled configuration file (found in the .hydra folder as
config.yaml
)?
There's only default.yaml file in the congifs/hydra dir. Are you looking for this?
There should be a full configuration file at logs/eval/runs/experiment_name/run_name/yyy-mm-dd_hh-mm-s/.hydra
/config.yaml, are you seeing anything like that?
No, it just has these 3 files.
Just so I clearly understand: I'm supposed to run eval.py
but all the config changes are to be done in configs/experiment/labelfree.yaml?
Correct, updating ckpt_patch
, model.save_dir
, and data.path
should be sufficient. Can you post config_tree.log
?
This is the current error I'm facing after running eval.py
Here's the config tree log: config_tree.log
This is how my experiment/labelfree.yaml looks like:
Are there any values in the split column besides "train", "training", "valid", "val", "validation", "test", "testing"
?
As a side note, replacing the relative paths in your .csv
with absolute paths might help
All values in the split column say 'test'. I want all of these brightfield images to run prediction on.
Changing .csv with absolute paths is resulting in this error:
[2024-01-18 12:03:35,591][cyto_dl.utils.template_utils][ERROR] -
Traceback (most recent call last):
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/utility/dictionary.py", line 266, in __call__
d[key] = self.adjuster(d[key], meta_dict)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/utility/array.py", line 204, in __call__
img = MetaTensor(img)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 116, in __new__
return torch.as_tensor(x, *args, **_kwargs).as_subclass(cls)
TypeError: new(): invalid data type 'str'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/compose.py", line 335, in __call__
result = execute_compose(
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
data = apply_transform(
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.utility.dictionary.EnsureChannelFirstd object at 0x7fafa6475ea0>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/utils/template_utils.py", line 53, in wrap
out = task_func(cfg=cfg)
File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/eval.py", line 87, in evaluate
output = method(model=model, dataloaders=data, ckpt_path=cfg.ckpt_path)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 852, in predict
return call._call_and_handle_interrupt(
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 894, in _predict_impl
results = self._run(model, ckpt_path=ckpt_path)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 980, in _run
results = self._run_stage()
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1018, in _run_stage
return self.predict_loop.run()
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 181, in _decorator
return loop_run(self, *args, **kwargs)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/prediction_loop.py", line 110, in run
batch, batch_idx, dataloader_idx = next(data_fetcher)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 126, in __next__
batch = super().__next__()
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 58, in __next__
batch = next(self.iterator)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 285, in __next__
out = next(self._iterator)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 123, in __next__
out = next(self.iterators[0])
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
data = self._next_data()
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 677, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 298, in __getitem__
return self.dataset[self.indices[idx]]
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/dataset.py", line 112, in __getitem__
return self._transform(index)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/dataset.py", line 98, in _transform
return apply_transform(self.transform, data_i) if self.transform is not None else data_i
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7fafa44a4d60>
[2024-01-18 12:03:35,615][cyto_dl.utils.template_utils][INFO] - Closing loggers...
Error executing job with overrides: ['experiment=im2im/labelfree.yaml', 'trainer=cpu', 'experiment_name=trial1-240115', 'run_name=trial1-run21']
Traceback (most recent call last):
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/utility/dictionary.py", line 266, in __call__
d[key] = self.adjuster(d[key], meta_dict)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/utility/array.py", line 204, in __call__
img = MetaTensor(img)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 116, in __new__
return torch.as_tensor(x, *args, **_kwargs).as_subclass(cls)
TypeError: new(): invalid data type 'str'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/compose.py", line 335, in __call__
result = execute_compose(
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
data = apply_transform(
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.utility.dictionary.EnsureChannelFirstd object at 0x7fafa6475ea0>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/eval.py", line 99, in main
evaluate(cfg)
File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/utils/template_utils.py", line 56, in wrap
raise ex
File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/utils/template_utils.py", line 53, in wrap
out = task_func(cfg=cfg)
File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/eval.py", line 87, in evaluate
output = method(model=model, dataloaders=data, ckpt_path=cfg.ckpt_path)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 852, in predict
return call._call_and_handle_interrupt(
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 894, in _predict_impl
results = self._run(model, ckpt_path=ckpt_path)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 980, in _run
results = self._run_stage()
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1018, in _run_stage
return self.predict_loop.run()
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 181, in _decorator
return loop_run(self, *args, **kwargs)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/prediction_loop.py", line 110, in run
batch, batch_idx, dataloader_idx = next(data_fetcher)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 126, in __next__
batch = super().__next__()
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 58, in __next__
batch = next(self.iterator)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 285, in __next__
out = next(self._iterator)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 123, in __next__
out = next(self.iterators[0])
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
data = self._next_data()
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 677, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 298, in __getitem__
return self.dataset[self.indices[idx]]
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/dataset.py", line 112, in __getitem__
return self._transform(index)
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/dataset.py", line 98, in _transform
return apply_transform(self.transform, data_i) if self.transform is not None else data_i
File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7fafa44a4d60>
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Alright I think I found the bug here. In all of the predict transforms, replace {data.columns}
with ${source_col}
. I'll fix this on main!
This resolved the issue! Thank you so much for your prompt replies. Just to confirm, this change has to be just in the data config file?
correct! Thanks for finding this bug! I'm going to close this thread, but feel free to start a new one if more issues come up.
Hey, I have been able to train on a bunch of 2D images successfully but the prediction yields no results. The documentation (https://github.com/AllenCellModeling/cyto-dl/blob/main/docs/using_examples.rst) seems to be limited in this regard.
The code breaks while training if there's a false csv value, but it doesn't in the case of prediction; which makes me think if the program is able to read
predict.csv
at all.My csv file looks like this:
The documentation says to run prediction by making changes in the training config. I've been running both eval.py and train.py after making the appropriate configuration changes as suggested in documentation. Neither seem to work. Any suggestions?