search

AllenCellModeling / cyto-dl

https://allencellmodeling.github.io/cyto-dl/
15 stars 5 forks source link

failure in generating predict_images in save_dir after running a prediction on 2D images #317

Closed KartikeyKansal1 closed 9 months ago

KartikeyKansal1 commented 9 months ago

Hey, I have been able to train on a bunch of 2D images successfully but the prediction yields no results. The documentation (https://github.com/AllenCellModeling/cyto-dl/blob/main/docs/using_examples.rst) seems to be limited in this regard.

The code breaks while training if there's a false csv value, but it doesn't in the case of prediction; which makes me think if the program is able to read predict.csv at all.

My csv file looks like this:

Screenshot 2024-01-18 at 10 54 49 AM

The documentation says to run prediction by making changes in the training config. I've been running both eval.py and train.py after making the appropriate configuration changes as suggested in documentation. Neither seem to work. Any suggestions?

benjijamorris commented 9 months ago

Hi @KartikeyKansal1! Are you able to post the compiled configuration file (found in the .hydra folder as config.yaml)?

KartikeyKansal1 commented 9 months ago

Hi @KartikeyKansal1! Are you able to post the compiled configuration file (found in the .hydra folder as config.yaml)?

There's only default.yaml file in the congifs/hydra dir. Are you looking for this?

Screenshot 2024-01-18 at 11 22 32 AM
benjijamorris commented 9 months ago

There should be a full configuration file at logs/eval/runs/experiment_name/run_name/yyy-mm-dd_hh-mm-s/.hydra/config.yaml, are you seeing anything like that?

KartikeyKansal1 commented 9 months ago

No, it just has these 3 files.

Screenshot 2024-01-18 at 11 31 52 AM

Just so I clearly understand: I'm supposed to run eval.py but all the config changes are to be done in configs/experiment/labelfree.yaml?

benjijamorris commented 9 months ago

Correct, updating ckpt_patch, model.save_dir, and data.path should be sufficient. Can you post config_tree.log?

KartikeyKansal1 commented 9 months ago

This is the current error I'm facing after running eval.py

Screenshot 2024-01-18 at 11 37 29 AM

Here's the config tree log: config_tree.log

This is how my experiment/labelfree.yaml looks like:

Screenshot 2024-01-18 at 11 40 45 AM
benjijamorris commented 9 months ago

Are there any values in the split column besides "train", "training", "valid", "val", "validation", "test", "testing"? As a side note, replacing the relative paths in your .csv with absolute paths might help

KartikeyKansal1 commented 9 months ago

All values in the split column say 'test'. I want all of these brightfield images to run prediction on.

Changing .csv with absolute paths is resulting in this error:

[2024-01-18 12:03:35,591][cyto_dl.utils.template_utils][ERROR] - 
Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/utility/dictionary.py", line 266, in __call__
    d[key] = self.adjuster(d[key], meta_dict)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/utility/array.py", line 204, in __call__
    img = MetaTensor(img)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 116, in __new__
    return torch.as_tensor(x, *args, **_kwargs).as_subclass(cls)
TypeError: new(): invalid data type 'str'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/compose.py", line 335, in __call__
    result = execute_compose(
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.utility.dictionary.EnsureChannelFirstd object at 0x7fafa6475ea0>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/utils/template_utils.py", line 53, in wrap
    out = task_func(cfg=cfg)
  File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/eval.py", line 87, in evaluate
    output = method(model=model, dataloaders=data, ckpt_path=cfg.ckpt_path)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 852, in predict
    return call._call_and_handle_interrupt(
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 894, in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 980, in _run
    results = self._run_stage()
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1018, in _run_stage
    return self.predict_loop.run()
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 181, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/prediction_loop.py", line 110, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 126, in __next__
    batch = super().__next__()
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 58, in __next__
    batch = next(self.iterator)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 285, in __next__
    out = next(self._iterator)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 123, in __next__
    out = next(self.iterators[0])
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 677, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 298, in __getitem__
    return self.dataset[self.indices[idx]]
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/dataset.py", line 112, in __getitem__
    return self._transform(index)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/dataset.py", line 98, in _transform
    return apply_transform(self.transform, data_i) if self.transform is not None else data_i
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7fafa44a4d60>
[2024-01-18 12:03:35,615][cyto_dl.utils.template_utils][INFO] - Closing loggers...
Error executing job with overrides: ['experiment=im2im/labelfree.yaml', 'trainer=cpu', 'experiment_name=trial1-240115', 'run_name=trial1-run21']
Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/utility/dictionary.py", line 266, in __call__
    d[key] = self.adjuster(d[key], meta_dict)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/utility/array.py", line 204, in __call__
    img = MetaTensor(img)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 116, in __new__
    return torch.as_tensor(x, *args, **_kwargs).as_subclass(cls)
TypeError: new(): invalid data type 'str'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/compose.py", line 335, in __call__
    result = execute_compose(
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.utility.dictionary.EnsureChannelFirstd object at 0x7fafa6475ea0>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/eval.py", line 99, in main
    evaluate(cfg)
  File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/utils/template_utils.py", line 56, in wrap
    raise ex
  File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/utils/template_utils.py", line 53, in wrap
    out = task_func(cfg=cfg)
  File "/Users/kartikeykansal/Documents/GAN Traning Data Set/CytoDL_Trial_240115/cyto-dl/cyto_dl/eval.py", line 87, in evaluate
    output = method(model=model, dataloaders=data, ckpt_path=cfg.ckpt_path)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 852, in predict
    return call._call_and_handle_interrupt(
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 894, in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 980, in _run
    results = self._run_stage()
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1018, in _run_stage
    return self.predict_loop.run()
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 181, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/prediction_loop.py", line 110, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 126, in __next__
    batch = super().__next__()
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 58, in __next__
    batch = next(self.iterator)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 285, in __next__
    out = next(self._iterator)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 123, in __next__
    out = next(self.iterators[0])
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 677, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 298, in __getitem__
    return self.dataset[self.indices[idx]]
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/dataset.py", line 112, in __getitem__
    return self._transform(index)
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/data/dataset.py", line 98, in _transform
    return apply_transform(self.transform, data_i) if self.transform is not None else data_i
  File "/Users/kartikeykansal/miniconda3/envs/myenv240115/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7fafa44a4d60>

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
benjijamorris commented 9 months ago

Alright I think I found the bug here. In all of the predict transforms, replace {data.columns} with ${source_col}. I'll fix this on main!

KartikeyKansal1 commented 9 months ago

This resolved the issue! Thank you so much for your prompt replies. Just to confirm, this change has to be just in the data config file?

benjijamorris commented 9 months ago

correct! Thanks for finding this bug! I'm going to close this thread, but feel free to start a new one if more issues come up.