cvlab-epfl / detecting-the-unexpected

Detecting the Unexpected via Image Resynthesis
Other
56 stars 10 forks source link

Saving demo images #6

Closed RamyaRaghuraman closed 4 years ago

RamyaRaghuraman commented 4 years ago

Hi @adynathos,

While saving the demo images from exec_evaluations.ipynb, I face the following error from E1_article_evaluation.py

0%| | 0/60 [00:00<?, ?it/s]Traceback (most recent call last): File "main.py", line 52, in eval_obj.run_demo_imgs(dset, b_show=False) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/a05_differences/E1_article_evaluation.py", line 376, in run_demo_imgs Frame.frame_list_apply(tr_make_demo_imgs, dset, n_threads=6, batch=4) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/frame.py", line 122, in frame_list_apply for result in pool.imap(task, frames, chunksize=batch): File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/pool.py", line 325, in return (item for chunk in result for item in chunk) File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, *kwds)) File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar return list(map(args)) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/frame.py", line 91, in frame_list_apply_worker fr.apply(func) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/frame.py", line 30, in apply result = f(frame=self, self) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/transforms.py", line 52, in call frame.apply(tr) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/frame.py", line 30, in apply result = f(frame=self, self) TypeError: tr_get_anomaly_gt() missing 1 required positional argument: 'labels_source' 0%| | 0/60 [00:01<?, ?it/s]

Any help would be appreciated. Thanks :)

RamyaRaghuraman commented 4 years ago

@adynathos ,

After the current commit aa78c5d, images under gen_image, demo_with_labels and demo_with_baselines folders are not being created. And also before the current commit, only images under sem_labels and gen_image folders were being created.

adynathos commented 4 years ago

There may be an error happening during the image being written. The images are saved in a background thread, so we would not see the exception. Now I put the exception in the log, could you please run it again to see if there is something in the log?

RamyaRaghuraman commented 4 years ago

you are right. It is to do with .webp extension, but I did not face this issue before.

Traceback (most recent call last): File "C:\Users\RAR7ABT\pj-val-ml\pjval_ml\OSR\GAN\src\datasets\dataset.py", line 44, in imwrite **IMWRITE_OPTS.get(path.suffix.lower()[1:], {}), File "C:\Users\RAR7ABT\AppData\Local\conda\conda\envs\pjval\lib\site-packages\PIL\Image.py", line 1951, in save raise ValueError('unknown file extension: {}'.format(ext)) ValueError: unknown file extension: .webp ERROR |dataset.py:47| Error in saving image \abtvdfs2.de.bosch.com\ismdfs\iad\PJ-Val_ML\data\samples_osr_yolo\GAN\exp\0500_Eval\road_anomaly_test\sem_BaySegBdd\gen_image\animals03_Zebras_in_the_road_gen_image.webp Traceback (most recent call last): File "C:\Users\RAR7ABT\AppData\Local\conda\conda\envs\pjval\lib\site-packages\PIL\Image.py", line 1949, in save format = EXTENSION[ext] KeyError: '.webp'

I tried installing libwebp via "conda install -c conda-forge libwebp" in my conda environment, but I does not seem to work. I am trying to find a workaround for it.

adynathos commented 4 years ago

That is probably an outdated PIL, an update should fix it: pip install --upgrade pillow

RamyaRaghuraman commented 4 years ago

thanks so much! Upgrading pillow worked! Will let you know if I face any other issues.

adynathos commented 4 years ago

I am glad it worked.

RamyaRaghuraman commented 4 years ago

Hi @adynathos

Sorry to reopen this, I think the following error is because of multiprocessing:

0%| | 0/51 [00:00<?, ?it/s]Traceback (most recent call last): File "main.py", line 50, in eval_obj.run_demo_imgs(dset, b_show=False) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/a05_differences/E1_article_evaluation.py", line 419, in run_demo_imgs Frame.frame_list_apply(tr_make_demo_imgs, dset.frames, n_proc=8, batch=4) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/frame.py", line 122, in frame_list_apply for result in pool.imap(task, frames, chunksize=batch): File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/pool.py", line 325, in return (item for chunk in result for item in chunk) File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks put(task) File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/rar7abt/.conda/envs/pjval/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) TypeError: can't pickle dict_items objects 0%| | 0/51 [00:00<?, ?it/s] terminate called without an active exception

I also tried using argument n_proc=1 in frame_list_apply in frame.py, but I got the following:

0%| | 0/51 [00:00<?, ?it/s]Traceback (most recent call last): File "main.py", line 50, in eval_obj.run_demo_imgs(dset, b_show=False) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/a05_differences/E1_article_evaluation.py", line 419, in run_demo_imgs Frame.frame_list_apply(tr_make_demo_imgs, dset.frames, n_proc=1, batch=4) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/frame.py", line 113, in frame_list_apply fr.apply(func) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/frame.py", line 30, in apply result = f(frame=self, self) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/transforms.py", line 52, in call frame.apply(tr) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/frame.py", line 30, in apply result = f(frame=self, self) File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/evaluations.py", line 32, in call self.channel.load(dset, frame, self.field_name) AttributeError: 'str' object has no attribute 'load' 0%| | 0/51 [00:00<?, ?it/s]

adynathos commented 4 years ago

The trick to deal with the multiprocessing problem may be to restart the kernel before running the demo image generation.

If we execute the previous steps, the dataset might be in an altered state, then multiprocessing tries to serialize it and fails. So I restart the kernel, run just the top 3 cells (create dataset, create pipeline object) and then the demo image generation.

File "/home/rar7abt/checkouts/pj-val-ml/pjval_ml/OSR/GAN/src/pipeline/evaluations.py", line 32, in call self.channel.load(dset, frame, self.field_name) AttributeError: 'str' object has no attribute 'load'

https://github.com/cvlab-epfl/detecting-the-unexpected/blob/master/src/pipeline/evaluations.py That is not what should be at that line, please try updating to the newest commit.

RamyaRaghuraman commented 4 years ago

Your suggestion worked. Everything else works fine after the latest commit!