Running Inference Error

[feritstar]

Hello,

I'm trying to make inference work on google colab with my google drive account.

At some point, as you can see below in Colab Output part, cityscapes validation part gives an error.

I'm not sure what is wrong.

Here my google drive structure:

at top : /content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL

my directory path in config.py file is:

"""Data Dir and Weight Dir""" C.city_root_path = '/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/cityscapes' C.coco_root_path = '/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/coco' C.fishy_root_path = '/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/fishyscapes' C.segment_me_root_path = '/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/segment_me' C.road_anomaly_root_path = '/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/roadanomaly'

C.rpl_corocl_weight_path = os.path.join(C.root_dir, 'ckpts', 'exp', '/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rev3.pth') C.pretrained_weight_path = os.path.join(C.root_dir, 'ckpts', 'pretrained_ckpts', '/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/cityscapes_best.pth')

Colab Output: [ours][INFO] validating segment_me_anomaly dataset with energy ... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:17<00:00, 1.74s/it] [ours][CRITICAL] AUROC score for segment_me_anomaly: 0.9806 [ours][CRITICAL] AUPRC score for segment_me_anomaly: 0.8854 [ours][CRITICAL] FPR@TPR95 for segment_me_anomaly: 0.0723 [numexpr.utils][INFO] NumExpr defaulting to 2 threads. INFO:numexpr.utils:NumExpr defaulting to 2 threads. [ours][INFO] validating segment_me_obstacle dataset with energy ... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [02:03<00:00, 4.11s/it] [ours][CRITICAL] AUROC score for segment_me_obstacle: 0.9996 [ours][CRITICAL] AUPRC score for segment_me_obstacle: 0.9672 [ours][CRITICAL] FPR@TPR95 for segment_me_obstacle: 0.0010 [ours][INFO] validating Fishyscapes_static dataset with energy ... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [02:02<00:00, 4.10s/it] [ours][CRITICAL] AUROC score for Fishyscapes_static: 0.9973 [ours][CRITICAL] AUPRC score for Fishyscapes_static: 0.9246 [ours][CRITICAL] FPR@TPR95 for Fishyscapes_static: 0.0085 [ours][INFO] validating road_anomaly dataset with energy ... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [01:48<00:00, 1.80s/it] [ours][CRITICAL] AUROC score for road_anomaly: 0.9572 [ours][CRITICAL] AUPRC score for road_anomaly: 0.7161 [ours][CRITICAL] FPR@TPR95 for road_anomaly: 0.1774 [ours][INFO] validating cityscapes dataset ... 0%| | 0/3 [00:00<?, ?it/s] Traceback (most recent call last): File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/test.py", line 126, in main(-1, 1, config=config, args=args) File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/test.py", line 96, in main valid_epoch(model, engine, cityscapes_val, vis_tool, evaluator=evaluator, logger=logger) File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/valid.py", line 353, in valid_epoch img, label = test_set[idx] File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/dataset/training/cityscapes.py", line 109, in getitem image = Image.open(self.images[index]).convert('RGB') File "/usr/local/lib/python3.10/dist-packages/PIL/Image.py", line 3227, in open fp = builtins.open(filename, "rb") IsADirectoryError: [Errno 21] Is a directory: '/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/cityscapes/images/city_gt_fine/val/munster'

Can you help me about this error? Thanks

[yyliu01]

Hi @feritstar

I appologise for the late reply.

I believe the issue is caused by the different folder structures, where ours is as follows: city_scape/ ├── annotation │ └── city_gt_fine │ ├── train │ └── val └── images └── city_gt_fine ├── train └── val coco/ ├── annotations │ └── ood_seg_train2017 └── train2017 .

Please feel free the modify the code, or alternatively you can download our dataset directly. Please see this page for more details.

Thanks for your interest.

Cheers, Yuyuan.

[feritstar]

This folder structure is different that you mentioned above. Which one is correct? I've been trying about 3 weeks to make it work. I had to changed your training/cityscapes.py => CityScapes.getItem() method to give proper self.images and self.targets. After that, utils/metric.py gives an error about predict has different size of target, so that, predict==target expression gives runtime error.

Both installation.md and beforestart.md files have lack of required information. Please can you update immediately?

[feritstar]

And please do not close an issue if it doesn't solved really. This is second time you did close an issue without really solving. This doesn't helpful.

I believe that you are being realistic about your scores!

[yyliu01]

Hey @feritstar,

I apologise to hear that, and it is quite surprising to me that you have been working on this issue for three weeks.

I have double-checked my folder structure, and I believe it is correct. With one additional level, we have the following tree structure:

Again, if you can't get it to work, please download the Cityscape dataset directly from this page, then you don't need to modify anything.

Edit: You can find the Google Drive link to our Cityscape dataset here: https://drive.google.com/file/d/1QNW_qRtCnL_Gn0blmHzxblfORCatiZb_/view?usp=drive_link and this link is also available on the page. Please see highlighted You can alternatively download both preprocessed fishyscapes & cityscapes datasets from here.

Cheers, Yuyuan

[feritstar]

Hey @yyliu01,

Thanks for quick reply. I will try this dataset. I just want to ask you this: Is this Cityscapes _leftImg8bit_trainvaltest.zip (11GB) [md5]_ Dataset??

Because, First I downloaded gtCoarse from Cityscapes and it gave me some directory errors as you saw in my first issue. And then I checked your code and saw you split leftImg8bit string from image names. After that I've downloaded leftImg8bit dataset. Is it correct one? Or which one is correct_?

My regards. Ferit

[yyliu01]

Hey @feritstar

1). Yes, it is.

2). The original Cityscape dataset consists of 33 categories. Typically, these are merged into 19 categories, as the objects in the remaining categories are not included in the validation/test sets. This process (https://github.com/mcordts/cityscapesScripts/tree/master/cityscapesscripts/preparation) might alter the prefix of the file names or even the folder structure, because it is performed individually.

And the one we need to download is NOT gtCoarse, we use only gtFine_trainvaltest.zip (241MB) [md5] for the initial 33 categories training label and leftImg8bit_trainvaltest.zip (11GB) [md5] for training images.

I thought all these questions are addressed in this page.

Cheers, Yuyuan

[feritstar]

Hi @yyliu01,

I have downloaded your city_scape dataset from the link you gave above. I wanted to make fresh start and download your github repo. I downloaded necessary other datasets as your said in installation.md. I want to run an inference, so i made preparations on config.py file. And try to run test.py file on googleColab. All datasets and project files are on my drive account. However I'm getting an error. I added error below:

wandb: Currently logged in as: feritstar (team_feritstar). Use wandb login --relogin to force relogin W&B online. Running your script from this directory will now sync to the cloud. wandb: Currently logged in as: feritstar (team_feritstar). Use wandb login --relogin to force relogin wandb: Tracking run with wandb version 0.17.5 wandb: Run data is saved locally in /content/wandb/run-20240728_160221-qd6zs86i wandb: Run wandb offline to turn off syncing. wandb: Syncing run rpl.code+corocl wandb: ⭐️ View project at https://wandb.ai/team_feritstar/OoD_Segmentation wandb: 🚀 View run at https://wandb.ai/team_feritstar/OoD_Segmentation/runs/qd6zs86i [ours][INFO] validating segment_me_anomaly dataset with energy ... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00, 2.24it/s] [ours][CRITICAL] AUROC score for segment_me_anomaly: 0.9806 [ours][CRITICAL] AUPRC score for segment_me_anomaly: 0.8853 [ours][CRITICAL] FPR@TPR95 for segment_me_anomaly: 0.0723 [numexpr.utils][INFO] NumExpr defaulting to 12 threads. INFO:numexpr.utils:NumExpr defaulting to 12 threads. [ours][INFO] validating segment_me_obstacle dataset with energy ... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:26<00:00, 1.11it/s] [ours][CRITICAL] AUROC score for segment_me_obstacle: 0.9996 [ours][CRITICAL] AUPRC score for segment_me_obstacle: 0.9672 [ours][CRITICAL] FPR@TPR95 for segment_me_obstacle: 0.0010 [ours][INFO] validating Fishyscapes_static dataset with energy ... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:27<00:00, 1.10it/s] [ours][CRITICAL] AUROC score for Fishyscapes_static: 0.9973 [ours][CRITICAL] AUPRC score for Fishyscapes_static: 0.9247 [ours][CRITICAL] FPR@TPR95 for Fishyscapes_static: 0.0085 [ours][INFO] validating Fishyscapes_ls dataset with energy ... 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [04:29<00:00, 2.69s/it] [ours][CRITICAL] AUROC score for Fishyscapes_ls: 0.9939 [ours][CRITICAL] AUPRC score for Fishyscapes_ls: 0.7062 [ours][CRITICAL] FPR@TPR95 for Fishyscapes_ls: 0.0252 [ours][INFO] validating road_anomaly dataset with energy ... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:23<00:00, 2.52it/s] [ours][CRITICAL] AUROC score for road_anomaly: 0.9572 [ours][CRITICAL] AUPRC score for road_anomaly: 0.7160 [ours][CRITICAL] FPR@TPR95 for road_anomaly: 0.1775 [ours][INFO] validating cityscapes dataset ... 0%| | 0/500 [00:00<?, ?it/s] Traceback (most recent call last): File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/test.py", line 127, in main(-1, 1, config=config, args=args) File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/test.py", line 97, in main valid_epoch(model, engine, cityscapes_val, vis_tool, evaluator=evaluator, logger=logger) File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/valid.py", line 355, in valid_epoch pred = evaluator(img, model) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(args, **kwargs) File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/engine/evaluator.py", line 27, in forward processed_pred += self.scale_process(img_scale, (ori_rows, ori_cols), File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/engine/evaluator.py", line 98, in scale_process img_pad, margin = self.pad_image_to_shape(img, crop_size, cv2.BORDER_CONSTANT, value=0) File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/engine/evaluator.py", line 139, in pad_image_to_shape shape = self.get_2dshape(shape) File "/content/drive/MyDrive/ColabNotebooks/RPL_Work/RPL/rpl_corocl.code/engine/evaluator.py", line 154, in get_2dshape if not isinstance(shape, collections.Iterable): AttributeError: module 'collections' has no attribute 'Iterable'

The other datasets are worked successfully but again city_scape gave an error. I tried to debug this error and got to this "crop size" has no feature of collection.Iterable. How can I fix that?

My regards. Ferit

[yyliu01]

@feritstar https://stackoverflow.com/questions/59809785/i-get-a-attributeerror-module-collections-has-no-attribute-iterable-when-i

Did you try that? I think it is a version mismatch issue.

[feritstar]

@yyliu01 it worked!! Inference took 55 mins on Colab L4 GPU with High Ram. Thanks for help. You can close it, the problem is really solved for good.