Help with precompute_knn

[Kampmarsvin]

precompute_knns.py looks for a specific folder which does not exist. It looks for STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val which is not among the downloaded datasets. I understand that the path is somehow generated on the basis of the config file, but how do I point it to one of the existing datasets instead?

[tanveer6715]

Change your data path train_config.yml file. Then it will generate nns folder by itself.

[Kampmarsvin]

Maybe my question was a bit unclear. I tried changing the path to potsdam, but still get this weird autogenerated path. Here is the error message output_root: ../ pytorch_data_dir: /home/STEGO/src/datadrive/pytorch-data experiment_name: exp1 log_dir: potsdam azureml_logging: true submitting_to_aml: false num_workers: 12 max_steps: 5000 batch_size: 16 num_neighbors: 7 dataset_name: potsdam dir_dataset_name: null dir_dataset_n_classes: 5 has_labels: false crop_type: None crop_ratio: 0.5 res: 224 loader_crop_type: center extra_clusters: 0 use_true_labels: false use_recalibrator: false model_type: vit_small arch: dino use_fit_model: false dino_feat_type: feat projection_type: nonlinear dino_patch_size: 8 granularity: 1 continuous: true dim: 70 dropout: true zero_clamp: true lr: 0.0005 pretrained_weights: null use_salience: false stabalize: false stop_at_zero: true pointwise: true feature_samples: 11 neg_samples: 5 aug_alignment_weight: 0.0 correspondence_weight: 1.0 neg_inter_weight: 0.63 pos_inter_weight: 0.25 pos_intra_weight: 0.67 neg_inter_shift: 0.46 pos_inter_shift: 0.12 pos_intra_shift: 0.18 rec_weight: 0.0 repulsion_weight: 0.0 crf_weight: 0.0 alpha: 0.5 beta: 0.15 gamma: 0.05 w1: 10.0 w2: 3.0 shift: 0.0 crf_samples: 1000 color_space: rgb reset_probe_steps: null n_images: 5 scalar_log_freq: 10 checkpoint_freq: 50 val_freq: 100 hist_freq: 100

Global seed set to 0 ../data ../ Since no pretrained weights have been provided, we load the reference pretrained DINO weights. /home/STEGO/src/datadrive/pytorch-data/nns/nns_vit_small_cocostuff27_val_five_224.npz not found, computing Error executing job with overrides: [] Traceback (most recent call last): File "precompute_knns.py", line 78, in my_app cfg=cfg, File "/home/STEGO/src/data.py", line 495, in init target_transform=target_transform, **extra_args) File "/home/STEGO/src/data.py", line 380, in init self.num_images = len(os.listdir(self.img_dir)) FileNotFoundError: [Errno 2] No such file or directory: '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'

[tanveer6715]

pytorch_data_dir

You should change your data directory here to the potsdam data where it exists in your system.

[Kampmarsvin]

The path '/home/STEGO/src/datadrive/pytorch-data' which I have provided in PyTorch_data_dir is the path to the potsdam directory. Yet precompute_knns.py apparently looks for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. I cannot figure out, why that path is being generated. Where in the config file is anything indicating that I want to look for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. Can you @tanveer6715 show me an example of a config file, which points to potsdam?

[tanveer6715]

The path '/home/STEGO/src/datadrive/pytorch-data' which I have provided in PyTorch_data_dir is the path to the potsdam directory. Yet precompute_knns.py apparently looks for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. I cannot figure out, why that path is being generated. Where in the config file is anything indicating that I want to look for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. Can you @tanveer6715 show me an example of a config file, which points to potsdam?

Actually I just trained it with custom dataset not with Potsdam. I simply change data directory to my dataset path. Then it generate precompute_knns successfully. I am not sure why it looks for that path in potsdam dataset.

[Kampmarsvin]

Ok. Thank you. Maybe I'll try testing with a custom set

[Guzaiwang]

The path '/home/STEGO/src/datadrive/pytorch-data' which I have provided in PyTorch_data_dir is the path to the potsdam directory. Yet precompute_knns.py apparently looks for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. I cannot figure out, why that path is being generated. Where in the config file is anything indicating that I want to look for '/home/STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val'. Can you @tanveer6715 show me an example of a config file, which points to potsdam?

You may first run the following code: python crop_datasets.py

[Kampmarsvin]

@Guzaiwang but crop_dataset.py is for training on custom dataset right? I am simply trying to follow the explicit steps in the readme file, and it says to call precompute_knns.py when using pretrained model. So I have the models downloaded, and I have the datasets. But precompute_knns doesn't work because it looks for a different folder than the one specified in the config file. How do I change the path, so that it finds the path to the potsdam dataset?

[KennyChen880127]

@Kampmarsvin Hellos sir, Do you fix this issue? I have same problem now,it drives me crazy.

[Kampmarsvin]

@Kampmarsvin Hellos sir, Do you fix this issue? I have same problem now,it drives me crazy.

Unfortunately not. I haven't had the time to do it. But I found out that there are som folders that need to be unzipped, so I got a little closer

[innat-asj]

precompute_knns.py looks for a specific folder which does not exist. It looks for STEGO/src/datadrive/pytorch-data/cropped/cocostuff27_None_crop_0.5/img/val which is not among the downloaded datasets. I understand that the path is somehow generated on the basis of the config file, but how do I point it to one of the existing datasets instead?

In the precompute_knn.py file, this supported data are hard coded. If you want to test one particular data among this (let's say potsdam), you need to update as follows:

# dataset_names = ["cocostuff27", "cityscapes", "potsdam"]
dataset_names = ["potsdam"]

Then run (instruction)

python precompute_knns.py