Getting IndexError while Running run_metrics.py with Places2

[hamzapehlivan]

Hi,

Thanks for sharing this great work!

I have a question about running run_metrics.py with Places2 dataset.

As it is indicated in Datasets section, I downloaded the validation set of Places2, and converted into TFRecords with --shuffle --compressed flags.

However, when I try to run `run_metrics.py' (I set metrics=idk36k5), I got the following:

dnnlib: Running run_metrics.run() on localhost... Evaluating metrics "ids36k5" for "models/co-mod-gan-places2-050000.pkl"... Setting up TensorFlow plugin "fused_bias_act.cu": Preprocessing... Loading... Done. Setting up TensorFlow plugin "upfirdn_2d.cu": Preprocessing... Loading... Done. truncation=None Traceback (most recent call last): File "run_metrics.py", line 80, in main() File "run_metrics.py", line 75, in main dnnlib.submit_run(sc, 'run_metrics.run', kwargs) File "/workspace/dnnlib/submission/submit.py", line 343, in submit_run return farm.submit(submit_config, host_run_dir) File "/workspace/dnnlib/submission/internal/local.py", line 22, in submit return run_wrapper(submit_config) File "/workspace/dnnlib/submission/submit.py", line 280, in run_wrapper run_func_obj(submit_config.run_func_kwargs) File "/workspace/run_metrics.py", line 30, in run num_gpus=num_gpus, num_repeats=num_repeats, resume_with_new_nets=resume_with_new_nets, truncations=truncations) File "/workspace/metrics/metric_base.py", line 188, in run metric.run(*args, kwargs) File "/workspace/metrics/metric_base.py", line 82, in run self._evaluate(Gs, Gs_kwargs=Gs_kwargs, num_gpus=num_gpus) File "/workspace/metrics/inception_discriminative_score.py", line 35, in _evaluate self._configure(self.minibatch_per_gpu, hole_range=self.hole_range) File "/workspace/metrics/metric_base.py", line 168, in _configure return self._get_dataset_obj().configure(minibatch_size, hole_range=hole_range) File "/workspace/metrics/metric_base.py", line 153, in _get_dataset_obj self._dataset_obj = dataset.load_dataset(data_dir=self._data_dir, self._dataset_args) File "/workspace/training/dataset.py", line 250, in load_dataset dataset = dnnlib.util.get_obj_by_name(class_name)(**kwargs) File "/workspace/training/dataset.py", line 87, in init self.resolution = resolution if resolution is not None else max_shape[1] IndexError: list index (1) out of range

When I debug the code, I realized that tfr_shapes=[[74989]] in training/dataset.py line 82. Here is the "features" dictionary without bytes_list:

{'num_val_images': int64_list { value: 36500 } , 'shape': int64_list { value: 74989 } , 'compressed': int64_list { value: 1 } }

I am using the provided Docker image.

[zsyzzsoft]

Oh, you need to specify '--resolution=512'

[hamzapehlivan]

Thanks for the reply, it now worked!

However, I was not able to replicate the results of the paper. In the paper, the followings were reported for Places2 dataset.

P-IDS: 13.3 ±0.1, U-IDS: 27.4 ±0.1, FID: 7.9 ±0.0

What I got: P-IDS 11.6, U-IDS 25.9 , FID 8.0788

Running it several times also led to the same slightly inaccurate results.

I have the same issue with FFHQ Dataset, too.

One more point is that I got very similar results to paper when I specified the masked ratio between 0 and 0.2

Any idea why this happens?

[zsyzzsoft]

Quite strange. What is your result for FFHQ?

[hamzapehlivan]

For the metric ids10k-h0, I am getting:

P-IDS: 30.0, U-IDS 44.5, FID: 0.5483

For the metric ids10k, I am getting: P-IDS:14.8, U-IDS: 28.2, FID: 3.8678

[zsyzzsoft]

What is your numpy and tf version?

[hamzapehlivan]

My TF version is "1.15.0", and my Numpy version is "1.17.3".

I am using the provided Dockerfile. However, metric evaluation was not working, therefore, I changed it a bit:

FROM tensorflow/tensorflow:1.15.0-gpu-py3 RUN pip install scipy==1.3.3 \ requests==2.22.0 \ Pillow==6.2.1 \ tqdm \ imageio \ scikit-learn

If it helps, here is my log.txt:

dnnlib: Running run_metrics.run() on localhost... Evaluating metrics "ids10k" for "models/co-mod-gan-ffhq-9-025000.pkl"... Setting up TensorFlow plugin "fused_bias_act.cu": Preprocessing... Loading... Done. Setting up TensorFlow plugin "upfirdn_2d.cu": Preprocessing... Loading... Done. truncation=None 100%|##########| 1250/1250 [08:05<00:00, 2.57it/s] co-mod-gan-ffhq-9-025000 time 9m 42s ids10k-FID 3.8398 ids10k-U 0.2777 ids10k-P 0.1442 dnnlib: Finished run_metrics.run() in 9m 56s.

[zsyzzsoft]

I still have no idea what causes this error... One hypothesis is that the randomness of the mask generator's behavior has changed, which may slightly bias the difficulty of the task