Can't run any type of analysis after training

[liashchynskyi]

For example, when I try to execute FA I get this error

[INFO] 2022-04-04 05:47:59 > Run frequency analysis (use 8 fake and valid images).
Traceback (most recent call last):
  File "src/main.py", line 186, in <module>
    hdf5_path=hdf5_path)
  File "/root/gan-experiments/src/loader.py", line 438, in load_worker
    worker.run_frequency_analysis(dataloader=eval_dataloader)
  File "/root/gan-experiments/src/worker.py", line 1253, in run_frequency_analysis
    N, C, H, W = np.shape(real_array)
UnboundLocalError: local variable 'real_array' referenced before assignment

My command is python src/main.py -cfg CFG -hdf5 -l -save SAVE -fa -ckpt CKPT_DIR -data DATA -ref="valid"

[liashchynskyi]

I get the same error for knn

tsne also doesn't work

[liashchynskyi]

tsne

[t-SNE] Mean sigma: 2.670893 [t-SNE] KL divergence after 250 iterations with early exaggeration: 73.358437 [t-SNE] KL divergence after 300 iterations: 1.787954 Traceback (most recent call last): File "src/main.py", line 186, in hdf5_path=hdf5_path) File "/root/gan-experiments/src/loader.py", line 443, in load_worker worker.run_tsne(dataloader=eval_dataloader) File "/root/gan-experiments/src/worker.py", line 1365, in run_tsne logging=self.global_rank == 0 and self.logger) File "/root/gan-experiments/src/utils/misc.py", line 463, in plot_tsne_scatter_plot alpha=0.5).legend(fontsize=15, loc="upper right") File "/root/anaconda3/lib/python3.7/site-packages/seaborn/relational.py", line 1402, in scatterplot alpha=alpha, x_jitter=x_jitter, y_jitter=y_jitter, legend=legend, File "/root/anaconda3/lib/python3.7/site-packages/seaborn/relational.py", line 898, in init self.parse_hue(plot_data["hue"], palette, hue_order, hue_norm) File "/root/anaconda3/lib/python3.7/site-packages/seaborn/relational.py", line 378, in parse_hue list(data), order, palette File "/root/anaconda3/lib/python3.7/site-packages/seaborn/relational.py", line 221, in categorical_to_palette raise ValueError(err) ValueError: The palette list has the wrong number of colors.

[mingukkang]

For me, it works well.

Could you please give me more information about your situation?

By the way, I used the following command:

CUDA_VISIBLE_DEVICES=0 python src/main.py -cfg "./src/configs/CIFAR10/SAGAN.yaml" -hdf5 -l -save "/root/code/studiogan" -fa -tsne -knn -ckpt "/root/ext_data/studiogan_ckpt/CIFAR10/checkpoints/CIFAR10-SAGAN-train-2022_01_13_01_21_17"
-data "/root/code/data/CIFAR10" -ref="test" -metrics "none"

And the results are

[liashchynskyi]

Oh, I see that you set metrics to none. Could it be the reason that it worked?

[mingukkang]

The code works well even if it isn't none.

[liashchynskyi]

Hmm, interesting. So the commands are almost the same but it doesn't work in my case. Ok, I'll give it a try a bit later with your command and let you know.

[liashchynskyi]

Nope, doesn't work

Command

CUDA_VISIBLE_DEVICES=0 python src/main.py -cfg "./src/configs/Custom/BIGGAN_CYT.yaml" -hdf5 -l -save "/root/gan-experiments/save/biggan_cyt" -v -knn -fa -tsne -sf -sf_num 300 -ckpt "/root/gan-experiments/save/biggan_cyt/checkpoints/cyt_64-BIGGAN_CYT-train-2022_04_04_06_02_59" -data "/root/gan-experiments/data/cyt_64" -ref="valid" -metrics="none"

Which Python version are you using? Mine is 3.7.6

[mingukkang]

Mine is 3.7.10.

---

[INFO] 2022-04-04 05:47:59 > Run frequency analysis (use 8 fake and valid images).
Traceback (most recent call last):
  File "src/main.py", line 186, in <module>
    hdf5_path=hdf5_path)
  File "/root/gan-experiments/src/loader.py", line 438, in load_worker
    worker.run_frequency_analysis(dataloader=eval_dataloader)
  File "/root/gan-experiments/src/worker.py", line 1253, in run_frequency_analysis
    N, C, H, W = np.shape(real_array)
UnboundLocalError: local variable 'real_array' referenced before assignment

Let see the first comment. The error message tells us the local variable "real_array" is not defined. This indicates that the variable "number of batches" is under 1, so the for loop is not executed. Could you tell me the structure of the dataset you used ?

for i in range(num_batches):
    real_images, real_labels = next(data_iter)
    fake_images, fake_labels, _, _, _, _, _ = sample.generate_images(z_prior=self.MODEL.z_prior,
                                                              truncation_factor=self.RUN.truncation_factor,
                                                              batch_size=self.OPTIMIZATION.batch_size,
                                                              z_dim=self.MODEL.z_dim,
                                                              num_classes=self.DATA.num_classes,
                                                              y_sampler="totally_random",
                                                              radius="N/A",
                                                              generator=generator,
                                                              discriminator=self.Dis,
                                                              is_train=False,
                                                              LOSS=self.LOSS,
                                                              RUN=self.RUN,
                                                              MODEL=self.MODEL,
                                                              device=self.local_rank,
                                                              is_stylegan=self.is_stylegan,
                                                              generator_mapping=generator_mapping,
                                                              generator_synthesis=generator_synthesis,
                                                              style_mixing_p=0.0,
                                                              stylegan_update_emas=False,
                                                              cal_trsp_cost=False)
    fake_images = fake_images.detach().cpu().numpy()

    real_images = np.asarray((real_images + 1) * 127.5, np.uint8)
    fake_images = np.asarray((fake_images + 1) * 127.5, np.uint8)

    if i == 0:
        real_array = real_images
        fake_array = fake_images
    else:
        real_array = np.concatenate([real_array, real_images], axis=0)
        fake_array = np.concatenate([fake_array, fake_images], axis=0)

Thank you.

[liashchynskyi]

Dataset structure:

data -> DATASET_NAME -> then this

[mingukkang]

Hmm... Could you print "num_batches" variable ?

    def run_frequency_analysis(self, dataloader):
        if self.global_rank == 0:
            self.logger.info("Run frequency analysis (use {num} fake and {ref} images).".\
                             format(num=len(dataloader), ref=self.RUN.ref_dataset))
        if self.gen_ctlr.standing_statistics:
            self.gen_ctlr.std_stat_counter += 1

        requires_grad = self.LOSS.apply_lo or self.RUN.langevin_sampling
        with torch.no_grad() if not requires_grad else misc.dummy_context_mgr() as ctx:
            misc.make_GAN_untrainable(self.Gen, self.Gen_ema, self.Dis)
            generator, generator_mapping, generator_synthesis = self.gen_ctlr.prepare_generator()

            data_iter = iter(dataloader)
            num_batches = len(dataloader) // self.OPTIMIZATION.batch_size
            for i in range(num_batches):
                real_images, real_labels = next(data_iter)
                fake_images, fake_labels, _, _, _, _, _ = sample.generate_images(z_prior=self.MODEL.z_prior,
                                                                          truncation_factor=self.RUN.truncation_factor,
                                                                          batch_size=self.OPTIMIZATION.batch_size,
                                                                          z_dim=self.MODEL.z_dim,
                                                                          num_classes=self.DATA.num_classes,
                                                                          y_sampler="totally_random",
                                                                          radius="N/A",
                                                                          generator=generator,
                                                                          discriminator=self.Dis,
                                                                          is_train=False,
                                                                          LOSS=self.LOSS,
                                                                          RUN=self.RUN,
                                                                          MODEL=self.MODEL,
                                                                          device=self.local_rank,
                                                                          is_stylegan=self.is_stylegan,
                                                                          generator_mapping=generator_mapping,
                                                                          generator_synthesis=generator_synthesis,
                                                                          style_mixing_p=0.0,
                                                                          stylegan_update_emas=False,
                                                                          cal_trsp_cost=False)
                fake_images = fake_images.detach().cpu().numpy()

                real_images = np.asarray((real_images + 1) * 127.5, np.uint8)
                fake_images = np.asarray((fake_images + 1) * 127.5, np.uint8)

                if i == 0:
                    real_array = real_images
                    fake_array = fake_images
                else:
                    real_array = np.concatenate([real_array, real_images], axis=0)
                    fake_array = np.concatenate([fake_array, fake_images], axis=0)

            N, C, H, W = np.shape(real_array)
            real_r, real_g, real_b = real_array[:, 0, :, :], real_array[:, 1, :, :], real_array[:, 2, :, :]
            real_gray = 0.2989 * real_r + 0.5870 * real_g + 0.1140 * real_b
            fake_r, fake_g, fake_b = fake_array[:, 0, :, :], fake_array[:, 1, :, :], fake_array[:, 2, :, :]
            fake_gray = 0.2989 * fake_r + 0.5870 * fake_g + 0.1140 * fake_b
            for j in tqdm(range(N)):
                real_gray_f = np.fft.fft2(real_gray[j] - ndimage.median_filter(real_gray[j], size=H // 8))
                fake_gray_f = np.fft.fft2(fake_gray[j] - ndimage.median_filter(fake_gray[j], size=H // 8))

                real_gray_f_shifted = np.fft.fftshift(real_gray_f)
                fake_gray_f_shifted = np.fft.fftshift(fake_gray_f)

                if j == 0:
                    real_gray_spectrum = 20 * np.log(np.abs(real_gray_f_shifted)) / N
                    fake_gray_spectrum = 20 * np.log(np.abs(fake_gray_f_shifted)) / N
                else:
                    real_gray_spectrum += 20 * np.log(np.abs(real_gray_f_shifted)) / N
                    fake_gray_spectrum += 20 * np.log(np.abs(fake_gray_f_shifted)) / N

        misc.plot_spectrum_image(real_spectrum=real_gray_spectrum,
                                 fake_spectrum=fake_gray_spectrum,
                                 directory=join(self.RUN.save_dir, "figures", self.run_name),
                                 logger=self.logger,
                                 logging=self.global_rank == 0 and self.logger)

        misc.make_GAN_trainable(self.Gen, self.Gen_ema, self.Dis)

[liashchynskyi]

I'll try a bit later. Could it be the consequence that my dataset is unbalanced? I mean each class consists of different count of images.

[mingukkang]

Could it be the consequence that my dataset is unbalanced? ==> I do not think so. Hope the problem is resolved.

Thank you.

Best,