NVlabs / NVAE

The Official PyTorch Implementation of "NVAE: A Deep Hierarchical Variational Autoencoder" (NeurIPS 2020 spotlight paper)
https://arxiv.org/abs/2007.03898
Other
999 stars 163 forks source link

tensorboard images not showing #8

Closed 23pointsNorth closed 3 years ago

23pointsNorth commented 3 years ago

Hi all,

I've been testing the default code for training mnist and custom rgb_64x64 dataset.

In both cases, I can see the resulting training graphs, but the images in the tensorboard/images section are not showing -- they look like a 1x1 black pixel/empty.

The shape of the visualized image for mnist reconsturction is (1, int(sqrt(batch)), 2*int(sqrt(batch))) as expected, but is not shown. Saving the image with torchvision.utils.save_image() saves the right size image.


What's the assumed tensorboard version?

tensorboard.version '1.15.0+nv' torch.version '1.6.0a0+9907a3e' torchvision.version '0.7.0a0'

arash-vahdat commented 3 years ago

Not sure if this has anything to do with the tensorboard version, but we were using tensorboard==2.0.1.

23pointsNorth commented 3 years ago

Indeed, updating tensorboard didn't have any positive effect. I am working within the container, though that shouldn't have any impact.

If you have time, can you pip list, so I can tag all the installed libs?

Lukelluke commented 3 years ago

hi,@23pointsNorth,

I used tensorboardx==2.1, and perform well in images part of tensorboard page.

In both cases, I can see the resulting training graphs, but the images in the tensorboard/images section are not showing -- they look like a 1x1 black pixel/empty.

Worth to mention that when i ran the data within about 10 epochs, there only can show Reconstruction part images, and there still black pixel in generated part.

So, in my personal opinion, update tensorboard version, and go ahead training process unless u have trained enough epochs.

Hoping this piece of experience can make any help for u !

image

image

arash-vahdat commented 3 years ago

@23pointsNorth, are you still having this issue? If yes, can you try running this command: pip install tensorboard --ignore-installed

Note that for the latest tensorboard you may need to add --bind_all: tensorboard --logdir $CHECKPOINT_DIR/eval-$EXPR_ID/ --bind_all