Not working for other datasets

[ghost]

Hi, great work on this project, I was able to get outputs using your face dataset see output 100000 attached but not for my own dataset. One I tried 10 images and it was giving random outputs at 100000 and the other I tried 36 images and it was black output even up to 50000 so I stopped it. Does the data have to be all aligned and same direction for it to work?

[volotat]

It should be aligned in a sense that all images must have same resolution. But no need to have special constraints on the data itself. I used kind of weird approach to store data in this project though...

Can you share your data file here so I can say if there is something wrong?

By the way, if black images start to appear after some trining time, than it is deffenetly problem in Adam optimizer. It have tendensy to break after some time when gradient became too small or too high. To resolve it you can make smaller learning rate and Adam's clipnorm parameter.

[ghost]

also do you have a script that you used to get your datasets like that for faces, animals etc?

[volotat]

@ak9250 I see the problem. Your image is 5000x500px while script looking for 200x200 squares. So make it 2000x200 and it should work just fine. Alternatively you can make INPUT_IMG_SIZE parameter 500.

And I had not use any script for that "datasets", there is not so much images to script it, just picked them by hands.

[ghost]

@Mylittlerapture ok great, Ill change the input_img_size and see if that changes the ouput.

Also does the input need to be cropped like a face only looking straight or can it be like different portraits that have faces with some without faces, so basically no data prep on the input image other than combined and making sure same resolution.

[volotat]

@ak9250 That's right, as I said there is no strict constraints on data. But obviously some things will work better than others. If there is some similarity in head position, for example, it easier for the network to understand relationships between images.

[volotat]

@ak9250 This is problem with Adam that I described earlier. It is not overfitting but problem with optimizer algorithm. The best advice I can give is to change learning rate and run it again from last not broken point. You can do it by copying models from 'saved_model' folder into script directory.

[ghost]

@Mylittlerapture ok just to make sure you mean to make these two lower for the two arguments, what number would you suggest?

[volotat]

@ak9250 There is no way to tell. You have to experiment with it.

[ghost]

@Mylittlerapture ok thanks i set it to this optimizer = Adam(0.0001, 0.5, clipnorm = 0.5) opt_small = Adam(0.00005, 0.5, clipnorm = 0.5)

[ghost]

@Mylittlerapture I am getting this memory usage error "Memory usage is close to the limit Your runtime is close to its memory limit. Further use may cause an out-of-memory crash and the loss of in-memory state for all sessions. Would you like to terminate some sessions in order to free up memory (state will be lost for those sessions)? Currently, 10.30 GB / 12.72 GB is being used."

size: (500, 500) imarr shape: (16500000, 3) grid shape: (250000, 2) grid shape: (16500000, 2) noise shape: (16500000, 66) /usr/local/lib/python3.6/dist-packages/keras/engine/training.py:490: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set model.trainable without calling model.compile after ? 'Discrepancy between trainable weights and collected trainable' 50 0.0026383342419285328 100 0.0009095268294913694 150 0.0008270545344566927 200 0.0009025334857869894 250 0.000910461038001813 300 0.0008266386564355344 350 0.0007749594975030049 400 0.000808201598119922 450 0.0006936696471530013 500 0.0007972916922881268 Img: 1 Img: 2 Img: 3 Img: 4 Img: 5 Img: 6 Img: 7 Img: 8 Img: 9 550 0.0006353477478842251 600 0.0007823277811985463 650 0.0007204865035600961 700 0.000742982859374024 750 0.0007568690326297656 800 0.0007215695083141327 850 0.000696479226462543 900 0.0007833414818742313 950 0.0007394474928150885 1000 0.0007551153778331354 Img: 1 Img: 2 Img: 3 Img: 4 Img: 5 Img: 6 Img: 7 Img: 8 Img: 9 1050 0.0007473280176054686 1100 0.0006938337002065964 1150 0.0007237436837749556

I am using google colab and can share my notebook if that helps

[volotat]

@ak9250 Yeah, I forget to tell, that SIZE paramenter is hightly importatant, beacause each pixel treated as individual datapoint. And given the actual performance of the network even 200x200px is defenetly an overkill (It cannot produce images with this amount of details). So, you probably should make images and SIZE parameter way smaller.

[ghost]

@Mylittlerapture ok thanks I am trying it with 50x50px now

[ghost]

@Mylittlerapture is it possible you can try it on the dataset I linked and see what you get, I also have a colab notebook I can share which I am running it on that has free gpu for 12 hours

[ghost]

@Mylittlerapture so I actually checked and now i have 2000x2000 image similar to your face.png file. I have 66 images in 11x6 grid which is 2000x2000 so each image is 333x181 but I used input_img_size 200 because im not sure how to get it to fit. I put 66 for the examples.

[volotat]

@ak9250 Sorry, but I didn't have time for it right now.

66 images in 11x6 grid which is 2000x2000 so each image is 333x181 but I used input_img_size 200 because im not sure how to get it to fit. I put 66 for the examples.

No no no... Each image should be SIZE x SIZE scale and size of whole image should be 2200x1200 in your case (11x6 for 200x200 images)

[ghost]

@ak9250 Sorry, but I didn't have time for it right now.

66 images in 11x6 grid which is 2000x2000 so each image is 333x181 but I used input_img_size 200 because im not sure how to get it to fit. I put 66 for the examples.

No no no... Each image should be SIZE x SIZE scale and size of whole image should be 2200x1200 in your case (11x6 for 200x200 images)

I used a collage maker first to get all 66 images into a grid https://github.com/hanzhi713/image-collage-maker then I used resize

from PIL import Image img = Image.open('collage.jpg') new_img = img.resize((2000,2000)) new_img.save("collage_resized2.jpg", "JPEG", optimize=True)

so it is 2000x2000 before i run it

also how to you calculate the number to put in input_img_size, I get it is the 200 for 200x200 but for 333x181?

[jwb95]

For now, I also train with a small and complex dataset (different human poses, mostly 2 figures in grayscale) with 42 images. Right now I am at 15k epochs and as you can see some images look promising whereas others turn out black. Is this a failure mode or is there a good chance for it to recover? It actually has been like this from the beginning (sometimes showing more, sometimes fewer black images).

Edit: Training is on 55k by now and although black images remain, the quality of the generated images improves. Btw: I did use the original learning rates of the code.