Closed xelmirage closed 6 years ago
@xelmirage I haven't tried IR/RGB conversion with this implementation and do not have an yaml file to share. But I am wondering about your setting.
1> How many IR and RGB images do you have? The fundamental assumption of the work is that there exists a joint distribution and we have a reasonable amount of images to approximate the joint distribution. If these do not hold, then the algorithm may fail.
2> IR and RGB images have different number of channels and different range values. Their reconstruction losses may not be comparable. You may need to balance them by playing with the parameter.
3> How many iterations have you trained. It takes several days in my computer to start seeing some reasonable results.
Finally, it would be great if you could share some images if possible.
@mingyuliutw Thank you for your attention. 1>At the beginning I wanted to do an exact repeat the IR/RGB conversion in the first edition of your paper, in which you mentioned that 31386 RGB images and 31191 IR images from KAIST dataset were used to train the Net. I took a wild guess that you may used set 2,5,7,10,11 to provide RGB images and set 1,3,6,9 to provide LWIR images. But set 5,10,11 are nighttime datasets, in my understanding, all the RGB images should be daytime images, so I used RGB images in set 1,2,7,8 instead of set 2,5,7,10,11. 2>I directly used the settings from synthia2cityscape.yaml, because in all the yamls provided, I think this conversion is the most similar one to IR/RGB conversion. The modified yaml is attached. The IR images are treated as RGB images with three same channels. kaist.zip
3>The training has about 250000 iterations so far and is not showing satisfying results. The gen_total_loss, dis_true_acc_0, and dis_fake_acc_0 are shown in the figures below:
The gen_total_loss increased after 8k iterations. because I'm not experienced in training networks, I don't know whether it's normal, in the training of CelebA dataset this also happens. So far the Net doesn't have a meaningful result, here are some results:
247500
225000
210000
165000
130000
1000000
85000
65000
30000
10000
2500
Thank you again for you precious time!
BTW, in the first version of the work, we used a context image idea. Basically, the input image is RGB+normalized y-coordinate. We also used the UNet architecture. I am not sure if this could be the difference. But these are something that I could thought of.
can you share the trained weights for this conversion please? thank you
Hi @mingyuliutw , I have tried the version01 of UNIT. Since the v01 branch doesn't have a yaml for TIR-to-RGB conversion either, I modified the kaist_day_and_night.yaml. The modifications are replacing the class_name for TIR image to "dataset_thermal_image" as luckily there is a class for TIR images in the dataset_image.py, and the channel for TIR images is set to 1. However, there is also no satisyfing results after a really long training. So I have following thoughts:
Thank you for your attention!
Some results of training using the version01 code:
gen_00055000
gen_00112500
gen_00150000
gen_00200000
@xelmirage
If you check the first version of the paper (I think you can find it in the arXiv website), you would see that in the implementation section, there is a paragraph about using context. Basically, we added one channel to both the RGB (3 channels to 4 channels) and thermal ( 1 channel to 2 channels) images where the additional channel is the normalized y-coordinates [0 1] of the pixels. I pointed out this in the early reply. I don't know if you noticed it. As I recall the version 1 code in the branch does not have this context image component implemented. You might need to implement the component to get better results.
@xelmirage and @xjoshramos
I do not have any pre-trained models (The true first version of the code was implemented in Caffe because PyTorch was not yet born. I reimplemented the algorithm using PyTorch and release the code for sharing.) from the first version anymore as I have shifted to my latest version and storing these models takes a lot of disk space.
BTW, if the goal is to make a comparison to the UNIT algorithm. I would recommend that you use the Cityscape and SYNTHIA datasets as images in the two datasets have high quality.
@mingyuliutw Is this the paper? https://arxiv.org/pdf/1703.00848v1.pdf
Yes
@mingyuliutw Thank you for your reply, as you first pointed out the context image component in your former replies, I checked the version01 and founded that version01 did have context image component. The related codes exist in dataset_image.py at line 26-34, and the yaml file also has context option switched to 1. So I'm pretty sure that I used the context image component as you mentioned.
@xelmirage I see. I forgot I reimplemented the context part. I guess in this case you need to explore various network design choices or various GAN training tricks.
@mingyuliutw Thank you very much for your help. Just one minor question, I am still wondering the details about the tranning dataset. Which part of KAIST images were chosen to make the 31386 IR images and 31191 RGB images... is this detail convenient to share?
@xelmirage , did you obtain any good results of the colourization from thermal images? If yes, can you please share how to reproduce your results!
Just to point there was a bug in the xy image creation as described in issues 39.
I see that the issue#39 is related to the context image component which was not part of the master branch at the time this issue was created. Does it mean I do not need to revert to version_01 for the image context part, and that the master branch would work just fine?
Sorry for the confusion. What’s I meant is the xy-image supposed to be useful for IR2RGB but it was not the case. I suspected it was due to the bug. That’s all. I will leave version one as it is.
Hello, I have the same concern, I need to convert IR images to RGB. Have you achieved acceptable result? @xelmirage I appreciate if you share your experience about this issue. Thx
Hi, Mingyu. I'm studying your amazing work about UNIT, and I'm very interested in UNIT's attractive poteintial of IR/RGB conversion. So I tried to repeat the experiment. Since the detailed parameters of training are not provided, I just tried borrowing settings in synthia2cityscape.yaml for the training. But the results were not satisfying. So I'm wondering is it right for me to do so? If it's not right to borrow settings in synthia2cityscape.yaml, is there any possibility that settings for IR/RGB conversion may be released? Thanks for your attention and amazing work:)