search

XuzheZ / PTNet3D

Public code for 3D version of Pyramid Transformer Network (PTNet). Our PTNet achieve consistently better performance in high-resolution and longitudinal infant brain MRI synthesis than previous SOTA methods.
Other
11 stars 4 forks source link

The generated image is incorrect #1

Closed XueyangWWW closed 2 years ago

XueyangWWW commented 2 years ago

Hi @XuzheZ Thanks so much for your work on this project !

I'd like to ask some questions about my use of this project. Why the image I input to the network is T1-W and the generated image is also T1-W? The uploaded image includes the original T1-W, T2-W, T2-W_20epo, and T2-W_60epo. The network parameters are basically not modified.

Best wishes!

Data.zip

XuzheZ commented 2 years ago

Hi @XueyangWWW

Thank you for the question. May I ask if you put the training scans (T1w and T2w) in the correct folder? If you are trying to convert T1w to T2w, in the training, the T1w MRI should be put in opt.dataroot/train_A, the T2w MRI should be put in opt.dataroot/train_B. I've updated the README about data structure and usage.

Please let me know if you have other questions.

Best, Xuzhe

XueyangWWW commented 2 years ago

Thank you for your timely reply. The dataset was correct. After asking questions, I doubled the epoch to 120, and got T2w after training and testing. The previous problem should be that the number of epochs was not enough. I will consult you again if I have any problems later.

XuzheZ commented 2 years ago

Hi @XueyangWWW

I am glad that it works. I apologize for the confusion here. In the original paper, we use an offline excessive sampling prior to training to sample 3D ROIs, and thus we have over 50,000 ROIs in training. Therefore, we only use a small number of epochs. However, in this repo, to simplify and smooth the workflow, we choose to conduct an online random sampling mechanism. Therefore, the amount of data in each epoch is dramatically reduced. For instance, to approach the number of iterations we achieved in offline datasampling, we need to set the epoch to 2,000 while using online sampling. For more details, please refer to the updated README.

Please don't hesitate to reach out to me with other questions.

All the best, Xuzhe

XueyangWWW commented 2 years ago

Hi @XuzheZ

Thank you very much for your answer to my question. Now, I have adjusted the epoch to 200, and the result is getting better and better, but the problem still pressing is that there is a boundary between patches in the resulting image, maybe the epoch is not enough? Thankfully, 400 epochs only took about 36 hours, and I can keep adding epochs to see if this problem can be improved. I only have 120 3D training datasets, do I really need thousands of epochs? wow

Best wishes

XuzheZ commented 2 years ago

Hi @XueyangWWW

The checkerboard (boundary) artifacts should be resolved by sliding sampling during inference. I will suggest you change the stride during sliding sampling. You may find the sliding sampling in test.py line 43. By default, stride was set as 'tuple(dim//2 for dim in opt.patch_size)'. If you are using a patch size of 64, you may test 48 as stride. A larger stride will reduce the overlapping among patches. And those overlapped pixels will be averaged by the number of overlapping times (test.py line 55).

In my experiments, a sufficiently trained model will not have any boundary between patches (as long as the stride is smaller than the patch size). If you still observe that artifact using patches sampled from sliding sampling, it may suggest that you should increase your training epoch.

Actually, with online sampling, using thousands of epochs is not unreasonable. Considering you have 120 3D volumetric scans, assuming a batch size of 4, in each epoch, you will only have 30 batches of sampled 3D patches, and the model is only trained for 30 iterations. Even with 1,000 epochs, the model is trained for 30,000 iterations - it is not a huge number for deep learning models, especially for transformers.

Please let me know if you have other questions.

Best, Xuzhe

XueyangWWW commented 2 years ago

Hi @XuzheZ

Thank you very much for your careful guidance! Since the last question, when I increased the epoch to 1000 and took your suggestion, the results have improved greatly. The Checkerboard has become so subtle that it cannot be seen without looking closely, and the results are very impressive. It will then try again to see if it can make the results better.

Thanks again for your hard work and helpful answer!

Best wishes Xueyang

XuzheZ commented 2 years ago

Great! Glad to know. I still suggest increasing the epoch - with about 300 scans, we need around 2,000 epochs to achieve the results we reported in the paper and showed in the README of this repo.

Please don't hesitate to reach out if you encounter other issues. Enjoy!