Training a denoising model on single exposure micrographs (no frames/movies), even possible?

[DennisDannecker]

Hello Topaz Team,

I would like to ask a question regarding the required input for the training of a denoiser model.

The Jupyter Notebook (denoising.ipynb) explains and illustrates well how to use a pre-trained denoising model but also how to train a model using one's own data.

I have single micrographs which did not originate from any movie recording but from a simple single exposure (no frames). I was wondering whether it is possible to use single micrographs as input for the training of a denoising model. Understanding the pre-processing for preparing the training data and the theoretical underlyings of noise2noise, "To train new denoising models, topaz denoise requires paired, independent samples from the same underlying signal, because we use the noise2noise training framework. This means we need paired images of the exact same content. Fortunately, these are easy to get from the raw camera movie frames." it appears to me difficult if not impossible to use single micrographs as input for training a model. I was wondering whether there is a way of implementing it or modifying the single micrographs so that it would be possible to use them as input. All single micrographs are different exposures.

I would appreciate a short comment on this and potential possibilities if any.

Thank you very much.

Kind regards Dennis

[Guillawme]

Hello,

All single micrographs are different exposures.

This is incompatible with the assumptions of noise2noise. You would need two micrographs of the exact same field of view, and even then I am not sure it would work equally well as when using movie frames: your two micrographs would have a relatively high dose, meaning the one acquired second would have significantly more radiation damage than the first one, so the signal content between the two micrographs could be different enough to not fulfill the assumption of noise2noise. This is easy to work around with movie frames by grouping even and odd frames, resulting in two groups with identical signal but independent noise.

Don't you get good results denoising your micrographs with the pre-trained models?

[DennisDannecker]

Hey Guillawme,

thank you for your quick response. This is what I thought. This is principally not really possible.

I have tried with all available pre-trained models. Unfortunately, all pre-trained models perform poorly that is they barely improve the contrast or noise level, in other words the micrographs display no noticeable change after denoising. This is not to surprising since the models were trained based on conventional TEM, but the imaging method used to generate the micrographs I intend to denoise is much different and thus I expect the nature of the noise is fundamentally different, hence my intention to train the model. Anyway, I think there are some interesting alternatives. Thank you again for your replay.

Best regards, Dennis

[Guillawme]

Maybe try JANNI? It is also a noise2noise denoiser, but probably trained on a different training set than topaz denoise (not sure, you might have to dig this up from their respective papers). In any case, they were both trained on data from EMPIAR, so if the noise in your micrographs is very different it might also not work well.

Do you have single-frame micrographs because you decided to record this way, or because your detector can't record movies? If you have the possibility to record movies, doing so and training your own model with topaz (or another noise2noise implementation like JANNI) is probably the most straightforward solution to your problem. Noise2noise works very well, and it might be easier to obtain a good training set for this approach than to find a different denoising method that you hope will perform well on your micrographs without fine-tuning or retraining.

[DennisDannecker]

Hey Guillawme,

thank you for your quick reply and sorry for my late response.

Indeed, I have already tested JANNI, while it works quite good on conventional micrographs, it performs poorly on my micrographs. Your comments are very helpful, as a matter of fact, we can record movies and have already done so. Training with these movies would have been my next step. Now I have actually already trained JANNI on these movies, however, the resulting model performed very weakly. Validation metrics also indicated rather poor performance.

Now, I wanted to try training the denoiser that comes with topaz using the same movies to see whether it would create a model of higher denoising capability. I worked through the tutorial notebook “04_denoising.ipynb” using first the provided tutorial dataset.

While working first with the tutorial dataset I was able to prepare training data from the tutorial dataset and then effectively denoise the micrographs. (I noticed that while downloading and unpacking the tar file no micrograph folder is created and no single micrographs are downloaded for later denoising. However, the notebook assumes the presence of such a directory that holds several micrographs for testing the just trained model. Either I missed something or there is something missing in the tutorial dataset downloaded.)

When I then moved on to train a model on my dataset I encountered an error thrown during running the notebook. I was able to prepare the training data by splitting each movie in even and odd frames and summing them up to create a pair of micrographs for each movie. (Just like for JANNI) But then when I moved on to the next cells for actually training the model on the created training data I quickly run into the following error:

using device=0 with cuda=True

training with 0 image pairs

validating on 0 image pairs

epoch loss_train loss_val Traceback (most recent call last): File "/programs/x86_64-linux/topaz/0.2.5_cu11/bin/topaz", line 11, in load_entry_point('topaz-em==0.2.4', 'console_scripts', 'topaz')() File "/programs/x86_64-linux/topaz/0.2.5_cu11/topaz_extlib/miniconda3-4.7.12.1-j4hb/lib/python3.7/site-packages/topaz/main.py", line 148, in main args.func(args) File "/programs/x86_64-linux/topaz/0.2.5_cu11/topaz_extlib/miniconda3-4.7.12.1-j4hb/lib/python3.7/site-packages/topaz/commands/denoise.py", line 435, in main for epoch,loss_train,loss_val in iterator: File "/programs/x86_64-linux/topaz/0.2.5_cu11/topaz_extlib/miniconda3-4.7.12.1-j4hb/lib/python3.7/site-packages/topaz/denoise.py", line 1137, in train_noise2noise , num_workers=num_workers) File "/programs/x86_64-linux/topaz/0.2.5_cu11/topaz_extlib/miniconda3-4.7.12.1-j4hb/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 262, in init sampler = RandomSampler(dataset, generator=generator) # type: ignore File "/programs/x86_64-linux/topaz/0.2.5_cu11/topaz_extlib/miniconda3-4.7.12.1-j4hb/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 104, in init "value, but got num_samples={}".format(self.num_samples)) ValueError: num_samples should be a positive integer value, but got num_samples=0

Apparently it does not start training on any image pairs. Openly speaking, I have not yet started to further investigate the error. Are you aware of an error of this kind. I suspect of course my input data to be the cause of this error. I will provide you with more information in a subsequent post and more specifically if you require it.

Thank you very much for your help.

Dennis

[tbepler]

This sounds like an issue with how your data is formatted. Do your even/odd frame micrographs have the same names in their respective directories? That could be the issue.

[tbepler]

Closing this issue since there hasn't been any more discussion.