DiffModel Experminet 1 - UNet - Gradient Descent - L2 - sklearn dataset

[mbaddar1]

apply the DDPM code to sklearn swissroll, conc circles, and half moons dataset. find an objective eval measure

[mbaddar1]

Make the CustomeDataSet class support mnist and sklearn datasets https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/ddpm_sandbox/ddpm_trainer.py#L44

[mbaddar1]

Make a separate test script for datasets https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/ddpm_sandbox/test/test_custom_dataset.py

[mbaddar1]

Next step: at this line https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/ddpm_sandbox/test/test_custom_dataset.py#L16 add plotting code for generated samples from mnist and sklearn datasets

[mbaddar1]

Finished adding sklearn datasets Commit https://github.com/mbaddar1/denoising-diffusion-pytorch/pull/4/commits/6b5923e0397fa8527f7ea66a6d40bc219b90060f Next : Add documentation

[mbaddar1]

Finished test script for CustomDataSet class that supports mnist (0 and 8) and sklearn datasets (circles,blobs, moons, swissroll2D) commit https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/dc64e4bc70e971cd8e6eec751d3b9f93950c2c67
nextstep : use this Class with main trainer code

[mbaddar1]

Tested the DDPM-Trainer class with the new CustomDataSet class with mnist-8 dataset https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/e8b492e2abe4bfa19eeef6176252030b52da76e5 see this line for the main change https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/e8b492e2abe4bfa19eeef6176252030b52da76e5#diff-e5ac8646018f44a9eed5a912470b1b4901de801c7392fcc110e411bd24690aacR384 and the result images are good enough https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/e8b492e2abe4bfa19eeef6176252030b52da76e5#diff-38ff16bdf60737cf9d741e93309426a60ab819b36a8b410774287d4328eaa1f5

[mbaddar1]

Two commits for testing the DDPM-trainer with mnist0 and mnist8 dataset batches mnist0 https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/ac15733566d7d89d18810311a421760966be38aa mnist8 https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/88ad456b7e796bb9a776813c2a769ba07ea9596b

[mbaddar1]

Trying to handle the image_size property in the diffusion model with sklearn datasets Currently this the property setting code https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L536 and we want to eliminate or modify it to adapt with sklearn datasets Questions

1. How the dim property in Unet model is linked to diffusion model image size

https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L305

https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L515

2. How can image_size be handled with sklearn dataset , should it be adapted or set to None
3. Can the classes in the file https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/denoising_diffusion_pytorch_1d.py be helpful

and this commit starts to handle the issue https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/d4ebfd43a400e5ff7bf56b9ee7e88b9b8f912fc3

currently if sklearn dataset name is passes like circles https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/d4ebfd43a400e5ff7bf56b9ee7e88b9b8f912fc3#diff-e5ac8646018f44a9eed5a912470b1b4901de801c7392fcc110e411bd24690aacR373 the code fails (intentionally) as follows

DEBUG:root:For quantiles at levels [0.0, 0.25, 0.5, 0.75, 1.0] = tensor([-1.1513e+00, -2.8661e-01, -9.2925e-04, 2.9009e-01, 1.1572e+00], dtype=torch.float64) Traceback (most recent call last): File "/home/mbaddar/Documents/mbaddar/phd/genmodel/denoising-diffusion-pytorch/denoising_diffusion_pytorch/ddpm_sandbox/ddpm_trainer.py", line 427, in trainer = DDPmTrainer(diffusion_model=diffusion, batch_size=batch_size, dataset=dataset, debug_flag=True, File "/home/mbaddar/Documents/mbaddar/phd/genmodel/denoising-diffusion-pytorch/denoising_diffusion_pytorch/ddpm_sandbox/ddpm_trainer.py", line 172, in init self.__validate_dataset() # FIXME remove later File "/home/mbaddar/Documents/mbaddar/phd/genmodel/denoising-diffusion-pytorch/denoising_diffusion_pytorch/ddpm_sandbox/ddpm_trainer.py", line 282, in __validate_dataset assert self.diffusion_model.image_size is None, (" For sklearn datasets , image_size property in " AssertionError: For sklearn datasets , image_size property in diffusion model is useless and must be None

[mbaddar1]

Next step : try to understand the Unet , Gaussian diffusion and how the image_size in diffmodel and dim in Unet works together , if they do

[mbaddar1]

Currently Trying to understand the data-flow in UNet https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/main/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L419

[mbaddar1]

Create a Generic UNet class that will be a blueprint to a UNet class to support mnist , sklearn and other data-set https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/fea95bf99181d3f5657e78859e99fac125dac327 https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/ddpm_sandbox/ddpm_trainer.py#L399

Also we did an experiment with this UNET generic class and mnist8 dataset, and results were good visually https://github.com/mbaddar1/denoising-diffusion-pytorch/tree/17-ddpm-sklearn-dataset/denoising_diffusion_pytorch/ddpm_sandbox/generated_images

[mbaddar1]

Create a simple FuncApprox class instead of Unet https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/b8286fd165b561dd32c0b7d91837d42acf89542a#diff-1ef0846017d5c27bd277b66e5afcab8798667afd39675e6f9382fc7e87a1fe28R301 and modified GaussianDiffusionClass https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/b8286fd165b561dd32c0b7d91837d42acf89542a#diff-1ef0846017d5c27bd277b66e5afcab8798667afd39675e6f9382fc7e87a1fe28R698 to handle sklearn and image dataset

Next steps

1. Backward Test the modified code against images data
2. Make it work with sklearn dataset : next to work on the function p_losses https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/b8286fd165b561dd32c0b7d91837d42acf89542a#diff-1ef0846017d5c27bd277b66e5afcab8798667afd39675e6f9382fc7e87a1fe28R1067

[mbaddar1]

Modified GaussianDiffusionModel class to handle image and non-image/flat datasets . However, this commit handles only testing the class over Mnist8 https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/3a649a09d2b82f4358f5a7d6a6eb66dd0b4f728f#diff-0522ac749ca27fb998354c5f796fd7dc8aab7f475aa22d59f1a828ec2dfa57d8 also the generated images are good https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/3a649a09d2b82f4358f5a7d6a6eb66dd0b4f728f#diff-0522ac749ca27fb998354c5f796fd7dc8aab7f475aa22d59f1a828ec2dfa57d8

[mbaddar1]

Now simplified the CustomDataset class to be ImageDataset only. The main reason is that batch-size has mixed semantics and when sklearn data generation code is embedded in get_item method for CustomDataset, another layer of batching is in the dataloader which make the complete process complex. So reverted back to simple ImageDataset setup and when using skelarn dataset is used, directly a batch is generated in the training loop https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/550f1de4521961d20c5af250bc35c76e96251681

The setup is tested against mnist8 dataset

[mbaddar1]

Trial 1 with a time-variant Step Model as the one in https://papers-100-lines.medium.com/diffusion-models-from-scratch-tutorial-in-100-lines-of-pytorch-code-5dac9f472f1c https://github.com/MaximeVandegar/Papers-in-100-Lines-of-Code/blob/main/Deep_Unsupervised_Learning_using_Nonequilibrium_Thermodynamics/diffusion_models.py#L15 which has a fixed head and a time variant / time-indexed tail the trial commit is here https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/218a7ed589659d26715db6e6f9d6b00ad0c2cde5

Why it is a failure : Sinkhorn didnot converge the result visual is bad

[mbaddar1]

Trial 2 with a fixed head and time-variant tail for step-modeling in DDPM for sklearn circles https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/0682cc20afa3ac2ff330fe49bc639a520780db68#diff-1ef0846017d5c27bd277b66e5afcab8798667afd39675e6f9382fc7e87a1fe28 results are bad . Trial failed

[mbaddar1]

https://arxiv.org/pdf/2006.11239.pdf Revise Sec 2 and 3 : Understand all loss and pred models and map to code part denoising_diffusion_pytorch.denoising_diffusion_pytorch.GaussianDiffusion.p_losses_images and test the three objectives : noise, x_start and _v denoising-diffusion-pytorch/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py:1016

[mbaddar1]

To conclude for this branch We have a working version for DDPM Unet2d MNIST8. Loss curve and Sinkhorn are good. Generated images are good https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/983f860a9b26f49dee3448ff10ac457d8b25b448

We have a non-working version for DDPM HeatTail Circles. Loss and Sinkhorn are bad. Generated images are bad https://github.com/mbaddar1/denoising-diffusion-pytorch/commit/7bf9ef816751fede1a80f217db844d083a2879ef

[mbaddar1]

Based this DDPM repo https://github.com/Jmkernes/Diffusion I have forked and modified it to a working DDPM repo https://github.com/mbaddar1/Diffusion

Look at this commit https://github.com/mbaddar1/Diffusion/commit/281e453d66d413976bc069c75d736c6df3c4a9de

I have experimented the author (JMKernes) BasicDiscreteTime Noise Model vs naive nn model the gifs diffusion/ddpm/animation_noise_model_basic_discrete_time.gif diffusion/ddpm/animation_noise_model_naive_nn.gif

Now see the iter losses and sinkhorn curves . Note that I use log-scale the Iter-loss diffusion/ddpm/iter_loss_noise_model_basic_discrete_time.png diffusion/ddpm/iter_loss_noise_model_naive_nn.png

The iter-sinkhorn diffusion/ddpm/iter_sinkhorn_noise_model_basic_discrete_time.png diffusion/ddpm/iter_sinkhorn_noise_model_naive_nn.png

The code base for this forked repo can be a good start to test TT based noise models