TensorFlow Implementation

[utsav-akhaury]

Code - https://github.com/utsav-akhaury/ADMM/blob/main/ADMM_tf.ipynb

• [x] Convert all NumPy arrays to Tensors
• [x] Use TensorFlow functions for implementation
• [x] Solve dimension & datatype inconsistencies
• [x] Compare final results with the NumPy implementation
• [x] Check the precision with tensor dtype 'complex128'
• [x] Downgrade from TensorFlow 2.2 to 1.15
• [x] Upsample images before convolution to avoid aliasing
• [x] Tune the ADMM hyperparameters for better convergence

[utsav-akhaury]

ADMM in TensorFlow vs. ADMM in NumPy -

• TensorFlow implementation significantly slower
• With an increase in the no. of iterations, the NumPy version diverges earlier (~400 iterations) than the TensorFlow one (~3000 iterations) [early stopping required]
• At 2987 iterations, TensorFlow NMSE ~5.7% (as compared to a min. NMSE of ~8.3% in NumPy at 396 iterations)

[utsav-akhaury]

Update -

• Running TF in eager mode speeds up the iteration drastically (although it's still slightly slower than NumPy on CPU)

• I replaced all denoising & update steps by native TF functions to further optimize the code (instead of accessing the value of the tensor and performing operations on those arrays) [Eg - cv.GaussianBlur() -> tf.nn.conv2d]

• I compared the NumPy and TF implementations function by function by giving random inputs and comparing the outputs in this notebook. The order of difference for all functions is very small (between 10^-10 & 10^-15)

• All images were downscaled to speed up the code: 96 X 96 -> 64 X 64

• The NumPy output oscillates and does not converge

• The TF output also diverges and after 2000 iterations, the pixel values are not defined (NaN)

• After upsampling by a factor of 4 to remove aliasing, the convergence is steeper (i.e. optimal reconstruction is achieved faster). But at the same time, the NumPy version diverges earlier (~300 iterations), while the TF version remains comparatively stable (until ~550 iterations)

• By updating alpha (the gradient descent step size) after every iteration (as implemented here), the optimal solution is reached after a higher no. of iterations

[utsav-akhaury]

Convergence achieved by updating the Gradient step size update (α) as given in Issue #3 Both NumPy and TensorFlow give similar results (order of difference ~10^-4)