implement epitope similarity regularization with absolute value approximation

[timcyu]

This PR implements the epitope similarity regularization using @zorian15's suggested absolute value approximation described in #124. Unfortunately, TLDR: unless I did something wrong, it's not working better.

The new equation for the regularization and its corresponding gradient are in optimization.ipynb.

There's a test for the gradient in test_gradients.ipynb. Like before, there's some difference between the numerical and analytical gradients. This happens only when the analytical gradient is 0.0. For this reason, I've excluded two of the models from being evaluated. The (non-zero) gradients for the remaining two match up very well, so I believe the analytical gradient is implemented correctly and the numerical gradient is just behaving weird.

Lastly, I've included the same notebook that attempts to deconvolve Frances' H3 HA mAb cocktail DMS data. I manually tried many different weights for the regularization but was unable to cleanly separate the two epitopes like before (see top of #128).

@jbloom let me know what you think! We can discuss tomorrow during our meeting.

[jbloom]

Nice job on this, @timcyu. But can you first try it out again implementing a different and larger value of $\epsilon$. Right now you are setting it to numpy.nextafter(0, 1), which on my computer is 5e-324.

Although I think you want a small number, I don't think you want one that small. It's unclear to me whether the optimization will work with such a small epsilon since I'm not even sure the calculated analytical gradient will be accurate given machine rounding.

Can you try with epsilon as a settable parameter that by default should maybe be something more like 1e-6 or 1e-4 (maybe try a few values) and see if that both makes the regularization work better and fixes the problem with the gradients.

[jbloom]

(Also, maybe @WSDeWitt or @matsen would have good intuition on what is an appropriate value of $\epsilon$ for the differentiable absolute value approximation.)

[matsen]

Recall that the issue is that we will have a 1/sqrt(c) term in the gradient if both of the betas are zero, and with the current setup we will have terms about 10^162 in the gradient. That does seem pretty big compared to what I think other components of the gradient will be.

So, basically, I agree with Jesse.

[timcyu]

Thanks all! Changing the epsilon helps. I was able to deconvolve the epitopes (though had to play around with the parameters quite a bit).

[jbloom]

@timcyu, should we close this pull request for now but keep the branch non-deleted? And maybe close #124 too.

[timcyu]

@jbloom yes that sounds good.

[jbloom]

Closed as #135 and #128 implement something similar.