search

shizhouxing / Fast-Certified-Robust-Training

[NeurIPS 2021] Fast Certified Robust Training with Short Warmup
https://arxiv.org/abs/2103.17268
BSD 3-Clause "New" or "Revised" License
23 stars 2 forks source link

Other distributions for IBP init #5

Closed nurlanov-zh closed 1 year ago

nurlanov-zh commented 2 years ago

Hi,

Have you tried other distributions for IBP init? Does it make difference? Since the IBP training has non-continuity at W_i = 0, would it make sense to initialize it with Laplace distribution W_i ~ Laplace(0, b)? Then |W_i| ~ Exponential(b^{-1}), so b = 2/n_i would also work.

Best regards, Zhakshylyk

shizhouxing commented 2 years ago

Hi Zhakshylyk,

I think we just assumed normal distribution. Sounds interesting to consider other distributions. Could you please explain a bit why making the distribution of W_i non-continuous at 0 would potentially help the training?

Thanks! Zhouxing