Closed rikonaka closed 1 year ago
First, the random start of PGDL2 is followed by some previous work (but I cannot remember the source code. Sorry). To my knowledge, there is no standard method to initialize the random noise at the beginning. I think there are several works to investigate the importance of random noise.
For the L2 norm, mathematically, the $i$-th component of delta can be formalized as follows: $\delta _i \times \min (\epsilon/|\delta|_2, 1)$ Thus, the norm of normalized delta is $\sum [ \delta_i \times \min (\epsilon/|\delta|_2, 1)]^2 = \times \min (\epsilon/|\delta|_2, 1) \sum \delta_i^2 = \min (\epsilon, |\delta|_2)$.
I hope this answer solves your questions! 👍
First, the random start of PGDL2 is followed by some previous work (but I cannot remember the source code. Sorry). To my knowledge, there is no standard method to initialize the random noise at the beginning. I think there are several works to investigate the importance of random noise.
For the L2 norm, mathematically, the i-th component of delta can be formalized as follows: δi×min(ϵ/|δ|2,1) Thus, the norm of normalized delta is ∑[δi×min(ϵ/|δ|2,1)]2=×min(ϵ/|δ|2,1)∑δi2=min(ϵ,|δ|2).
I hope this answer solves your questions! +1
Thank you very much for your answer, I probably understand it now. 😜
❔ Any questions
First, why PGDL2 random_start like that? Is such a restriction necessary?
Different from PGD.![PGD](https://user-images.githubusercontent.com/13602602/223745263-4789f1a0-bf2e-4bd1-bd43-7dade6ba7d19.png)
Second, how does the code limit L2 to eps? If we wanna limi L2 to eps, here is some equation.![eps](https://user-images.githubusercontent.com/13602602/223758885-2e6ceb53-adea-49fb-b130-d726c0d1bed8.png)
So we only need to restrict the size of each delta value to that value and it will ensure that the final L2 is less than eps. But I don't see any similar restriction in the code.
HM-GM-AM-QM_inequalities
In the code.![code limit](https://user-images.githubusercontent.com/13602602/223757551-ef1cd743-e150-4bb7-88a5-da8d5fe63e94.png)
I think there are some errors here, am I correct?