Hi, first of all thank you for your excellent work
I found that freelb uses a lot of detach operations, which will detach delta.grad, delta_norm, and delta. I know that detach can separate variables from the current graph and can be used to prevent the gradient from propagating further, then If want to prevent the calculation of the gradient for the parameters before the final delta is used to calculate the final delta, it feels that a step of detach is enough here on line 290. I am not sure if this is correct.（https://github.com/zhuchen03/FreeLB/blob/master/huggingface-transformers/examples/run_glue_freelb.py#L290）
In addition, regarding denorm and delta_norm, is there any difference between them, why delta_norm does a detach operation, but denorm does not. And I found that many researchers will detach delta.grad. Is there anything special about grad?
So I want to know what your purpose is in the following five detaches, what role they play, and whether some detach operations can be deleted.
Hi, first of all thank you for your excellent work
I found that freelb uses a lot of detach operations, which will detach delta.grad, delta_norm, and delta. I know that detach can separate variables from the current graph and can be used to prevent the gradient from propagating further, then If want to prevent the calculation of the gradient for the parameters before the final delta is used to calculate the final delta, it feels that a step of detach is enough here on line 290. I am not sure if this is correct.（https://github.com/zhuchen03/FreeLB/blob/master/huggingface-transformers/examples/run_glue_freelb.py#L290） In addition, regarding denorm and delta_norm, is there any difference between them, why delta_norm does a detach operation, but denorm does not. And I found that many researchers will detach delta.grad. Is there anything special about grad?
So I want to know what your purpose is in the following five detaches, what role they play, and whether some detach operations can be deleted.
Related code： https://github.com/zhuchen03/FreeLB/blob/master/huggingface-transformers/examples/run_glue_freelb.py#L239 https://github.com/zhuchen03/FreeLB/blob/master/huggingface-transformers/examples/run_glue_freelb.py#L278 https://github.com/zhuchen03/FreeLB/blob/master/huggingface-transformers/examples/run_glue_freelb.py#L284 https://github.com/zhuchen03/FreeLB/blob/master/huggingface-transformers/examples/run_glue_freelb.py#L286 https://github.com/zhuchen03/FreeLB/blob/master/huggingface-transformers/examples/run_glue_freelb.py#L290