search

IDKiro / DehazeFormer

[IEEE TIP] Vision Transformers for Single Image Dehazing
MIT License
357 stars 34 forks source link

about Nonlinear activation functions #16

Closed Merenguelkl closed 1 year ago

Merenguelkl commented 1 year ago

Hi, thanks for your amazing work.

I'm very interested in your study of 'Nonlinear activation functions'.

Paper says changing GELU to ReLU is effective in dehazing. Does it performs well on other low-level tasks, e.g. denoising and debluring?

IDKiro commented 1 year ago

Some points summarized from my experiments, which are not always correct:

  1. For Vision Transformer, ReLU is also better than GELU in tasks such as denoising.
  2. For CNN, the performance is close, and GELU performs better in image deblurring.

I think there is a trade-off here between the network's capability to extract semantics and details, with GELU being better at extracting semantics and ReLU being better at filtering details, so different networks behave differently.

Merenguelkl commented 1 year ago

Thanks for your reply! I will try it in my experiments :)