search

IDKiro / DehazeFormer

[IEEE TIP] Vision Transformers for Single Image Dehazing
MIT License
377 stars 36 forks source link

about Nonlinear activation functions #16

Closed Merenguelkl closed 2 years ago

Merenguelkl commented 2 years ago

Hi, thanks for your amazing work.

I'm very interested in your study of 'Nonlinear activation functions'.

Paper says changing GELU to ReLU is effective in dehazing. Does it performs well on other low-level tasks, e.g. denoising and debluring?

IDKiro commented 2 years ago

Some points summarized from my experiments, which are not always correct:

  1. For Vision Transformer, ReLU is also better than GELU in tasks such as denoising.
  2. For CNN, the performance is close, and GELU performs better in image deblurring.

I think there is a trade-off here between the network's capability to extract semantics and details, with GELU being better at extracting semantics and ReLU being better at filtering details, so different networks behave differently.

Merenguelkl commented 2 years ago

Thanks for your reply! I will try it in my experiments :)