Conclusion about long-range dependency seems not true

[iumyx2612]

Authors stated in the paper that: "Contrary to popular belief, the long-range dependency hinders NN optimization.". However, recent models that adopts long-range dependency achieves really great results like: VAN, ConvNeXt or RepLKNet

Therefore, the statement I mentioned above seems a little bit wrong? I know there's an issue that discuss about large kernel Conv, however, the issue did not mention the statement above.

Moreover, the Experiments in Fig 7, you use Convolutional SANs. This model has 2 variants: 1D-CSANs and 2D-CSANs. The one you are doing experiments on is 2D-CSANs right? It not only consider the interaction among tokens in a single, but also consider the interaction among different heads. The "long-range dependency" is still very beneficial in the 1D-CSANs (Fig below), which typically, is what I consider the true "long-range dependency" in Self-attention.

When using 2D-CSANs, it considers both aspects: interaction among heads, and tokens, which brings negative performance when scaling up window sizes. The results is align with Convolutional SANs paper.

However, I don't consider 2D-CSANs negative performance when scaling up window sizes is: "long-range dependency hinders NN optimization" since it consider 2 aspects in the model. Sorry for writing this long, if you don't understand any parts in my question, I can clarify it for you

[xxxnell]

Hi @iumyx2612,

To explain the statement, we first clarify the terminologies. If an inductive bias (e.g., locality) improves the performance of NNs, this is either a method to help the NNs learn “strong representations”, or a method to “regularize” it. An improved training NLL (i.e., lower training NLL) suggests that this bias helps the “optimization” and the NN learn strong representations. Conversely, a compromised training NLL indicates that the bias or technique regularizes the NN. Improved accuracy does not always implies that improved optimization.

Fig. 7a shows that locality improves (i.e., reduces) the training NLL by comparing the 5×5 kernel and the 8×8 kernel. This implies that long-range dependencies hinder optimization and locality helps to learn even strong representations in this setting (in this context, we can paraphrase "the long-range dependency hinders NN optimization" as "the locality helps to learn strong representations” — sorry for the confusion). This observation is consistent with the observation in Fig. 5. Stronger locality compromises the training NLL but improves accuracy, which implies that stronger locality constraints regularize NNs. But too strong regularization may degrade the accuracy (even though Fig. 7 does not show this effect).

In some settings, moderate (neither global nor local) long-range dependencies can help improve performance; I believe this is not contrary to the statement. In practice, after our paper was published, several papers introduce longer-range dependencies into CNNs to improve the accuracy. The 3×3 kernel of conventional CNNs may be too local.

[iumyx2612]

Hi @iumyx2612,

To explain the statement, we first clarify the terminologies. If an inductive bias (e.g., locality) improves the performance of NNs, this is either a method to help the NNs learn “strong representations”, or a method to “regularize” it. An improved training NLL (i.e., lower training NLL) suggests that this bias helps the “optimization” and the NN learn strong representations. Conversely, a compromised training NLL indicates that the bias or technique regularizes the NN. Improved accuracy does not always implies that improved optimization.

Fig. 7a shows that locality improves (i.e., reduces) the training NLL by comparing the 5×5 kernel and the 8×8 kernel. This implies that long-range dependencies hinder optimization and locality helps to learn even strong representations in this setting (in this context, we can paraphrase "the long-range dependency hinders NN optimization" as "the locality helps to learn strong representations” — sorry for the confusion). This observation is consistent with the observation in Fig. 5. Stronger locality compromises the training NLL but improves accuracy, which implies that stronger locality constraints regularize NNs. But too strong regularization may degrade the accuracy (even though Fig. 7 does not show this effect).

In some settings, moderate (neither global nor local) long-range dependencies can help improve performance; I believe this is not contrary to the statement. In practice, after our paper was published, several papers introduce longer-range dependencies into CNNs to improve the accuracy. The 3×3 kernel of conventional CNNs may be too local.

Sorry I mistook your statement. I thought Long-range dependency degrades the performance of NN (lower accuracy).

However, as I mentioned above the locality experiment using Convolutional SANs, maybe the higher training NLL, and lower the accuracy is because of including too many attention heads? Because authors of ConvSANs stated that: However, when the number of heads in attention goes up, the translation quality inversely drops. One possible reason is that the model still has the flexibility of learning a different distribution for each head with few interactions, while a large amount of interactions assumes more heads make “similar contributions”(page 4). Since I consider the long-range dependency we're talking about is only the range of tokens to attend in one attention head, I do not consider long-range dependency is including multiple attention heads

[xxxnell]

Hi @iumyx2612,

Thank you for the further explanation. But my two-dimensional convolutional self-attentions used in Fig 7 does not consider the interaction between different heads, so they are correspond to the one-dimentional self-attentions in “Convolutional SANs”. I should have explained this too. I agree that the description of my paper can be quite confusing — sorry for the confusion and thank you for the opportunity to clarify this point.

[iumyx2612]

Hi @iumyx2612,

Thank you for the further explanation. But my two-dimensional convolutional self-attentions used in Fig 7 does not consider the interaction between different heads, so they are correspond to the one-dimentional self-attentions in “Convolutional SANs”. I should have explained this too. I agree that the description of my paper can be quite confusing — sorry for the confusion and thank you for the opportunity to clarify this point.

Thank you for the clarification. So after this issues I can safely say that:

1. Long-range dependency makes the model harder to optimize, but it does not decrease the performance of fully-trained model on a validation set.
2. Long-range dependency has a regularization effect (training NLL is higher).
3. Med-Locality, normal-range dependency makes the model easier to optimize, also have a regularization effect which, in the end, increase the performance of fully-trained model on validation set.
4. Too local may weaker the model representation (higher training NLL compared to 5x5 Conv), however have a strong regularization effect (lower test error than 5x5 Conv)

Another question: if 4 is correct, then can you explain the statement "Although the test errors of 3 × 3 and 5 × 5 kernels are comparable, the robustness of 5 × 5 kernel is significantly better than that of 3 × 3 kernel on CIFAR-100-C"

[xxxnell]

Hi @iumyx2612,

Thank you for the great follow-up question. I believe the observation you mention implies that stronger representations lead to stronger generalizations. However, this observation is underexplored and requires further investigation. In addition, I would like to point out that too strong (e.g. global) long-range dependencies degrade both the optimization and the performance of ViTs, so (moderate) locality constraints are often important to improve not only the performance but also the optimization.

[iumyx2612]

Hi @iumyx2612,

Thank you for the great follow-up question. I believe the observation you mention implies that stronger representations lead to stronger generalizations. However, this observation is underexplored and requires further investigation. In addition, I would like to point out that too strong (e.g. global) long-range dependencies degrade both the optimization and the performance of ViTs, so (moderate) locality constraints are often important to improve not only the performance but also the optimization.

Thank you, now I fully understand!