Using the given checkpoints, but I get very different test results from the original paper.

[MmelodYy]

Dear Author,

I'm very sorry to come and bother you.

When I was testing the code, I found that the inference results on my computer were quite different from the original text. So I would like to ask if the dataset provided by the code repository has been updated and is not the original dataset? My specific test results are as follows:

[UeFan]

Hi,

Thanks for your question.

the data is not changed since it is released. However, the code was implemented in early 2022 and was less mature in terms of the coding style. I did noticed some precision problems causing the variation of results with different environment settings. Broadly speaking, there are several factors that could cause slight variations in inference results when using different versions of PyTorch:

• Floating-point Arithmetic Differences: Different versions of PyTorch may have slight changes in the underlying floating-point arithmetic due to changes in the libraries or optimizations used (e.g., cuBLAS, cuDNN, MKL). These changes can cause small variations in results, especially with models that rely heavily on precision, like transformers.
• Different Kernel Implementations: PyTorch frequently updates kernel implementations for GPU-accelerated operations to improve performance or stability. These changes can cause differences in floating-point computations.
• and more.

I plan to update the code base and reimplement the HAA-Transformer with some newer coding style. Please stay tuned. For now, I'd suggest to use the torch=1.11+cu113 as the result seems close enough to the numbers from the paper.