gaudetcj
commented
3 years ago Sorry the code is very behind the paper that is up on arxiv and in fact even that paper itself is very outdated.
There is some pretty significant updates to the way the L parameter works in the current version.
We are submitting to conferences and are waiting to update everything if accepted.
Thanks, Chase
On Mon, Nov 30, 2020, 4:08 AM Bharath notifications@github.com wrote:
I have tried to reproduce the results of the CIFAR-10 classification task described in your paper. For your convenience, please find here https://github.com/UpStride/VectorMapConvolution the dockerfile we used to run the code.
In the table below, I report the final and the minimum validation error that I obtain when running your original code to reproduce some of the experiments of Table 1 in the paper: Classification CIFAR-10 ours - final error(epoch 200) ours - min error reported in the paper ResNet-18 Real 9.16 7.60 (epoch 169) 5.92 ResNet-18 Quaternion 9.90 7.06 (epoch 172) 5.92 ResNet-18 Vector Map 9.13 6.69 (epoch 174) 6.05 ResNet-34 Real 7.99 6.44 (epoch 174) 5.73 ResNet-34 Quaternion 9.27 7.03 (epoch 174) 5.73 ResNet-34 Vector Map 8.34 6.26 (epoch 174) 5.55
As you can notice from the table, I get higher errors than those reported in the paper. In addition, the performance at the last epoch is far from the best one achieved during the training, which seems to occur around epoch 170.
I would really appreciate any input on your side that could guide me in the right direction to match the results of the paper. More concretely:
1.
I am currently running the experiments with the default hyper-parameters defined in the code, since this information is not specified in the paper for the classification task. Could you please provide the hyper-parameters you used to obtain your results? 2.
What evaluation strategy have you used to produce the results of Table 1? Have you reported the error of the last epoch or the minimum error during the training? Have you reported the best or the mean/median result out of N runs for a single experiment?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/gaudetcj/VectorMapConvolution/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABV6VVDKEHI75UVUVNRDWQLSSNVKFANCNFSM4UHLA6EQ .
asbharath
Sahel13
I have tried to reproduce the results of the CIFAR-10 classification task described in your paper. For your convenience, please find here the dockerfile we used to run the code.
In the table below, I report the final and the minimum validation error that I obtain when running your original code to reproduce some of the experiments of Table 1 in the paper:
As you can notice from the table, I get higher errors than those reported in the paper. In addition, the performance at the last epoch is far from the best one achieved during the training, which seems to occur around epoch 170.
I would really appreciate any input on your side that could guide me in the right direction to match the results of the paper. More concretely:
I am currently running the experiments with the default hyper-parameters defined in the code, since this information is not specified in the paper for the classification task. Could you please provide the hyper-parameters you used to obtain your results?
What evaluation strategy have you used to produce the results of Table 1? Have you reported the error of the last epoch or the minimum error during the training? Have you reported the best or the mean/median result out of N runs for a single experiment?