reproduce the results

[xingchenzhang]

Hi,

I am wondering do you have any idea that the running environment will affect the results?

In last week I can train the model and obtain similar results. But in the weekend I installed something (to run another code), now my performance becomes much worse. I am using the same code but I can't reproduce my results in last week.

This is very wield. I am wondering have you encountered this kind of issue, and have any suggestions to solve it?

Best wishes, Xingchen

[abduallahmohamed]

Hi,

I had a similar situation with two machines, one had gtx1080ti installed and one with titanV installed. The gtx1080ti produced better results than the titanV one. Not sure why! I noticed overall even with the same code different machines with different versions of software stack produce different results. How far is your difference?

[xingchenzhang]

Hi,

I had a similar situation with two machines, one had gtx1080ti installed and one with titanV installed. The gtx1080ti produced better results than the titanV one. Not sure why! I noticed overall even with the same code different machines with different versions of software stack produce different results. How far is your difference?

Hi,

Thank you very much for your reply.

Generally, for ADE, the difference is around 3%, while for FDE the difference is more than 5%. The max difference I got for FDE was 9%! For instance, last week I got an FDE 0.69, but with the same code it is 0.78 now...

That's too strange and I don't why this happen...I am not sure what I have installed over the weekend and why these affect the results so much. I have tried to setup several env with different version of python and torch, but still I cannot reproduce the one with FDE 0.69...

[xingchenzhang]

By the way, using your code (the exact CVPR version), now after training with my machine, the ADE is 0.466 while FDE is 0.802, which are much worse than you reported in the paper. Previously I can obtain better results, but now I cannot reproduce them...

[abduallahmohamed]

I think it's highly related to the environment + random seeds maybe? The exact CVPR version have the weights which the results were reported which can be helpful. This is the requirements file at the time of CVPR https://github.com/abduallahmohamed/Social-STGCNN/blob/master/checkpoint/requirements.txt I just uploaded it, it might be helpful to you.

[xingchenzhang]

I think it's highly related to the environment + random seeds maybe? The exact CVPR version have the weights which the results were reported which can be helpful. This is the requirements file at the time of CVPR https://github.com/abduallahmohamed/Social-STGCNN/blob/master/checkpoint/requirements.txt I just uploaded it, it might be helpful to you.

Hi,

Thank you for your file. To be honest I don't know the reason. Possibly it is related to the environment. I will look into your requirements file maybe I can find some differences.

[xingchenzhang]

I think it's highly related to the environment + random seeds maybe? The exact CVPR version have the weights which the results were reported which can be helpful. This is the requirements file at the time of CVPR https://github.com/abduallahmohamed/Social-STGCNN/blob/master/checkpoint/requirements.txt I just uploaded it, it might be helpful to you.

Hi,

I found that when I run train.sh, the data processing stage showed this. Is this normal? I mean some progress bar wer not 100%. When you trained your model, how was this shown?

Thanks!

8%|████▊ | 165/2076 [00:04<00:54, 34.93it/s]Processing Data ..... 9%|█████▊ | 197/2076 [00:05<00:59, 31.53it/s]Processing Data ..... 0%|▏ | 6/2112 [00:01<06:53, 5.10it/s]Processing Data ..... 0%| | 3/2322 [00:00<06:03, 6.38it/s]Processing Data ..... 100%|████████████████████████████████████████████████████████████| 2076/2076 [01:19<00:00, 26.10it/s] 14%|████████▋ | 300/2112 [01:16<09:20, 3.23it/s]Processing Data ..... 100%|██████████████████████████████████████████████████████████████| 530/530 [00:26<00:00, 19.77it/s] 27%|████████████████▎ | 621/2322 [01:43<01:13, 23.15it/s] Data and model loaded

[abduallahmohamed]

Hi,

This is normal it’s a progress bar issue when the screen is tight. Also you get multiple progress bars(5) per each training session.

On Wed, Oct 21, 2020 at 4:20 PM Xingchen Zhang notifications@github.com wrote:

I think it's highly related to the environment + random seeds maybe? The exact CVPR version have the weights which the results were reported which can be helpful. This is the requirements file at the time of CVPR https://github.com/abduallahmohamed/Social-STGCNN/blob/master/checkpoint/requirements.txt I just uploaded it, it might be helpful to you.

Hi,

I found that when I run train.sh, the data processing stage showed this. Is this normal? I mean some progress bar wer not 100%. When you trained your model, how was this shown?

Thanks!

8%|████▊ | 165/2076 [00:04<00:54, 34.93it/s]Processing Data ..... 9%|█████▊ | 197/2076 [00:05<00:59, 31.53it/s]Processing Data ..... 0%|▏ | 6/2112 [00:01<06:53, 5.10it/s]Processing Data ..... 0%| | 3/2322 [00:00<06:03, 6.38it/s]Processing Data ..... 100%|████████████████████████████████████████████████████████████| 2076/2076 [01:19<00:00, 26.10it/s] 14%|████████▋ | 300/2112 [01:16<09:20, 3.23it/s]Processing Data ..... 100%|██████████████████████████████████████████████████████████████| 530/530 [00:26<00:00, 19.77it/s] 27%|████████████████▎ | 621/2322 [01:43<01:13, 23.15it/s] Data and model loaded

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/abduallahmohamed/Social-STGCNN/issues/34#issuecomment-713884401, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG7OYHJU6NR6J232UBLSL5GDVANCNFSM4SZ7LPNQ .

--

Abduallah Mohamed abduallahmohamed.com

[xingchenzhang]

Hi, This is normal it’s a progress bar issue when the screen is tight. Also you get multiple progress bars(5) per each training session. On Wed, Oct 21, 2020 at 4:20 PM Xingchen Zhang @.***> wrote: I think it's highly related to the environment + random seeds maybe? The exact CVPR version have the weights which the results were reported which can be helpful. This is the requirements file at the time of CVPR https://github.com/abduallahmohamed/Social-STGCNN/blob/master/checkpoint/requirements.txt I just uploaded it, it might be helpful to you. Hi, I found that when I run train.sh, the data processing stage showed this. Is this normal? I mean some progress bar wer not 100%. When you trained your model, how was this shown? Thanks! 8%|████▊ | 165/2076 [00:04<00:54, 34.93it/s]Processing Data ..... 9%|█████▊ | 197/2076 [00:05<00:59, 31.53it/s]Processing Data ..... 0%|▏ | 6/2112 [00:01<06:53, 5.10it/s]Processing Data ..... 0%| | 3/2322 [00:00<06:03, 6.38it/s]Processing Data ..... 100%|████████████████████████████████████████████████████████████| 2076/2076 [01:19<00:00, 26.10it/s] 14%|████████▋ | 300/2112 [01:16<09:20, 3.23it/s]Processing Data ..... 100%|██████████████████████████████████████████████████████████████| 530/530 [00:26<00:00, 19.77it/s] 27%|████████████████▎ | 621/2322 [01:43<01:13, 23.15it/s] Data and model loaded — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG7OYHJU6NR6J232UBLSL5GDVANCNFSM4SZ7LPNQ . -- ------------------------------------------ Abduallah Mohamed abduallahmohamed.com

Many thanks for your reply!

I have tried your requirements file, and also tried to recover my previous running environment. But so far, the results are still not good.

Apart from the running environment, you mentioned that the random seeds may affect the results. But I think this only affect the testing results, and will not affect training, right?

Thanks for your help!

[abduallahmohamed]

I think even though the environment might be the same a different GPU architecture and Cuda version can effect it too. Not sure about the random seed, nonetheless you have the weights in the repo to reproduce the results.

On Fri, Oct 23, 2020 at 9:37 AM Xingchen Zhang notifications@github.com wrote:

Hi, This is normal it’s a progress bar issue when the screen is tight. Also you get multiple progress bars(5) per each training session. On Wed, Oct 21, 2020 at 4:20 PM Xingchen Zhang @.***> wrote: I think it's highly related to the environment + random seeds maybe? The exact CVPR version have the weights which the results were reported which can be helpful. This is the requirements file at the time of CVPR https://github.com/abduallahmohamed/Social-STGCNN/blob/master/checkpoint/requirements.txt I just uploaded it, it might be helpful to you. Hi, I found that when I run train.sh, the data processing stage showed this. Is this normal? I mean some progress bar wer not 100%. When you trained your model, how was this shown? Thanks! 8%|████▊ | 165/2076 [00:04<00:54, 34.93it/s]Processing Data ..... 9%|█████▊ | 197/2076 [00:05<00:59, 31.53it/s]Processing Data ..... 0%|▏ | 6/2112 [00:01<06:53, 5.10it/s]Processing Data ..... 0%| | 3/2322 [00:00<06:03, 6.38it/s]Processing Data ..... 100%|████████████████████████████████████████████████████████████| 2076/2076 [01:19<00:00, 26.10it/s] 14%|████████▋ | 300/2112 [01:16<09:20, 3.23it/s]Processing Data ..... 100%|██████████████████████████████████████████████████████████████| 530/530 [00:26<00:00, 19.77it/s] 27%|████████████████▎ | 621/2322 [01:43<01:13, 23.15it/s] Data and model loaded — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#34 (comment) https://github.com/abduallahmohamed/Social-STGCNN/issues/34#issuecomment-713884401>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG7OYHJU6NR6J232UBLSL5GDVANCNFSM4SZ7LPNQ . -- ------------------------------------------ Abduallah Mohamed abduallahmohamed.com

Many thanks for your reply!

I have tried your requirements file, and also tried to recover my previous running environment. But so far, the results are still not good.

Apart from the running environment, you mentioned that the random seeds may affect the results. But I think this only affect the testing results, and will not affect training, right?

Thanks for your help!

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/abduallahmohamed/Social-STGCNN/issues/34#issuecomment-715381748, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG6CY5QK3UOY47LJEP3SMGIIZANCNFSM4SZ7LPNQ .

--

Abduallah Mohamed abduallahmohamed.com

[xingchenzhang]

I think even though the environment might be the same a different GPU architecture and Cuda version can effect it too. Not sure about the random seed, nonetheless you have the weights in the repo to reproduce the results. On Fri, Oct 23, 2020 at 9:37 AM Xingchen Zhang notifications@github.com wrote: Hi, This is normal it’s a progress bar issue when the screen is tight. Also you get multiple progress bars(5) per each training session. On Wed, Oct 21, 2020 at 4:20 PM Xingchen Zhang @.***> wrote: I think it's highly related to the environment + random seeds maybe? The exact CVPR version have the weights which the results were reported which can be helpful. This is the requirements file at the time of CVPR https://github.com/abduallahmohamed/Social-STGCNN/blob/master/checkpoint/requirements.txt I just uploaded it, it might be helpful to you. Hi, I found that when I run train.sh, the data processing stage showed this. Is this normal? I mean some progress bar wer not 100%. When you trained your model, how was this shown? Thanks! 8%|████▊ | 165/2076 [00:04<00:54, 34.93it/s]Processing Data ..... 9%|█████▊ | 197/2076 [00:05<00:59, 31.53it/s]Processing Data ..... 0%|▏ | 6/2112 [00:01<06:53, 5.10it/s]Processing Data ..... 0%| | 3/2322 [00:00<06:03, 6.38it/s]Processing Data ..... 100%|████████████████████████████████████████████████████████████| 2076/2076 [01:19<00:00, 26.10it/s] 14%|████████▋ | 300/2112 [01:16<09:20, 3.23it/s]Processing Data ..... 100%|██████████████████████████████████████████████████████████████| 530/530 [00:26<00:00, 19.77it/s] 27%|████████████████▎ | 621/2322 [01:43<01:13, 23.15it/s] Data and model loaded — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#34 (comment) <#34 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG7OYHJU6NR6J232UBLSL5GDVANCNFSM4SZ7LPNQ . -- ------------------------------------------ Abduallah Mohamed abduallahmohamed.com Many thanks for your reply! I have tried your requirements file, and also tried to recover my previous running environment. But so far, the results are still not good. Apart from the running environment, you mentioned that the random seeds may affect the results. But I think this only affect the testing results, and will not affect training, right? Thanks for your help! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG6CY5QK3UOY47LJEP3SMGIIZANCNFSM4SZ7LPNQ . -- ------------------------------------------ Abduallah Mohamed abduallahmohamed.com

Yes, I can reproduce your results using your pretrained model. But I am just wondering why the performance are so easily affected.

Anyway, many thanks for your help!

[abduallahmohamed]

To be honest it makes me wonder too! Fo instance I do have other works and even the same model can produce different results because the GPU is different! I’d suggest that something is off down there in the GPU itself

On Fri, Oct 23, 2020 at 10:27 AM Xingchen Zhang notifications@github.com wrote:

I think even though the environment might be the same a different GPU architecture and Cuda version can effect it too. Not sure about the random seed, nonetheless you have the weights in the repo to reproduce the results. On Fri, Oct 23, 2020 at 9:37 AM Xingchen Zhang notifications@github.com wrote: Hi, This is normal it’s a progress bar issue when the screen is tight. Also you get multiple progress bars(5) per each training session. On Wed, Oct 21, 2020 at 4:20 PM Xingchen Zhang @.***> wrote: I think it's highly related to the environment + random seeds maybe? The exact CVPR version have the weights which the results were reported which can be helpful. This is the requirements file at the time of CVPR https://github.com/abduallahmohamed/Social-STGCNN/blob/master/checkpoint/requirements.txt I just uploaded it, it might be helpful to you. Hi, I found that when I run train.sh, the data processing stage showed this. Is this normal? I mean some progress bar wer not 100%. When you trained your model, how was this shown? Thanks! 8%|████▊ | 165/2076 [00:04<00:54, 34.93it/s]Processing Data ..... 9%|█████▊ | 197/2076 [00:05<00:59, 31.53it/s]Processing Data ..... 0%|▏ | 6/2112 [00:01<06:53, 5.10it/s]Processing Data ..... 0%| | 3/2322 [00:00<06:03, 6.38it/s]Processing Data ..... 100%|████████████████████████████████████████████████████████████| 2076/2076 [01:19<00:00, 26.10it/s] 14%|████████▋ | 300/2112 [01:16<09:20, 3.23it/s]Processing Data ..... 100%|██████████████████████████████████████████████████████████████| 530/530 [00:26<00:00, 19.77it/s] 27%|████████████████▎ | 621/2322 [01:43<01:13, 23.15it/s] Data and model loaded — You are receiving this because you commented. Reply to this email directly, view it on GitHub <

34 https://github.com/abduallahmohamed/Social-STGCNN/issues/34

(comment) <#34 (comment) https://github.com/abduallahmohamed/Social-STGCNN/issues/34#issuecomment-713884401>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG7OYHJU6NR6J232UBLSL5GDVANCNFSM4SZ7LPNQ . -- ------------------------------------------ Abduallah Mohamed abduallahmohamed.com Many thanks for your reply! I have tried your requirements file, and also tried to recover my previous running environment. But so far, the results are still not good. Apart from the running environment, you mentioned that the random seeds may affect the results. But I think this only affect the testing results, and will not affect training, right? Thanks for your help! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#34 (comment) https://github.com/abduallahmohamed/Social-STGCNN/issues/34#issuecomment-715381748>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG6CY5QK3UOY47LJEP3SMGIIZANCNFSM4SZ7LPNQ . -- ------------------------------------------ Abduallah Mohamed abduallahmohamed.com

Yes, I can reproduce your results using your pretrained model. But I am just wondering why the performance are so easily affected.

Anyway, many thanks for your help!

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/abduallahmohamed/Social-STGCNN/issues/34#issuecomment-715412166, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLGZ2EWCQRWVMR6X54W3SMGOHJANCNFSM4SZ7LPNQ .

--

Abduallah Mohamed abduallahmohamed.com

[xingchenzhang]

Hi,

In your paper you mentioned 'A property to note regards TXP-CNN layer that it is not a permutation invariant as changes in the graph embedding right before TXP-CNN leads to different results.' Is this the reason why results are different? Because in each run, the input is actually different, for instance the order of pedestrians and sequences. Do you think this is one reason?

[abduallahmohamed]

Hi, I don't think so. The input arrangement of the pedestrians will results in the same graph embedding (permuted), but permuting the embeddings will result in a different behavior. Nonetheless, in the paper:

The 100% is the full data, we show the mean and std of our results based on different random seeds. On each run on a machine you will start the weights with a different set of variables that will lead to a different minima. I hope this helps!

[xingchenzhang]

Thank you very much for your reply!

[xingchenzhang]

Hi @abduallahmohamed,

In your script, you used train.sh to train five models together. Can I train five models one by one using the same hyper-parameters? I mean is this allowed for fair comparison?

Also, do we have to use the same hyper-parameters (the same learning rate, batch size, etc.) to train five models right? can we fine-tune each model (ETH, HOTEL, UNIV, ZARA1, ZARA2) and computer the average ADE and FDE?

[abduallahmohamed]

Hi Xingchen,

All models indeed use the same hyper-parameters. There's no issue in fair comparison even if you fine tuned each one for each dataset, nonetheless you have to pick up the best model based on the validation set and report the results on the test set. I think you can fine tune for each model but it might take a while.

Best,

Abduallah Mohamed abduallahmohamed.com

On Wed, Oct 28, 2020 at 1:26 PM Xingchen Zhang notifications@github.com wrote:

Hi @abduallahmohamed https://github.com/abduallahmohamed,

In your script, you used train.sh to train five models together. Can I train five models one by one using the same hyper-parameters? I mean is this allowed for fair comparison?

Also, do we have to use the same hyper-parameters (the same learning rate, batch size, etc.) to train five models right? can we fine-tune each model (ETH, HOTEL, UNIV, ZARA1, ZARA2) and computer the average ADE and FDE?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/abduallahmohamed/Social-STGCNN/issues/34#issuecomment-718124684, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG2SHL5SB25A6ENNRBLSNBO5HANCNFSM4SZ7LPNQ .

[xingchenzhang]

Hi Abduallah,

If I understand you correctly, it is allowed to fine tune each model (for example using different hyper-parameters for each model) as long as I pick up the best model based on the validation set and report the results on the test set. However, people normally use the same hyper-parameters, right? Because in your paper and other related papers, I just saw you guys reported one set of hyper-parameters. I assumed that these are applied to all five models.

Thank you very much for your reply! You are very kind! Xingchen

[abduallahmohamed]

Hi Xingchen,

Indeed most of papers as far as I know use the same settings of hyper parameters same as mine. But it’s the normal that’s not mandated, it’s just a hassle to fine tune for each dataset.

Best, Abduallah

On Wed, Oct 28, 2020 at 1:56 PM Xingchen Zhang notifications@github.com wrote:

Hi Abduallah,

If I understand you correctly, it is allowed to fine tune each model (for example using different hyper-parameters for each model) as long as I pick up the best model based on the validation set and report the results on the test set. However, people normally use the same hyper-parameters, right? Because in your paper and other related papers, I just saw you guys reported one set of hyper-parameters. I assumed that these are applied to all five models.

Thank you very much for your reply! You are very kind! Xingchen

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/abduallahmohamed/Social-STGCNN/issues/34#issuecomment-718141411, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIVLG3ARCVAKK6QV7JSOBTSNBSOVANCNFSM4SZ7LPNQ .

--

Abduallah Mohamed abduallahmohamed.com

[xingchenzhang]

Hi Abduallah,

Thank you very much for your reply!

Bests, Xingchen