search

KunpengLi1994 / VSRN

PyTorch code for ICCV'19 paper "Visual Semantic Reasoning for Image-Text Matching"
288 stars 47 forks source link

Python and PyTorch version #15

Closed Jiangtong-Li closed 4 years ago

Jiangtong-Li commented 4 years ago

I have switched the PyTorch from 0.4.1 to 1.3.1 and the Python from 2.7.15 to 3.6.9. During switching the version, I only perform minor changes towards the code to make the code fit for the newer version. But the performance on f30k has dropped greatly (i2t-r1:from 70% to 35%; t2i:from 50% to 26%). Are there any reasons that might cause this situation? Thanks!

Jiangtong-Li commented 4 years ago

Besides, I also find that if I switch the python version from 2 to 3. The evaluation result for your provided pre-trained model will also drop from 71 to 65(i2t-r1).

KunpengLi1994 commented 4 years ago

Hi,

Our code is built on top of VSE++ with python 2.7. We did not release a version to support Python 3 and latest Pytorch. There should be some difference between different Python or Pytorch versions, which may not lead to errors but can cause different computation results. We did not check the code line by line to find out these difference.

There is a issue in VSE++ talking about the Python 3 version and also a brunch related. I did not check whether it can really work well but hope this can help.

Jiangtong-Li commented 4 years ago

Thank you for your reply! I have find the cause of dropping performance. In data.py line 206 ”with open(loc+'%s_caps.txt' % data_split, 'rb') as f:”, you need to change the open mode from rb to r if you switch the python from 2 to 3, otherwise, the line from the txt file will start with b’,which will confuse the word tokenizer of nltk. Besides, there are still some points need to revise, like the use of “/“ and “iteritem”.

Jiangtong Li

Ph.D Student

Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

Mobile: +8613772437471

E-mail: keep_moving-Lee@sjtu.edu.cn jiangtongli1997@gmail.com

2020年1月15日 22:36，Kunpeng Li notifications@github.com 写道：

Hi,

Our code is built on top of VSE++ https://github.com/fartashf/vsepp with python 2.7. We did not release a version to support Python 3 and latest Pytorch. There should be some difference between different Python or Pytorch versions, which may not lead to errors but can cause different computation results. We did not check the code line by line to find out these difference.

There is a issue https://github.com/fartashf/vsepp/issues/12 in VSE++ talking about the Python 3 version and also a brunch https://github.com/fartashf/vsepp/tree/python3 related. I did not check whether it can really work well but hope this can help.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/KunpengLi1994/VSRN/issues/15?email_source=notifications&email_token=ANGJKU4MDT2EYXYDABFOIFDQ54NQLA5CNFSM4KG5HHEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJAQQZQ#issuecomment-574687334, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANGJKU2373VMA43ITZH4CDDQ54NQLANCNFSM4KG5HHEA.

KunpengLi1994 commented 4 years ago

Thanks for pointing out. Once you finish all revisions and can get the same results with using Python 2, you can then create a new branch with pull request for this Python 3 version. Just as how VSE++ did here.

Jiangtong-Li commented 4 years ago

Sorry to interrupt you again.

In your code about VSRN, in data.py. I find that the collect_fn sorted the loaded data by the caption length, which means the order of image in a mini-batch is disrupted. So the cap_emb and img_emb you get from encode_data may not have the property that the n5 to (n+1)5 belongs to the same corresponding relationship.

From what I find above, I evaluation results you present may not be the real Recall@k. I’d like to know if I have anything misunderstood and hope you can point out.

Thanks!

Jiangtong Li

Ph.D Student

Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

Mobile: +8613772437471

E-mail: keep_moving-Lee@sjtu.edu.cn jiangtongli1997@gmail.com

2020年1月16日 03:21，Kunpeng Li notifications@github.com 写道：

Thanks for pointing out. Once you finish all revisions and can get the same results with using Python 2, you can then create a new branch with pull request for this Python 3 version. Just as how VSE++ did here https://github.com/fartashf/vsepp/pull/13.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/KunpengLi1994/VSRN/issues/15?email_source=notifications&email_token=ANGJKU7ZFQL43ACJBVW6BPTQ55O2PA5CNFSM4KG5HHEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJBPTOQ#issuecomment-574814650, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANGJKUYAW35OTXY6TRIADVDQ55O2PANCNFSM4KG5HHEA.

KunpengLi1994 commented 4 years ago

data.sort won't change the pair relation between cap_emb and img_emb. You easily can check this.

This is a common operation so that padding can be done later. You can also find this operation at this line of VSE++.

LetsGoFir commented 4 years ago

Sorry to interrupt you again. In your code about VSRN, in data.py. I find that the collect_fn sorted the loaded data by the caption length, which means the order of image in a mini-batch is disrupted. So the cap_emb and img_emb you get from encode_data may not have the property that the n5 to (n+1)5 belongs to the same corresponding relationship. From what I find above, I evaluation results you present may not be the real Recall@k. I’d like to know if I have anything misunderstood and hope you can point out. Thanks! -------------------------------------------------------------------------------- Jiangtong Li Ph.D Student Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China Mobile: +8613772437471 E-mail: keep_moving-Lee@sjtu.edu.cn jiangtongli1997@gmail.com … 2020年1月16日 03:21，Kunpeng Li @.***> 写道： Thanks for pointing out. Once you finish all revisions and can get the same results with using Python 2, you can then create a new branch with pull request for this Python 3 version. Just as how VSE++ did here <fartashf/vsepp#13>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#15?email_source=notifications&email_token=ANGJKU7ZFQL43ACJBVW6BPTQ55O2PA5CNFSM4KG5HHEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJBPTOQ#issuecomment-574814650>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANGJKUYAW35OTXY6TRIADVDQ55O2PANCNFSM4KG5HHEA.

Hi there, have you get the same result with torch 1.3?

Jiangtong-Li commented 4 years ago

Yes, I can reproduce the same results in PyTorch 1.3

Jiangtong Li

Ph.D Student

Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

Mobile: +8613772437471

E-mail: keep_moving-Lee@sjtu.edu.cn mailto:keep_moving-Lee@sjtu.edu.cn jiangtongli1997@gmail.com

2020年5月26日 16:09，LetsGoFir <notifications@github.com mailto:notifications@github.com> 写道：

Sorry to interrupt you again. In your code about VSRN, in data.py. I find that the collect_fn sorted the loaded data by the caption length, which means the order of image in a mini-batch is disrupted. So the cap_emb and img_emb you get from encode_data may not have the property that the n5 to (n+1)5 belongs to the same corresponding relationship. From what I find above, I evaluation results you present may not be the real Recall@k. I’d like to know if I have anything misunderstood and hope you can point out. Thanks! -------------------------------------------------------------------------------- Jiangtong Li Ph.D Student Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China Mobile: +8613772437471 E-mail: keep_moving-Lee@sjtu.edu.cn mailto:keep_moving-Lee@sjtu.edu.cn jiangtongli1997@gmail.com mailto:jiangtongli1997@gmail.com … <x-msg://2/#> 2020年1月16日 03:21，Kunpeng Li @.*> 写道： Thanks for pointing out. Once you finish all revisions and can get the same results with using Python 2, you can then create a new branch with pull request for this Python 3 version. Just as how VSE++ did here <fartashf/vsepp#13 https://github.com/fartashf/vsepp/pull/13>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#15 https://github.com/KunpengLi1994/VSRN/issues/15?email_source=notifications&email_token=ANGJKU7ZFQL43ACJBVW6BPTQ55O2PA5CNFSM4KG5HHEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJBPTOQ#issuecomment-574814650>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANGJKUYAW35OTXY6TRIADVDQ55O2PANCNFSM4KG5HHEA https://github.com/notifications/unsubscribe-auth/ANGJKUYAW35OTXY6TRIADVDQ55O2PANCNFSM4KG5HHEA.

Hi there, have you get the same result with torch 1.3?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/KunpengLi1994/VSRN/issues/15#issuecomment-633875976, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANGJKU7FR3H5QWTUK3WGEE3RTN2KFANCNFSM4KG5HHEA.

MauritsBleeker commented 3 years ago

How did you manage to reproduce the scores in the end with PyTorch 1.3? What are the modifications you have to make?

Thanks, Maurits