Strange recall values

[zzangjinsun]

Hello.

I ran the code with Pitts30K and TokyoTM dataset.

When I ran the code with "vd16_pitts30k_conv5_3_vlad_preL2_intra_white" or "vd16_tokyoTM_conv5_3_vlad_preL2_intra_white" models with Pitts30K test set, I got following recall values.

Recall@ 1 0.0117 2 0.0239 3 0.0333 4 0.0427 5 0.0543

If I run the same code with TokyoTM dataset, it gives perfect result (above 95%). Although I utilized provided code examples from demo.m, the result looks strange. Following is my evaluation code:

netID= 'vd16_pitts30k_conv5_3_vlad_preL2_intra_white'; paths= localPaths(); load( sprintf('%s%s.mat', paths.ourCNNs, netID), 'net' ); net= relja_simplenntidy(net); dbTest = dbPitts(false, 'test'); dbFeatFn= sprintf('%s%s%sdb.bin', paths.outPrefix, netID, dbTest.name); qFeatFn = sprintf('%s%s%s_q.bin', paths.outPrefix, netID, dbTest.name); batchSize = 32; serialAllFeats(net, dbTest.dbPath, dbTest.dbImageFns, dbFeatFn, 'batchSize', batchSize, 'useGPU', true); serialAllFeats(net, dbTest.qPath, dbTest.qImageFns, qFeatFn, 'batchSize', batchSize, 'useGPU', true); N = 50; listN = [1:N]; opts= struct(... 'nTestRankSample', 0, ... 'nTestSample', inf, ... 'recallNs', listN, ... 'margin', 0.1, ... 'nNegChoice', 1000, ... 'cropToDim', 0 ... ); [recall, ~, ~, opts]= testFromFn(dbTest, dbFeatFn, qFeatFn, opts);

Is there any suggestion for possible fault from my side? In addition, is it possible to get exact values in Fig. 5 of the paper?

[Relja]

Hi,

Strange, I don't see anything wrong with your code right now. Are you sure that serialAllFeats computed the features correctly (e.g. it can happen that first time you ran it it failed due to some setup being incorrect, making a corrupt feature file; then you fix your problem and rerun the code, but the file exists so features are not recomputed and you still have a corrupt file)? E.g. just check if it is the correct size (should be db.numImages 4096 4 bytes for dbFeatFn and db.numQueries 4096 4 for qFeatFn). If this seems right, can you maybe send me the two feature files, so I can check if a) the features are correct b) I get the same recalls.

Also, just to know for later - what version of MatConvNet are you using? (sometimes their updates broke my code, but if it runs without errors then it should be fine..)

I'll see if I can dig out the exact values from Fig.5 tonight.

[zzangjinsun]

I am using following environments:

• Ubuntu 14.04 LTS
• Cuda 8.0
• MatConvNet 1.0-beta25
• Matlab R2017b

After deleting existing files, I got the same values. I will send you extracted features through e-mail!

[gittigxuy]

@Relja ,could you please share your result?my loss is about 0.3,and my accuracy is about 60%~70%,is it your result?how could I improve my accuracy up to 90%?thank you

[Relja]

What do you mean by sharing my result? The paper has the results, the networks have been released so you can use them to get my result.

[Mendel1]

@zzangjinsun Did you find what the problem is? I got the same issue as yours.When I test the model on Pitts250k dataset, it runs well and the result is just as same as those in your paper.However when I test the model on Tokyo24/7 , it's far away from your results.The result is as follow:

001 0.0984
002 0.1492
003 0.1714
004 0.1778
005 0.1968
010 0.2381
015 0.2857
020 0.3048
025 0.3492
030 0.3746
035 0.3968
040 0.4222
045 0.4540
050 0.4635
055 0.4698
060 0.4762
065 0.4889
070 0.5079
075 0.5206
080 0.5270
085 0.5333
090 0.5429
095 0.5460
100 0.5524

And the model I use is vd16_tokyoTM_conv5_3_vlad_preL2_intra_white.mat downloaded from official website.The test procedure is as same as that in demo.m.And I have also checked the sizes of generated feature files amd they are both correct.

[Relja]

It concludes with his email:

After re-installing matconvnet and netvlad codes, I think I got the correct recall values (Recall@1 about 85%).

I am not sure which part was wrong but I think the problem is solved.

[Mendel1]

It concludes with his email:

After re-installing matconvnet and netvlad codes, I think I got the correct recall values (Recall@1 about 85%).

I am not sure which part was wrong but I think the problem is solved.

OK. I'll try that solution.Thank you very much