search

cvlab-stonybrook / DM-Count

Code for NeurIPS 2020 paper: Distribution Matching for Crowd Counting.
MIT License
218 stars 52 forks source link

Influence of data preprocessing on model performance #10

Closed Fang-Lansheng closed 3 years ago

Fang-Lansheng commented 3 years ago

Hi, thanks for your excellent work and the open-source codes! I'm very interested in the novel optimization criterion proposed in your paper and tried to reproduce it locally. But there is a confusing problem of performance degradation when I restrained the height and width of the test images to make sure that they are divisible by a particular number (like, 8, 16, or 32). This data preprocessing strategy is inspired by C^3 Framework and guarantees the output size of some down-sampling layers count meet the requirements of subsequent processing. More details of the test results on ShanghaiTech Part_A are shown in the table below.

pretrained resize MAE MSE
Reported ✔️ 59.68 95.72
Reproduce ✔️ ✔️ 62.47 101.98

This issue is to sincerely ask you how such a simple operation can have such an obvious impact on the experimental results.

Boyu-Wang commented 3 years ago

Hi, these changes seem minor, I don't think it should affect the performance a lot. Have you tried the following: 1) Test the performance of the pre-trained model on your resized images. You could find the pre-trained model in the Google Drive link from the main repo. 2) Before training, preprocess the training images using the same preprocessing pipeline. These are some steps I could think of to identify the problem.

Fang-Lansheng commented 3 years ago
Thanks for your reply! The 'reproduce results' mentioned above are in fact the test results on the resized images using the pre-trained model. Here are some other experimental data: MAE MSE
Test the pre-trained model on resized images 62.47 101.98
Train and test on resized images 61.87 100.95
Train and test on unprocessed images 61.26 102.07

Unfortunately, the results still fell short of expectations.

Boyu-Wang commented 3 years ago

This is interesting. You could find those images whose performance changes a lot with preprocessing and w./o. preprocessing. And study these examples. Make sure the preprocessing is performed consistently for both image and annotation. In theory, such preprocessing will not affect loss computation.

Fang-Lansheng commented 3 years ago

Thanks very much😃. Maybe I should check my code and analyzing the test results to figure out where the problem is.