search

LiheYoung / FreeMask

[NeurIPS 2023] FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models
https://arxiv.org/abs/2310.15160
MIT License
129 stars 1 forks source link

failure to reproduce accuracy with provided data and code #7

Open HongkLin opened 3 months ago

HongkLin commented 3 months ago

Thank you for your work. However, I encountered a problem in the process of reproduction. I repeated the experiment several times using the data and code provided by you, but I could not reproduce the results in the paper. Specifically, I trained a segformer-b4 model using the ade synthetic data provided by you and the real data according to your code. Training four times, we get 50.1, 49.5, 49.7, 49.8, but we can't reach the 50.6 in the paper. Even using mask2former-t with mixed training following your code is only 50.8. Not only did we mix real data, but even when we trained segformer-b4 on synthetic data, we only got 47.9. Is the synthetic data you provided consistent with the experiment in the paper? Or maybe some Settings in the code didn't reset after a few trials.

LiheYoung commented 3 months ago

Here are our training logs: https://github.com/LiheYoung/FreeMask/tree/main/training-logs.

Could you please provide your training logs?

HongkLin commented 3 months ago

20240716_171240.log 20240719_001050.log 20240721_061630.log These are our training logs.

HongkLin commented 3 months ago

Here are our training logs: https://github.com/LiheYoung/FreeMask/tree/main/training-logs.

Could you please provide your training logs?

linyq2117 commented 3 months ago

Here are our training logs: https://github.com/LiheYoung/FreeMask/tree/main/training-logs. Could you please provide your training logs?

Hi, I encountered the same problem and only got 49.7 mIoU during reproduction. Have you addressed it?

HongkLin commented 3 months ago

Here are our training logs: https://github.com/LiheYoung/FreeMask/tree/main/training-logs. Could you please provide your training logs?

Hi, I encountered the same problem and only got 49.7 mIoU during reproduction. Have you addressed it?

Not yet, but it looks like a big error. You can try it a few times.

linyq2117 commented 3 months ago

Here are our training logs: https://github.com/LiheYoung/FreeMask/tree/main/training-logs. Could you please provide your training logs?

Hi, I encountered the same problem and only got 49.7 mIoU during reproduction. Have you addressed it?

Not yet, but it looks like a big error. You can try it a few times.

Thanks a lot!