Cannot reproduce experimental results on Openlane?

[inconnu11]

Hello all,

I ran the Openlane experiment based on the provided codebase and didn't change any parameters in the config file. I tested multiple models saved at different epochs and only got the best results of 56.3% F-score on Openlane validation set, which is still 2% lower than the number shown in your paper (i.e., 58.4% F-score). Is there anything wrong? Could you please give me some advice?

[qinjian623]

hi,

You can try adjusting the training hyperparameters, as different numbers of GPUs and hardware environments may yield different results.

It is recommended to adjust the learning rate (lr) and batch size.

Using this version of the open-source code, we can achieve F1 score of 60.

[yuaoze]

hi,

You can try adjusting the training hyperparameters, as different numbers of GPUs and hardware environments may yield different results.

It is recommended to adjust the learning rate (lr) and batch size.

Using this version of the open-source code, we can achieve F1 score of 60.

hi, I've tried and get the best result of F-score is 57.4% on Openlane, My device is 2x3090, batch_size is 32, other hyperparameters are same as the code. It shows that loss oscillated around 5 without decline. Could you please give me some advice?

[secret104278]

HI, @inconnu11 and @yuaoze, is there any possible that you could share your own trained model for quick evaluation?

[qinjian623]

hi, @yuaoze

We use a hardware configuration of 8 x V100, which you can use as a reference to adjust hyperparameters.

[yuaoze]

sure, please leave your email

[Laulian]

sure, please leave your email @yuaoze lanlian919@gmail.com ~ It will be greatly appreciated

[secret104278]

sure, please leave your email

secret104278@gmail.com

thanks a lot ~

[zhanghui75]

Could you also kindly share your trained model with me? Thanks a lot. My email: zhh9297@163.com

[imyyf]

hi, You can try adjusting the training hyperparameters, as different numbers of GPUs and hardware environments may yield different results. It is recommended to adjust the learning rate (lr) and batch size. Using this version of the open-source code, we can achieve F1 score of 60.

hi, I've tried and get the best result of F-score is 57.4% on Openlane, My device is 2x3090, batch_size is 32, other hyperparameters are same as the code. It shows that loss oscillated around 5 without decline. Could you please give me some advice?

Hi which epoch you choose to get 57.4% on openlane? we use epoch049(batch_size is 32) to test, only get 53%.

[adeelahmedrana]

Hi, first of a very impressive paper. Can you please share the weights with me too at adeel0311@gmail.com

Many thanks

[Yutong-gannis]

@yuaoze Could you please share your weights with me just for a early demo. Thanks a lot.

My email: 3118891520@qq.com yutongliu840@gmail.com

[westlife35]

@yuaoze Could you please share your weights(57.4% on openlane). Thanks~ My email: 2364426805@qq.com

[superchenyan]

sure, please leave your email

my email: superchenyan@sina.com ths!!!

[drilistbox]

could you share your weight to me, my email is drilistbox@buaa.edu.cn

[PyNancy]

sure, please leave your email

could you please share your weight to me, thanks a lot, guoweiyang8199@gmail.com

[WwwwYz666]

Could you please share your checkpoint weight ? My email is wangyic19@gmail.com.

[CaiFuyao]

could you please share your best model to me , thanks a lot! My email is cfycfy@hust.edu.cn.

[s95huang]

Here is what I got, {'f1_score': 0.5651117116621163, 'precision': 0.6838964779914308, 'recall': 0.4814843714408606, 'x_error_close': 0.25393218394610845, 'x_error_far': 0.6785732514552418, 'z_error_close': 0.19615825120228755, 'z_error_far': 0.6227202588181409}

[Pocan0]

Here is what I got, {'f1_score': 0.5651117116621163, 'precision': 0.6838964779914308, 'recall': 0.4814843714408606, 'x_error_close': 0.25393218394610845, 'x_error_far': 0.6785732514552418, 'z_error_close': 0.19615825120228755, 'z_error_far': 0.6227202588181409}

could you please share your model to me , thanks a lot! My email is cshenwhut@126.com.

[Mollylulu]

@yuaoze Could you please share your reproduced model weights with me? Thanks a lot! Email address: gracegreen77@163.com.

[EnternalTwinkle]

@yuaoze Can you share the source code? Thank you very much，1017094591@qq.com

[Ywenjun123]

Hello all,

I ran the Openlane experiment based on the provided codebase and didn't change any parameters in the config file. I tested multiple models saved at different epochs and only got the best results of 56.3% F-score on Openlane validation set, which is still 2% lower than the number shown in your paper (i.e., 58.4% F-score). Is there anything wrong? Could you please give me some advice?

could you please share your model to me , thanks a lot! My email is 1119782358@qq.com

[happyday-lkj]

Here is what I got, {'f1_score': 0.5651117116621163, 'precision': 0.6838964779914308, 'recall': 0.4814843714408606, 'x_error_close': 0.25393218394610845, 'x_error_far': 0.6785732514552418, 'z_error_close': 0.19615825120228755, 'z_error_far': 0.6227202588181409}

could you share your pretrained model, happyday_lkj@163.com

[spring35]

sure, please leave your email

could you share your model, thanks! my email is 2893102339@qq.com

[Benson722]

Could you also kindly share your trained model with me? Thanks a lot. My email: 583032099@qq.com thx!