search

paul007pl / MVP_Benchmark

MVP Benchmark for Multi-View Partial Point Cloud Completion and Registration
https://mvp-dataset.github.io/
Apache License 2.0
119 stars 10 forks source link

Questions about pre-trained weight and so on #16

Closed ganddam closed 2 years ago

ganddam commented 3 years ago

Hello, I have three questions about this competition.

  1. Will it be illegal to use the pre-trained weight from other methods, like "PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers"? The result could be better if I train the network based on the pre-trained PCN/C3D weight.
  2. Is the submission on the competition website still effective during the private submission stage?
  3. Can we only give the private submission without showing it on the leaderboard?

Thank you very much!

paul007pl commented 3 years ago

Hi,

  1. It is actually not fair to use those pre-trained weights, because they are actually trained with additional data. Rather than performing a retrival task, it can be more interesting if studying a conditional generation task.
  2. We have not closed the submission, but the main purpose is to help you prepare your pretrained model. As for the final ranking list and the reward, we may not consider those results that are submitted after the deadline.
  3. You can provide those pretrained weights that are previously submitted to the leaderboard, no matter whether you show it or hide it. But we should be able to find the corresponding submission record. Good luck~
xljh0520 commented 3 years ago

Hi,

  1. It is actually not fair to use those pre-trained weights, because they are actually trained with additional data. Rather than performing a retrival task, it can be more interesting if studying a conditional generation task.
  2. We have not closed the submission, but the main purpose is to help you prepare your pretrained model. As for the final ranking list and the reward, we may not consider those results that are submitted after the deadline.
  3. You can provide those pretrained weights that are previously submitted to the leaderboard, no matter whether you show it or hide it. But we should be able to find the corresponding submission record. Good luck~

Hi, I think it's common and usual to use pretrained weight. In 2d cv task like detection and classification, imageNet, COCO are often used to pretrain a model. I think it's also reasonable to use pretrained weight in 3d task and this challenge. Best wishes.

paul007pl commented 3 years ago

Hi,

  1. It is actually not fair to use those pre-trained weights, because they are actually trained with additional data. Rather than performing a retrival task, it can be more interesting if studying a conditional generation task.
  2. We have not closed the submission, but the main purpose is to help you prepare your pretrained model. As for the final ranking list and the reward, we may not consider those results that are submitted after the deadline.
  3. You can provide those pretrained weights that are previously submitted to the leaderboard, no matter whether you show it or hide it. But we should be able to find the corresponding submission record. Good luck~

Hi, I think it's common and usual to use pretrained weight. In 2d cv task like detection and classification, imageNet, COCO are often used to pretrain a model. I think it's also reasonable to use pretrained weight in 3d task and this challenge. Best wishes.

Hi,

I agree. It is common to use pretrained model to boost performance for many challenging tasks. Also, many interesting research topics, such as transfer learning, domain adaption, few-shot learning and so on, are highly related to pretrained models.

However, in this challenge, we aim to achieve fair comparisons between different methods, and we hope the performance gain is mainly resulted from your smart design and novel method, which can be more valuable. As for pretraining, you can of course use the allowed data, the MVP Training Set (multi-stage training, iterative refinement, or other strategies). But it is definitely not fair to use the weights that are pretrained on extra data, and we clearly stated in advance that no additional data are allowed. I do think it is a serious issue.

On the other hand, I understand. And I suggest you can report your results with the help of pretrained weights, as well as the results without using the pretrained weights. The core idea is to highlight your contributions and novelty.