search

zeliu98 / CloserLook3D

A Closer Look at Local Aggregation Operators in Point Cloud Analysis(ECCV 2020)
MIT License
252 stars 36 forks source link

S3DIS performance #8

Open TXH-mercury opened 4 years ago

TXH-mercury commented 4 years ago

Hi zeliu Thanks for the awesome work and open-source code! I have encontered some trouble running the code. I changed nothing with the code and only use the default configs to train on S3DIS. But I get the result(adaptive weights : 57.5mIOU ,pospool-xyz: 57.6mIOU ) which harshly lower than your report (66.5 , 66.5) I can't figure out the performance gap , is there something I am supposed to change ? \

Another question is : In adaptive-weights method , the weight is a vector and not a matrix . Even if this can reduce the param nums and FLOPs, but how about the performance influence?

Thank you!

TXH-mercury commented 4 years ago

image trained using pytorch , 4 gpu ,batch_size=8 (the same as the instructions)

zeliu98 commented 4 years ago

Hi @TXH-mercury Thanks for your interest in our work. The performance drop may due to the different reproduction details in PyTorch, like some hyper parameters and the masks in downsampling & neighborhoods. We will figure it out and update the code as soon as possible. As for the weights, we use a depth-wise convolution and we found little influence compared with the normal convolution.

alitabet commented 3 years ago

Hi @zeliu98 thank you for your code. Have you managed to look into the issue of hyperparameters in the PyTorch version? If not, do you have some guidance as to which parameters we need to tune and some range of values for them? I'm trying to reproduce your results and get similar performance as @TXH-mercury, I can do a sweep of hyperparameters given the appropriate guidance.

Thanks again for your great work!

zhulf0804 commented 3 years ago

Hi @zeliu98 , i ran the code you provided above trained on s3dis datasets with pytorch/cfgs/s3dis/pospool_sin_cos_avg.yaml. I also can't get the higher iou as your paper. The result can be seen bellow: image

Can you figure out what causes the low iou ?

Hi @TXH-mercury , have you solved the the lower metric problem ?

zeliu98 commented 3 years ago

Hi all, we have released the pytorch models of S3DIS. Please let me you know if you have other questions.

guochengqian commented 3 years ago

@zeliu98 i am wondering if you are using the default parameter provided in the cfgs? all experiments are conducted using 4 GPUs? I have run S3DIS using pospool with 4 V100s. I get mIoU 64.5, which is fairly good, but still 1 point lower than your reported value. Is that because of the randomness or I used incorrect experiment setting (like not 4GPUs).

tangbohu commented 3 years ago

How you get the final performance. Evalute the models at 600 epochs, or the model with the best validation accuracy?