search

XuyangBai / D3Feat

[TensorFlow] Official implementation of CVPR'20 oral paper - D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features https://arxiv.org/abs/2003.03164
MIT License
259 stars 38 forks source link

Accuracy and other parameters drop precipitously after about 40 epochs #37

Closed wu546300070 closed 2 years ago

wu546300070 commented 3 years ago

Hi Xuyang I didn't modify any code, and when I trained the 3DMatch dataset, the accuracy suddenly dropped to almost zero at about 40 epochs, and then slowly rebounded, can you explain why? (My environment is set up according to environment.yaml)

The link shows the accuracy results (https://user-images.githubusercontent.com/42727019/127865849-4d4059f2-6fa2-4a62-b9e9-e6afc2c53426.png)

best Yichao

XuyangBai commented 3 years ago

Hi @wu546300070 Thanks for your interest in our work. I also met such problems that the training broke at some time if I remember correctly, but it happens very accidentally. You may just restart another training.

Best, Xuyang

wu546300070 commented 3 years ago

Hi Xuyang Thank you for your quick reply! I've tested it three times (second time I changed Batch_num=16) But unfortunately, it seems to encounter this problem all the time.. https://user-images.githubusercontent.com/42727019/127881335-15ff9942-9911-42a1-ac03-73ea69771488.jpg

XuyangBai commented 3 years ago

Did you use circle loss? I remember this cases happens more often for circle loss, for which the network converges fast but have the risk of broken down and I stop training more early to avoid the broke down. And you may evaluate the model checkpoint just before broken down to see the performance so far

获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: wu546300070 @.> 发送时间: Monday, August 2, 2021 10:55:56 PM 收件人: XuyangBai/D3Feat @.> 抄送: Xuyang BAI @.>; Comment @.> 主题: Re: [XuyangBai/D3Feat] Accuracy and other parameters drop precipitously after about 40 epochs (#37)

Hi Xuyang Thank you for your quick reply! But I've tested it three times (second time I changed Batch_num=16) But unfortunately, it seems to encounter this problem all the time.. https://user-images.githubusercontent.com/42727019/127881335-15ff9942-9911-42a1-ac03-73ea69771488.jpghttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F42727019%2F127881335-15ff9942-9911-42a1-ac03-73ea69771488.jpg&data=04%7C01%7Cxbaiad%40connect.ust.hk%7C8bc8cf7d442f4ef79f2308d955c5a34f%7C6c1d415239d044ca88d9b8d6ddca0708%7C1%7C0%7C637635129607606671%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HplY7pMhKIMfgd0viLrBW08mPL2X8au9pl8xYvd3u%2FI%3D&reserved=0

― You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FXuyangBai%2FD3Feat%2Fissues%2F37%23issuecomment-891093527&data=04%7C01%7Cxbaiad%40connect.ust.hk%7C8bc8cf7d442f4ef79f2308d955c5a34f%7C6c1d415239d044ca88d9b8d6ddca0708%7C1%7C0%7C637635129607616614%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vddX5TbT3MegfPhAckNqxLU8uS0kbOEXnGX9pn4KQIc%3D&reserved=0, or unsubscribehttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADTWVCOTFFNE4EHNK475G4LT22WXZANCNFSM5BMV7HRA&data=04%7C01%7Cxbaiad%40connect.ust.hk%7C8bc8cf7d442f4ef79f2308d955c5a34f%7C6c1d415239d044ca88d9b8d6ddca0708%7C1%7C0%7C637635129607626575%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3Ipl2n0amFkH8GcpgTGaMzWICnM%2BOLXTwbWl5w%2BDcq4%3D&reserved=0.

wu546300070 commented 3 years ago

Thank you for your prompt reply, I am already trying to use different loss parameters and I will let you know if the problem is solved as soon as possible ;)

best Yichao