search

liuzhengzhe / One-Thing-One-Click

https://arxiv.org/abs/2104.02246 One Thing One Click (CVPR 2021) https://arxiv.org/abs/2303.14727 One Thing One Click++ (Arxiv)
53 stars 8 forks source link

training time #10

Open Colin97 opened 3 years ago

Colin97 commented 3 years ago

Hi, How long does it take to train your network from scratch? It seems that you need to train 512 epochs (3D U-Net) for each iteration, and it needs ~5 iterations in total. So how long does it take? Which GPUs are you using?

liuzhengzhe commented 3 years ago

In 2 to 5 iterations, we fine-tune the model from the previous iter for 256 epochs to save time. Also, it is ok to train for 3 iters since 3 iter can already achieve good enough performance.

However, it takes a very long time, around 7 to 10 days to train a model.

We use 1 single 1080.

Liu Minghua @.***> 于2021年10月19日周二 上午11:00写道：

Hi, How long does it take to train your network from scratch? It seems that you need to train 512 epochs (3D U-Net) for each iteration, and it needs ~5 iterations in total. So how long does it take? Which GPUs are you using?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/liuzhengzhe/One-Thing-One-Click/issues/10, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7WPDGUOQ6TD4XJA2HBDFDUHTNNVANCNFSM5GIBW36Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Colin97 commented 3 years ago

May I know how many iterations in each epoch?

liuzhengzhe commented 3 years ago

iter1 is 512, others are 256.

Liu Minghua @.***> 于2021年10月19日周二 上午11:49写道：

May I know how many iterations in each epoch?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/liuzhengzhe/One-Thing-One-Click/issues/10#issuecomment-946343958, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7WPDFCQ33XX3GARS56UQ3UHTTFRANCNFSM5GIBW36Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Colin97 commented 3 years ago

not number of epochs, how many (gradient descent) iterations of each epoch?

liuzhengzhe commented 3 years ago

batch size=4, 1201 samples, 300 iterations per epoch.

Liu Minghua @.***> 于2021年10月19日周二 下午12:06写道：

not number of epochs, how many iterations of each epoch?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/liuzhengzhe/One-Thing-One-Click/issues/10#issuecomment-946349540, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7WPDBEKTEIGQUCIEQ3Q3LUHTVC5ANCNFSM5GIBW36Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Colin97 commented 3 years ago

thanks

Colin97 commented 3 years ago

Hi~ For Relation Net, how many epochs shall we train for each iteration?

derkaczda commented 2 years ago

In 2 to 5 iterations, we fine-tune the model from the previous iter for 256 epochs to save time. Also, it is ok to train for 3 iters since 3 iter can already achieve good enough performance. However, it takes a very long time, around 7 to 10 days to train a model. We use 1 single 1080. Liu Minghua @.***> 于2021年10月19日周二 上午11:00写道： … Hi, How long does it take to train your network from scratch? It seems that you need to train 512 epochs (3D U-Net) for each iteration, and it needs ~5 iterations in total. So how long does it take? Which GPUs are you using? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7WPDGUOQ6TD4XJA2HBDFDUHTNNVANCNFSM5GIBW36Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

the 7 to 10 days per model, does this mean 7 to 10 days for one iteration of the complete pipeline or just to train the relation or unet for 500 epochs?

liuzhengzhe commented 2 years ago

the complete pipeline

derkaczda @.***> 于2021年11月12日周五 下午6:23写道：

In 2 to 5 iterations, we fine-tune the model from the previous iter for 256 epochs to save time. Also, it is ok to train for 3 iters since 3 iter can already achieve good enough performance. However, it takes a very long time, around 7 to 10 days to train a model. We use 1 single 1080. Liu Minghua @.***> 于2021年10月19日周二 上午11:00写道： … <#m-3632165755689304514> Hi, How long does it take to train your network from scratch? It seems that you need to train 512 epochs (3D U-Net) for each iteration, and it needs ~5 iterations in total. So how long does it take? Which GPUs are you using? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10 https://github.com/liuzhengzhe/One-Thing-One-Click/issues/10>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7WPDGUOQ6TD4XJA2HBDFDUHTNNVANCNFSM5GIBW36Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

the 7 to 10 days per model, does this mean 7 to 10 days for one iteration of the complete pipeline or just to train the relation or unet for 500 epochs?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/liuzhengzhe/One-Thing-One-Click/issues/10#issuecomment-966988669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7WPDFGWF35VJAZ3DQ2PMTULTTJRANCNFSM5GIBW36Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

derkaczda commented 2 years ago

Interesting because it already takes up to 10 days to train the unet for me (RTX 6000) , did you do any further improvements?

liuzhengzhe commented 2 years ago

Oh, config/pointgroup_run1_scannet.yaml, line 32 should be 16. Sorry about that.

derkaczda @.***> 于2021年11月12日周五 下午7:21写道：

Interesting because it already takes up to 10 days to train the unet for me (RTX 6000) , did you do any further improvements?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/liuzhengzhe/One-Thing-One-Click/issues/10#issuecomment-967025613, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7WPDHII6UV224O5UQLIUDULT2E5ANCNFSM5GIBW36Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.