search

hanjanghoon / BERT_FP

Fine-grained Post-training for Improving Retrieval-based Dialogue Systems - NAACL 2021
97 stars 19 forks source link

Is it normal for me to spend 22 hour per epoch of training? #6

Closed bott1eee closed 2 years ago

bott1eee commented 2 years ago

When I post-train with the ubuntu dataset, it takes 22 hours per epoch, 25 epochs means it takes close to a month, maybe the time is too long. Is this also the case for the author when training? Is this normal in my case? Thanks!

Iteration: 100%|██████████| 81865/81865 [22:53:38<00:00, 1.01s/it]

hanjanghoon commented 2 years ago

Hello!

In my case it took 10 hours per epochs. but it different per gpu spec. so i recommand post-training a model about 5~10 times. because that's enough to improve the performance. (but not SOTA)

--

JangHoon Han AI Research Engineer | Professional Language Lab | LG AI Research, Seoul, Korea Mobile : (+82)10-8591-1081 lgresearch.ai

-----Original Message----- From: @.> To: @.>; Cc: @.***>; Sent: 2022-05-13 (금) 11:28:57 (GMT+09:00) Subject: [hanjanghoon/BERT_FP] Is it normal for me to spend 1 hour per epoch of training? (Issue #6)

When I post-train with the ubuntu dataset, it takes 22 hours per epoch, 25 epochs means it takes close to a month, maybe the time is too long. Is this also the case for the author when training? Is this normal in my case? Thanks!

Iteration: 100%|██████████| 81865/81865 [22:53:38<00:00, 1.01s/it]

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

bott1eee commented 2 years ago

Hello! In my case it took 10 hours per epochs. but it different per gpu spec. so i recommand post-training a model about 5~10 times. because that's enough to improve the performance. (but not SOTA) … -- JangHoon Han AI Research Engineer | Professional Language Lab | LG AI Research, Seoul, Korea Mobile : (+82)10-8591-1081 lgresearch.ai -----Original Message----- From: @.> To: @.>; Cc: @.>; Sent: 2022-05-13 (금) 11:28:57 (GMT+09:00) Subject: [hanjanghoon/BERT_FP] Is it normal for me to spend 1 hour per epoch of training? (Issue #6) When I post-train with the ubuntu dataset, it takes 22 hours per epoch, 25 epochs means it takes close to a month, maybe the time is too long. Is this also the case for the author when training? Is this normal in my case? Thanks! Iteration: 100%|██████████| 81865/81865 [22:53:38<00:00, 1.01s/it] — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.>

Thanks for your reply!