Closed xiaoyudanaa closed 6 months ago
When the training ends automatically, I get the following weights file:: epoch=0-step=1000000.ckpt
Same problem here. I tried to train on a single A100 and got a epoch=0-step=1000000.ckpt
and a hamer_last.ckpt
then load the epoch=0-step=1000000.ckpt
but it's not working at all (always fluctuating). Moreover, the loss value seems not to converge in the log file and the epoch number is also 0.
Here are examples of using official checkpoints and checkpoints of mine:
(using
epoch=0-step=1000000.ckpt
)
(using the officially released model
hamer.ckpt
)
Here's the tensorboard log:
Same problem here. I tried to train on a single A100 and got a
epoch=0-step=1000000.ckpt
and ahamer_last.ckpt
then load theepoch=0-step=1000000.ckpt
but it's not working at all (always fluctuating). Moreover, the loss value seems not to converge in the log file and the epoch number is also 0.Here are examples of using official checkpoints and checkpoints of mine:
(using
epoch=0-step=1000000.ckpt
)(using the officially released model
hamer.ckpt
)Here's the tensorboard log:
Hi, can I have your contact details? Let's discuss this together, my WeChat is: Weafree
Same problem here. I tried to train on a single A100 and got a
epoch=0-step=1000000.ckpt
and ahamer_last.ckpt
then load theepoch=0-step=1000000.ckpt
but it's not working at all (always fluctuating). Moreover, the loss value seems not to converge in the log file and the epoch number is also 0. Here are examples of using official checkpoints and checkpoints of mine:(using
epoch=0-step=1000000.ckpt
)(using the officially released model
hamer.ckpt
) Here's the tensorboard log:Hi, can I have your contact details? Let's discuss this together, my WeChat is: Weafree
Sure I have added.
Replied at issue #24. Could you pull the latest changes and try again?
Dear author, I'm a student of Yebin Liu at Tsinghua University, I have a problem: I have trained the hamer on 8*A800 GPU as the original configuration, but the epoch is still 0 until I get to 1,000,000 steps and after I run the demo, there is a mismatch in the visualisation results, what can I do to fix it? Meanwhile, could you share the pre-processing steps for tar? Looking forward to your reply.