search

vye16 / slahmr

MIT License
441 stars 50 forks source link

Continue processing #34

Closed carlosedubarreto closed 5 months ago

carlosedubarreto commented 11 months ago

Is there a possibility to stop processing and continue later? I tried forcing the process to stop and them start again, but it usually ends up without cotinuing processing, it seems that it sees that there was something running before and just wraps up.

I'm asking that in case I need to stop the process and if I could conitnue later, or in a case that I get a 'nan" value, if I could change the learning rate and run again not loosing all the previous process that was done

Thanks

vye16 commented 11 months ago

There should be checkpointing during optimization, but I will check if there’s a bug. There are multiple stages of optimization so if it’s already finished one of them, it will just skip it and move on to the next one.

On Thu, Aug 3, 2023 at 3:22 PM Carlos Barreto @.***> wrote:

Is there a possibility to stop processing and continue later? I tried forcing the process to stop and them start again, but it usually ends up without cotinuing processing, it seems that it sees that there was something running before and just wraps up.

I'm asking that in case I need to stop the process and if I could conitnue later, or in a case that I get a 'nan" value, if I could change the learning rate and run again not loosing all the previous process that was done

Thanks

— Reply to this email directly, view it on GitHub https://github.com/vye16/slahmr/issues/34, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLOKW2QICU4I66ILBMIVNTXTQQBZANCNFSM6AAAAAA3DMVCRI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

carlosedubarreto commented 11 months ago

just to be sure. for example, it does 4 steps. first it will do the full step 1 them step 2 and so on?

I got the impression that it did in chunks. for exmaple, it splited the frames in "n" parts. it would do the step 1 to 4 in the frist part, then step 1 to 4 in part 2 and so on...

the way you are saying, if I removed the step the it was not finished when I stopped the execution it would continue from the previous step?

It could be enough, I'll study it more.

vye16 commented 11 months ago

Yes, in the last step of optimization, we optimize in chunks. We also have some optimization phases to help initialize the main motion_chunks phase of optimization. In any of these phases, if the optimization is interrupted, it will resume from the step it last saved a checkpoint at.

On Fri, Aug 4, 2023 at 1:57 AM Carlos Barreto @.***> wrote:

just to be sure. for example, it does 4 steps. first it will do the full step 1 them step 2 and so on?

I got the impression that it did in chunks. for exmaple, it splited the frames in "n" parts. it would do the step 1 to 4 in the frist part, then step 1 to 4 in part 2 and so on...

the way you are saying, if I removed the step the it was not finished when I stopped the execution it would continue from the previous step?

It could be enough, I'll study it more.

— Reply to this email directly, view it on GitHub https://github.com/vye16/slahmr/issues/34#issuecomment-1665272853, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLOKW6PFSUR2H6QHM65T7TXTS2QRANCNFSM6AAAAAA3DMVCRI . You are receiving this because you commented.Message ID: @.***>

carlosedubarreto commented 11 months ago

So if I remove the last chunk (probably not completed if I manually stopped the process and I had some sort of error), theorically it should work?

I'll give it a try anyway