How to save checkpoints

[n00mkrad]

Running this as we speak, and it seems to work.

However, there are no checkpoints after almost 3k iterations.

1) Will a checkpoint be saved if I stop the training with Ctrl+C?
2) How can I manually specify the checkpoint frequency?

[n00mkrad]

Okay, I found out that Ctrl+C does indeed save a checkpoint.

Please clarify this in the readme, I was really scared I might lose hours of training time.

[n00mkrad]

Closed for now though I still would like an option to save a checkpoint every x iterations.

[nicolai256]

Closed for now though I still would like an option to save a checkpoint every x iterations.

i'll look into it

[nicolai256]

it basically already does save checkpoints whenever you pass an epoch but i'll check if it's possible to save it every 500 iterations :)

[n00mkrad]

it basically already does save checkpoints whenever you pass an epoch but i'll check if it's possible to save it every 500 iterations :)

Yeah, only problem is that epochs can be very long depending on the dataset size.

[nicolai256]

yeah i have the same problem hopefully the original publisher will fix it