Closed bryandam closed 11 months ago
Ok, I figured out what was going on with the snapshot restore and have a PR in to fix it: #16
I also think I understand the perf issue thing in that I don't think there's an actual perf issue. The train() method passes in 100,000 for minnum_training_steps and on my RTX3080 I was averaging around 8.5s per step. However, if you look at the model that's included in the code it apparently only took 17,900 steps based on the checkpoint files. That is to say, it shouldn't take 100k steps to train the model.
So I've been trying to train my own model in the possibly vain hope of fixing the Q, X, and Z characters. I figure I've got a RTX 3080 so why not give it a shot? I've bought a Beam system off eBay in case I need to flesh out the dataset but I figured step one was to just go through and make sure I could successfully do a training run to replicate the current behavior.
At first I figured I should clear out the existing model by clearing out the model/checkpoint folder. That went fine until around set 1520:
It struck me as odd that you'd need the existing model to create a new model as that seems like a catch-22 but ... ok ... the instructions didn't say to clear that directory so I restored the files there and tried again. However, I was met with a very similar error:
This second time it's acting as if the BaseModel.Restore() function received a step (Step 20) as it's calculating a folder caled 'model-20' but that folder doesn't exist.
Near as I can tell then, it feels like BaseModel.Restore() is being called before BaseModel.Save() is called. Any thoughts @otuva?
Lastly, any guidance on real-world training performance? It supposedly took a couple of days on the Tesla K80 which, in theory, my RTX 3080 should beat by a fair margin. However, based on how long it took to do those 1540 steps I'm looking at 8+ days here. I'm running it via Windows 11 WSL on Ubuntu 22.04.2 LTS and the GPU is being used:
I wonder if WSL presents some kind of bottleneck and I'll need to dual-boot to get better perf. Anyone have experience to share here?