search

CUNY-CL / yoyodyne

Small-vocabulary sequence-to-sequence generation with optional feature conditioning
Apache License 2.0
25 stars 15 forks source link

adding expert to hp ignore due to pickeling issue #197

Open bonham79 opened 2 weeks ago

bonham79 commented 2 weeks ago

Fixes: https://github.com/CUNY-CL/yoyodyne/issues/192

TQDM in the transducer's expert module doesn't pickle currently. So when doing multi-gpu training, pytorch gets annoyed. Since it's not really needed beyond training, I'm dropping it from the model.

Note, when running prediction with the model, the lack of the hyperparameter means a new expert will have to be created. This is dumb, so need to add a dummy class to the transducer model to avoid this. This portion I'm leaving in draft atm.

Adamits commented 1 week ago

Multiprocess + pickle has given me so many issues recently across interdependent libraries...

One edge case: how about continued training from a checkpoint? Then we need the expert right, which I assume has some relevant state?

kylebgorman commented 1 week ago

@Adamits I think this would pose exactly the same issue as the multi-GPU case.

Adamits commented 1 week ago

@kylebgorman Right, I meant an edge case where this solution would break something. If I understand correctly, Travis wants to just not save the expert state to avoid the pickle issue with TQDM. But if we want to continue form a checkpoint, i think we need the expert state.

tbartley94 commented 1 week ago

One edge case: how about continued training from a checkpoint? Then we need the expert right, which I assume has some relevant state?

you'd have to train the expert again. thankfully your np fix makes it less annoying. but yeah, that's why i was curious if there were better ideas.

fwiw, the entire point behind the expert transducer is for single gpu setup. (Actually it works better on CPU for most cases due to the expert queries). You don't get really parallelism speed ups since there's so many gpu-> cpu transfers.

kylebgorman commented 1 week ago

Yeah I also find the transducer is better on CPU. (The same was true with the original code in DyNet.)

bonham79 commented 1 week ago

Yeah I also find the transducer is better on CPU. (The same was true with the original code in DyNet.)

Yeah it's the main motivation for its use so not surprised.