search

gordicaleksa / Open-NLLB

Effort to open-source NLLB checkpoints.
MIT License
419 stars 37 forks source link

Hydra pickle issue in generate_multi.py #3

Open vienneraphael opened 1 year ago

vienneraphael commented 1 year ago

Figure out the pickle issue mentioned here: https://github.com/facebookresearch/fairseq/issues/5315

Conf file

conf.zip

Current workaround:

@hydra.main(config_path="conf", config_name="generate_multi_full")
def main(config: DictConfig) -> None:
    launcher = hydra.utils.instantiate(config.launcher)
    module = GenerateMultiModule(config)
    asyncio.run(module.run())
    # asyncio.run(tabulate(config))

and modify the run method of GenerateMultiModule class by prepending it with the following:

jobs = self.array()
        for iteration_value in jobs: