The issue with passing the target language this way is that we would have to reload the pipeline/model when we have to change the target language, which would result in a lot of unnecessary overhead.
To Do:
To avoid the overhead with reloading and to effectively showcase the capabilities of multilingual models, we would like to be able to pass in the target language to the models dynamically.
Refactor the current target language passing mechanism to make it dynamic.
Hint: passing it through the input segment and then through the agent states could work.
Additionally, we would also like to allow specifying multiple target languages (as opposed to just one shown in dummy model), this would eventually help us to design pipelines for simultaneously translating to multiple languages and for getting ASR output.
There are 2 parts to this issue:
Changes in SimulEval repo to enable passing in the target language as part of the instances/segments to the pipeline and then passing it through the agents in the pipelines through their states.
Change in Seamless-Experiences repo to hook up the SimulEval pipeline with the changes mentioned above into the demo.
After running the pipeline, you should be able to see a instances.log file under the output folder. You should be able to see the output of the pipeline there as "predictions". Try changing the tgt-lang and observe how the output changes accordingly.
Using this knowledge, we would like you to make changes so that tgt_lang can be inferred from the source dataset and passed along via the input segment and states ultimately to the agent that uses it.
Context:
Currently when loading pipelines for multilingual models we use the runtime cli param/option to set the target language for the translation output.
This is emulated in the dummy model that was recently added: https://github.com/facebookresearch/SimulEval/blob/main/examples/speech_to_text/counter_in_tgt_lang_agent.py#L22-L24 When used in the demo in its current state,
tgt_langneeds to be set invad_main.yamland loaded from there when building the model.The issue with passing the target language this way is that we would have to reload the pipeline/model when we have to change the target language, which would result in a lot of unnecessary overhead.
To Do:
To avoid the overhead with reloading and to effectively showcase the capabilities of multilingual models, we would like to be able to pass in the target language to the models dynamically. Refactor the current target language passing mechanism to make it dynamic.
Hint: passing it through the input segment and then through the agent states could work.
Additionally, we would also like to allow specifying multiple target languages (as opposed to just one shown in dummy model), this would eventually help us to design pipelines for simultaneously translating to multiple languages and for getting ASR output.
There are 2 parts to this issue:
Hints/Pointers:
cd SimulEval/examples/speech_to_textsimuleval --agent counter_in_tgt_lang_agent.py --user-dir . --agent-class agents.CounterInTargetLanguageAgent --source-segment-size 1000 --source source.txt --target reference/en.txt --output <path to output folder> --tgt-lang esinstances.logfile under the output folder. You should be able to see the output of the pipeline there as "predictions". Try changing thetgt-langand observe how the output changes accordingly.tgt_langcan be inferred from the source dataset and passed along via the input segment and states ultimately to the agent that uses it.