[meta] Cost efficiency

[eu9ene]

Let's make sure the infrastructure settings are adjusted based on the current needs of the pipeline. There shouldn't be any waste but at the same time infrastructure shouldn't limit experimentation speed. We can also pursue some quick optimizations here.

### Optimization
- [ ] https://github.com/mozilla/firefox-translations-training/issues/455
- [ ] #419
- [ ] #414
- [ ] #394
- [ ] #415
- [ ] #459
- [ ] https://github.com/mozilla/firefox-translations-training/issues/663

### On-premises cluster
- [ ] https://github.com/mozilla/firefox-translations-training/issues/300
- [ ] https://github.com/mozilla/firefox-translations-training/issues/253
- [ ] https://github.com/mozilla/firefox-translations-training/issues/230
- [ ] https://github.com/mozilla/firefox-translations-training/issues/391

[gregtatum]

I don't know that I have a specific issue to tie this to, but here is a performance profile of all the en-ca training tasks: https://share.firefox.dev/49SotyG

[gregtatum]

And here is a visualization of the training run: https://gregtatum.github.io/taskcluster-tools/?taskGroupIds=GU9ZyWFhRDe_nxlAHcen8g&fetchDependentTasks=true&ignoredTaskGroupIds=aCUq1LgtQCeHkmk-wJhbtQ%2CcKY8O-guTcKXZXRYpWLGiw%2CdYEG7rEWS8esjEvDrzmtrA%2CXvaP5gPWSO68cWmub-moLw%2CO-49dfOARwumu49ZkmhhGQ