[Machine Translation] Modify script to add files to be translated to the queue

[moonlight-komorebi]

Summary

We will need to update the scripts for enabling Machine Translation (MT). The way we will be interacting with the Smartling API is identical to how we do it now, we just need some tweaks to the existing scripts along with some additional functionality.

<!> We should test changes to the scripts using the Test RDS Environment updated here #1538

Project Logic

In add-files-to-translation-queue.js we will need to pick up both project_id's as environment variables (these will eventually be read up from the updated workflow).

The relevant project_id should be added to the new column in the translations table. In the workflow both MACHINE_TRANSLATION_PROJECT_ID and HUMAN_TRANSLATION_PROJECT_ID will be set. The logic below will help determine which ID should be entered in to the tables:

>a value or set of values for the translate frontmatter field indicates which languages are sent for human translation. the difference between the set of all languages we support, and the set in the translate frontmatter, is what we request for machine translation.

For example, in the mdx frontmatter of a file, if we have...

    translate
      - jp

...and the defined set of languages we support is [jp, kr], then:

• jp is Human Translated

• kr is Machine Translated

  If there is no translate key in the frontmatter, it should be translated into all available locales.

🛑 Testing Scripts

You can use the MT Project ID for this but only test with a 1 word change, [see here](https://github.com/newrelic/docs-website/blob/develop/scripts/actions/translation_workflow/testing/README.md#make-a-change-to-translate) The reason for this is we have a `2 million` word limit per year specifically for Machine Translation >Average is about 850 words per document. total is about 1.6 million. For MT it’s a total of 2 million words can be translated over a year. we have approximately 1800 pages x 850 (avg word count per page) = 1.6 million

Acceptance Criteria

• add-files-to-translation-queue script should:
  • [x] Pick up both project_id's as environment variables
  • [x] Be able to determine if a page needs to be Machine/Human Translated and to which languages
  • [x] Add relevant project_id to translations table entries

Useful links

• Instructions for Testing
• html in smartling, placeholder directives

Relates to: #2341

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be automatically closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it was a stale issue that had no recent activity. Thank you for your contributions.

[jmiraNR]

Hi. Regarding this point: making sure protected text is ignored from translation. implementing placeholder directives if our current process doesn't work. Clarification: the text to be protect is within a sentence. text like "New Relic", or "Add more data" when referring to a button: Please click Add more data. On these pages:

• https://help.smartling.com/hc/en-us/articles/360007894614-HTML
• https://help.smartling.com/hc/en-us/articles/360012489774-Markdown

there is some information about the directives. I tested PLACEHOLDER_FORMAT_CUSTOM on an HTML to protect some random text and it worked. I was able to put some of our keywords between {} and it was not translated. I'd appreciate discussing this topic in more details when we start to work on it as I want to make sure we can automate/simplify this process as much as possible. Thx

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be automatically closed if no further activity occurs. Thank you for your contributions.

[jpvajda]

We want to spend more time on specifying approach and consider breaking this up into seperate tickets.

[jpvajda]

We discussed breaking this ticket up differently...

[jpvajda]

@rudouglas

[moonlight-komorebi]

i think we should break this up, just for clarity and organizing the complexity and different pieces. We estimated this at 5-8, i dont think splitting the ticket changes the overall effort, but is just to have a better mental map of what needs to get done and how things relate.

This is just my opinion, so don't feel obligated:

ticket 1 - add translations workflow

• create file exclusion logic (not content exclusion), apply it in this workflow so only non-excluded files are queued for translation
  • maybe there is another ticket to describe what this implementation is, but this ticket should call that implementation.
• add project id to translation table rows (when we created translation records). the id added is determined by whether a doc should be human translated or machine translated.
• anything else

ticket 2 - send translations workflow

• fetch files from translation table -- with new project_id added by previous workflow
• add project id to jobs table rows
• add content exclusion.
  • maybe there is another ticket to describe what this implementation is, but this ticket should call that implementation.
• create jobs, batches in the correct project, send to the correct project
• anything else