WordPress / openverse-frontend

The gateway to the Openverse. Openverse is a search tool for CC-licensed and public domain content across the internet.
https://wordpress.org/openverse
MIT License
71 stars 63 forks source link

Download translations in bulk to prevent GlotPress throttling #2188

Closed dhruvkb closed 1 year ago

dhruvkb commented 1 year ago

Fixes

Fixes #2184 by @dhruvkb

Description

This PR adds bulk downloading for translations by logging into GlotPress, downloading the ZIP and extracting the JSON content. It reuses the existing framework for massaging them from jed1x to Vue i18n's format and then writing them to the destination dir.

If GlotPress credentials are not present in the environment, the old throttled approach will be used as the fallback, a Make WordPress membership is not needed for development.

Other improvements

Testing Instructions

  1. Check out the PR.
  2. Run the following command:
    $ pnpm i18n

    You will see the parallelised download that might throttle.

  3. Clear out the translations files.
  4. Run the following command:
    $ env GLOTPRESS_USERNAME="<WP.org username>" GLOTPRESS_PASSWORD="<WP.org password>" pnpm i18n

    You will see the bulk download that does not throttle.

Checklist

[best_practices]: https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project#_commit_guidelines

Developer Certificate of Origin

Developer Certificate of Origin ``` Developer Certificate of Origin Version 1.1 Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 1 Letterman Drive Suite D4700 San Francisco, CA, 94129 Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. ```
github-actions[bot] commented 1 year ago

Storybook and Tailwind configuration previews: Ready

Storybook: https://wordpress.github.io/openverse-frontend/_preview/2188 Tailwind: https://wordpress.github.io/openverse-frontend/_preview/2188/tailwind

Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again.

You can check the GitHub pages deployment action list to see the current status of the deployments.

github-actions[bot] commented 1 year ago

Size Change: -25.6 kB (-3%)

Total Size: 882 kB

Filename Size Change
./.nuxt/dist/client/app.js 141 kB -10.4 kB (-7%)
./.nuxt/dist/client/app.modern.js 116 kB -8.91 kB (-7%)
./.nuxt/dist/client/commons/app.js 87.7 kB -79 B (0%)
./.nuxt/dist/client/commons/app.modern.js 78.1 kB -77 B (0%)
./.nuxt/dist/client/components/v-all-results-grid.js 7.46 kB -86 B (-1%)
./.nuxt/dist/client/components/v-all-results-grid.modern.js 4.96 kB -90 B (-2%)
./.nuxt/dist/client/components/v-audio-track.js 5.2 kB -75 B (-1%)
./.nuxt/dist/client/components/v-audio-track.modern.js 5.15 kB -72 B (-1%)
./.nuxt/dist/client/components/v-external-search-form.js 1.91 kB -156 B (-8%)
./.nuxt/dist/client/components/v-external-search-form.modern.js 1.9 kB -147 B (-7%)
./.nuxt/dist/client/components/v-full-layout.js 1.51 kB -77 B (-5%)
./.nuxt/dist/client/components/v-full-layout.modern.js 1.51 kB -75 B (-5%)
./.nuxt/dist/client/components/v-home-gallery.js 4.79 kB -14 B (0%)
./.nuxt/dist/client/components/v-image-carousel.js 4.73 kB -34 B (-1%)
./.nuxt/dist/client/components/v-image-carousel.modern.js 4.7 kB -24 B (-1%)
./.nuxt/dist/client/components/v-image-grid.js 4.85 kB -78 B (-2%)
./.nuxt/dist/client/components/v-image-grid.modern.js 2.39 kB -77 B (-3%)
./.nuxt/dist/client/components/v-old-homepage-content.js 0 B -1.88 kB (removed) 🏆
./.nuxt/dist/client/components/v-old-homepage-content.modern.js 0 B -1.85 kB (removed) 🏆
./.nuxt/dist/client/components/v-related-images.modern.js 2.95 kB -79 B (-3%)
./.nuxt/dist/client/components/v-search-grid.js 5.74 kB -134 B (-2%)
./.nuxt/dist/client/components/v-search-grid.modern.js 5.66 kB -141 B (-2%)
./.nuxt/dist/client/components/v-skip-to-content-container.js 911 B +24 B (+3%)
./.nuxt/dist/client/components/v-skip-to-content-container.modern.js 918 B +22 B (+2%)
./.nuxt/dist/client/components/v-sources-table.js 16.1 kB -512 B (-3%)
./.nuxt/dist/client/components/v-sources-table.modern.js 16.1 kB -506 B (-3%)
./.nuxt/dist/client/pages/about.js 1.49 kB -31 B (-2%)
./.nuxt/dist/client/pages/about.modern.js 1.49 kB -33 B (-2%)
./.nuxt/dist/client/pages/audio/_id/index.js 8.39 kB +388 B (+5%) 🔍
./.nuxt/dist/client/pages/audio/_id/index.modern.js 5.25 kB +381 B (+8%) 🔍
./.nuxt/dist/client/pages/external-sources.js 1.51 kB -30 B (-2%)
./.nuxt/dist/client/pages/external-sources.modern.js 1.51 kB -27 B (-2%)
./.nuxt/dist/client/pages/feedback.js 1.29 kB -29 B (-2%)
./.nuxt/dist/client/pages/feedback.modern.js 1.29 kB -29 B (-2%)
./.nuxt/dist/client/pages/image/_id/index.js 9.7 kB +382 B (+4%)
./.nuxt/dist/client/pages/image/_id/index.modern.js 7.7 kB +255 B (+3%)
./.nuxt/dist/client/pages/image/_id/report.js 4.22 kB +629 B (+18%) ⚠️
./.nuxt/dist/client/pages/image/_id/report.modern.js 4.67 kB +425 B (+10%) ⚠️
./.nuxt/dist/client/pages/index.js 6.85 kB -1.8 kB (-21%) 🎉
./.nuxt/dist/client/pages/index.modern.js 6.77 kB -1.75 kB (-20%) 🎉
./.nuxt/dist/client/pages/privacy.js 963 B -32 B (-3%)
./.nuxt/dist/client/pages/privacy.modern.js 966 B -29 B (-3%)
./.nuxt/dist/client/pages/search-help.js 1.56 kB -62 B (-4%)
./.nuxt/dist/client/pages/search-help.modern.js 1.53 kB -89 B (-6%)
./.nuxt/dist/client/pages/search.js 5.05 kB -126 B (-2%)
./.nuxt/dist/client/pages/search.modern.js 2.55 kB -123 B (-5%)
./.nuxt/dist/client/pages/search/audio.js 6.07 kB -56 B (-1%)
./.nuxt/dist/client/pages/search/audio.modern.js 3.58 kB -42 B (-1%)
./.nuxt/dist/client/pages/search/image.js 563 B -33 B (-6%)
./.nuxt/dist/client/pages/search/image.modern.js 2.63 kB -94 B (-3%)
./.nuxt/dist/client/pages/search/index.js 508 B +28 B (+6%) 🔍
./.nuxt/dist/client/pages/search/index.modern.js 515 B +30 B (+6%) 🔍
./.nuxt/dist/client/pages/sources.js 1.52 kB -34 B (-2%)
./.nuxt/dist/client/pages/sources.modern.js 1.52 kB -39 B (-2%)
./.nuxt/dist/client/vendors/app.js 63.8 kB +47 B (0%)
./.nuxt/dist/client/vendors/app.modern.js 63 kB -106 B (0%)
./.nuxt/dist/client/components/v-modal.js 991 B +991 B (new file) 🆕
./.nuxt/dist/client/components/v-modal.modern.js 981 B +981 B (new file) 🆕
ℹ️ View Unchanged | Filename | Size | Change | | :--- | :---: | :---: | | `./.nuxt/dist/client/237.js` | 273 B | 0 B | | `./.nuxt/dist/client/237.modern.js` | 277 B | 0 B | | `./.nuxt/dist/client/238.js` | 1.85 kB | 0 B | | `./.nuxt/dist/client/components/loading-icon.js` | 747 B | -1 B (0%) | | `./.nuxt/dist/client/components/loading-icon.modern.js` | 752 B | +3 B (0%) | | `./.nuxt/dist/client/components/table-sort-icon.js` | 509 B | +1 B (0%) | | `./.nuxt/dist/client/components/table-sort-icon.modern.js` | 513 B | 0 B | | `./.nuxt/dist/client/components/v-audio-cell.js` | 387 B | +3 B (+1%) | | `./.nuxt/dist/client/components/v-audio-cell.modern.js` | 392 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-audio-details.js` | 2.54 kB | +5 B (0%) | | `./.nuxt/dist/client/components/v-audio-details.modern.js` | 1.79 kB | 0 B | | `./.nuxt/dist/client/components/v-audio-track-skeleton.js` | 1.01 kB | -1 B (0%) | | `./.nuxt/dist/client/components/v-audio-track-skeleton.modern.js` | 1.01 kB | -3 B (0%) | | `./.nuxt/dist/client/components/v-back-to-search-results-link.js` | 539 B | +1 B (0%) | | `./.nuxt/dist/client/components/v-back-to-search-results-link.modern.js` | 542 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-bone.js` | 684 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-bone.modern.js` | 689 B | 0 B | | `./.nuxt/dist/client/components/v-box-layout.js` | 1.23 kB | -1 B (0%) | | `./.nuxt/dist/client/components/v-box-layout.modern.js` | 1.23 kB | 0 B | | `./.nuxt/dist/client/components/v-content-link.js` | 1.11 kB | 0 B | | `./.nuxt/dist/client/components/v-content-link.modern.js` | 1.09 kB | -3 B (0%) | | `./.nuxt/dist/client/components/v-content-page.js` | 467 B | +1 B (0%) | | `./.nuxt/dist/client/components/v-content-page.modern.js` | 470 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-content-report-button.js` | 779 B | +5 B (+1%) | | `./.nuxt/dist/client/components/v-content-report-button.modern.js` | 780 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-content-report-form.js` | 6.1 kB | -2 B (0%) | | `./.nuxt/dist/client/components/v-content-report-form.modern.js` | 3.58 kB | -3 B (0%) | | `./.nuxt/dist/client/components/v-content-report-popover.js` | 1.23 kB | +5 B (0%) | | `./.nuxt/dist/client/components/v-content-report-popover.modern.js` | 4.23 kB | -2 B (0%) | | `./.nuxt/dist/client/components/v-copy-button.js` | 3.99 kB | -1 B (0%) | | `./.nuxt/dist/client/components/v-copy-button.modern.js` | 4 kB | +4 B (0%) | | `./.nuxt/dist/client/components/v-copy-license.js` | 999 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-copy-license.modern.js` | 1 kB | 0 B | | `./.nuxt/dist/client/components/v-copy-license/components/v-error-image/components/v-media-reuse/components/v-search-grid/d219393b.js` | 9.88 kB | -2 B (0%) | | `./.nuxt/dist/client/components/v-copy-license/components/v-error-image/components/v-media-reuse/components/v-search-grid/d219393b.modern.js` | 9.86 kB | +4 B (0%) | | `./.nuxt/dist/client/components/v-dmca-notice.js` | 748 B | +4 B (+1%) | | `./.nuxt/dist/client/components/v-dmca-notice.modern.js` | 752 B | +1 B (0%) | | `./.nuxt/dist/client/components/v-error-image.js` | 1.69 kB | -2 B (0%) | | `./.nuxt/dist/client/components/v-error-image.modern.js` | 1.68 kB | +2 B (0%) | | `./.nuxt/dist/client/components/v-error-section.js` | 372 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-error-section.modern.js` | 375 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-external-source-list.js` | 898 B | -3 B (0%) | | `./.nuxt/dist/client/components/v-external-source-list.modern.js` | 902 B | +3 B (0%) | | `./.nuxt/dist/client/components/v-grid-skeleton.js` | 1.61 kB | -4 B (0%) | | `./.nuxt/dist/client/components/v-grid-skeleton.modern.js` | 1.62 kB | -4 B (0%) | | `./.nuxt/dist/client/components/v-home-gallery.modern.js` | 4.78 kB | +4 B (0%) | | `./.nuxt/dist/client/components/v-homepage-content.js` | 1.72 kB | +1 B (0%) | | `./.nuxt/dist/client/components/v-homepage-content.modern.js` | 1.69 kB | -1 B (0%) | | `./.nuxt/dist/client/components/v-image-cell-square.js` | 1.02 kB | 0 B | | `./.nuxt/dist/client/components/v-image-cell-square.modern.js` | 1.02 kB | 0 B | | `./.nuxt/dist/client/components/v-image-cell.js` | 1.45 kB | +1 B (0%) | | `./.nuxt/dist/client/components/v-image-cell.modern.js` | 1.45 kB | +2 B (0%) | | `./.nuxt/dist/client/components/v-image-details.js` | 2.15 kB | +6 B (0%) | | `./.nuxt/dist/client/components/v-image-details.modern.js` | 1.42 kB | 0 B | | `./.nuxt/dist/client/components/v-license-tab-panel.js` | 521 B | 0 B | | `./.nuxt/dist/client/components/v-license-tab-panel.modern.js` | 525 B | 0 B | | `./.nuxt/dist/client/components/v-load-more.js` | 3.16 kB | -2 B (0%) | | `./.nuxt/dist/client/components/v-load-more.modern.js` | 683 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-media-license.js` | 817 B | -2 B (0%) | | `./.nuxt/dist/client/components/v-media-license.modern.js` | 828 B | +1 B (0%) | | `./.nuxt/dist/client/components/v-media-reuse.js` | 1.62 kB | -1 B (0%) | | `./.nuxt/dist/client/components/v-media-reuse.modern.js` | 1.62 kB | +1 B (0%) | | `./.nuxt/dist/client/components/v-media-tag.js` | 429 B | +1 B (0%) | | `./.nuxt/dist/client/components/v-media-tag.modern.js` | 433 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-no-results.js` | 751 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-no-results.modern.js` | 750 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-radio.js` | 1.5 kB | -5 B (0%) | | `./.nuxt/dist/client/components/v-radio.modern.js` | 1.47 kB | -1 B (0%) | | `./.nuxt/dist/client/components/v-related-audio.js` | 1.25 kB | +1 B (0%) | | `./.nuxt/dist/client/components/v-related-audio.modern.js` | 1.25 kB | +4 B (0%) | | `./.nuxt/dist/client/components/v-related-images.js` | 1.05 kB | 0 B | | `./.nuxt/dist/client/components/v-report-desc-form.js` | 965 B | 0 B | | `./.nuxt/dist/client/components/v-report-desc-form.modern.js` | 966 B | -2 B (0%) | | `./.nuxt/dist/client/components/v-row-layout.js` | 1.7 kB | -2 B (0%) | | `./.nuxt/dist/client/components/v-row-layout.modern.js` | 1.71 kB | -2 B (0%) | | `./.nuxt/dist/client/components/v-scroll-button.js` | 813 B | 0 B | | `./.nuxt/dist/client/components/v-scroll-button.modern.js` | 818 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-search-results-title.js` | 595 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-search-results-title.modern.js` | 598 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-search-type-radio.js` | 796 B | +5 B (+1%) | | `./.nuxt/dist/client/components/v-search-type-radio.modern.js` | 768 B | +1 B (0%) | | `./.nuxt/dist/client/components/v-server-timeout.js` | 299 B | 0 B | | `./.nuxt/dist/client/components/v-server-timeout.modern.js` | 303 B | -1 B (0%) | | `./.nuxt/dist/client/components/v-sketch-fab-viewer.js` | 3.37 kB | -1 B (0%) | | `./.nuxt/dist/client/components/v-sketch-fab-viewer.modern.js` | 894 B | 0 B | | `./.nuxt/dist/client/components/v-snackbar.js` | 1.18 kB | 0 B | | `./.nuxt/dist/client/components/v-snackbar.modern.js` | 1.19 kB | +2 B (0%) | | `./.nuxt/dist/client/components/v-warning-suppressor.js` | 298 B | 0 B | | `./.nuxt/dist/client/components/v-warning-suppressor.modern.js` | 303 B | 0 B | | `./.nuxt/dist/client/pages/preferences.js` | 1.23 kB | +4 B (0%) | | `./.nuxt/dist/client/pages/preferences.modern.js` | 1.22 kB | -1 B (0%) | | `./.nuxt/dist/client/pages/search/model-3d.js` | 242 B | 0 B | | `./.nuxt/dist/client/pages/search/model-3d.modern.js` | 246 B | +1 B (0%) | | `./.nuxt/dist/client/pages/search/search-page.types.js` | 266 B | 0 B | | `./.nuxt/dist/client/pages/search/search-page.types.modern.js` | 271 B | 0 B | | `./.nuxt/dist/client/pages/search/video.js` | 240 B | 0 B | | `./.nuxt/dist/client/pages/search/video.modern.js` | 244 B | +1 B (0%) | | `./.nuxt/dist/client/runtime.js` | 2.74 kB | 0 B | | `./.nuxt/dist/client/runtime.modern.js` | 2.74 kB | +1 B (0%) |

compressed-size-action

dhruvkb commented 1 year ago

Success update:

With the right secrets in the actions, all 200 translations are being downloaded...

Screenshot 2023-02-15 at 6 45 43 PM

...and incorporated into the Docker image.

Screenshot 2023-02-15 at 6 45 10 PM
dhruvkb commented 1 year ago

@zackkrida I'm very bad with JS Promises, even getting this PR to this usable state was very difficult for me. If you can push those changes for improved logging or fallback, that'll be very helpful.

I'm considering refactoring these scripts (and maybe use TypeScript) when I have some free time because it's very difficult to work with Promise chains, at least for me if not anyone else.

zackkrida commented 1 year ago

@dhruvkb sure, I can make some suggestions later today.

dhruvkb commented 1 year ago

@zackkrida my grasp on Promises was holding me back, I replaced all of that with async/await in #2192. Will merge that into this PR if you agree with those changes.