Integrate with Crowdin localization project

bvaughn commented 6 years ago

Migrated from facebook/react/issues/8063

Coordinate with docs translators via @ericnakagawa

This issue involves multiple steps:

[x] Integrate with Crowdin to download localized content
[ ] Generate localized URLs using said content
[ ] Automatically sync English content changes from GitHub to Crowdin (see this comment and this repo)
[ ] Maintain backwards compatibility for all current (non-localized, English) links
[ ] Add localization toggle to header
[ ] Update the "Edit this page" link to point to GitHub for English pages and Crowdin for other languages.

Integrate with Crowdin

This could be done via a custom Gatsby plug-in. We've already written a couple. (It would be nice to work with @KyleAMathews to share this plug-in back with Gatsby so that other users could benefit from it as well.)

If we run into any performance problems, @ericnakagawa has volunteered to work with Netlify and Crowdin to simplify or speed up this syncing process. Ideally this sync process will be fast enough to allow Netlify's PR previews to also be used to test localized builds of the site.

Misc thoughts and questions

In the case of non-localized content, Crowdin serves English as fallback.
How can site consume list of active locales? Is there a Crowdin API?
How does this impact urls (eg /docs/<file>, /docs/<lang>/<file>)?

KyleAMathews commented 6 years ago

Happy to help create a Gatsby source plugin for Crowdin! That'd be really powerful. And a lot of people would use it including sooner than later gatsbyjs.org

From a bit of research, it looks like it'd be pretty easy to pull data using their API:

https://support.crowdin.com/api/download/
Guide for creating a source plugin https://www.gatsbyjs.org/docs/create-source-plugin/ — this guide is a bit wordy right now. When someone wants to tackle this, I could take a pass at improving the guide + adding a mini-tutorial on creating source plugins.

I spent a bunch of time adding localization support to https://www.gatsbyjs.org/packages/gatsby-source-contentful/ as well so have some opinions on this. See the example site for Contentful + its source

bvaughn commented 6 years ago

Fantastic 😄 There's a good chance we'll end up collaborating on this soon then.

Daniel15 commented 6 years ago

Netlify also has a way of doing redirects based on Accept-Language header and GeoIP. For example, if someone from Italy (or someone with their browser or OS set to request Italian content) visits the home page and they don't have a language cookie already set, we can automatically redirect to the Italian version. We use this on the Yarn site - You can look at how Yarn does it for inspiration

How does this impact urls (eg /docs/, /docs//)?

I'd suggest /<lang>/ (eg. /<lang>/docs/<file>), this is consistent with other Facebook projects that use localization: https://yarnpkg.com/en/docs/getting-started https://flow.org/en/docs/getting-started/

If we run into any performance problems, @ericnakagawa has volunteered to work with Netlify and Crowdin to simplify or speed up this syncing process. Ideally this sync process will be fast enough to allow Netlify's PR previews to also be used to test localized builds of the site.

We have perf issues with the Yarn site, but a lot of it is because we're using Jekyll. Most of the build time is actually the Jekyll build time rather than CrowdIn. We had a prototype using Hugo that was much faster. I don't think CrowdIn takes a significant amount of time there.

From a bit of research, it looks like it'd be pretty easy to pull data using their API:

Would the React site actually need a custom API integration? It could be useful, but CrowdIn has a CLI tool (https://support.crowdin.com/cli-tool/) which we might be able to use with less effort. For Yarn we just run crowdin-cli download -b master to download all the localized files as part of the build (in that case, there's a bunch of Markdown files as well as some yml files for language strings). Then you just build the site like normal and everything just works, as long as the files are in the correct locations.

bvaughn commented 6 years ago

How does this impact urls (eg /docs/, /docs//)?

I'd suggest // (eg. //docs/), this is consistent with other Facebook projects that use localization:

Don't have time to fully digest this response now- but wanted to mention that we should ensure whatever change we make does not break pre-existing links (like how links broke for Jest).

KyleAMathews commented 6 years ago

Would the React site actually need a custom API integration

It doesn't need it but it seems simpler perhaps to just query the data through graphql instead of writing it out first then pulling it in.

But yeah, definitely more straightforward to just use the cli.

Would you checkin the translated content?

Daniel15 commented 6 years ago

but wanted to mention that we should ensure whatever change we make does not break pre-existing links

Jest is limited in what it can do, because it's using GitHub Pages. With Netlify, you can generate some redirects as part of the build. Just need to redirect "raw" URLs (/docs/foo) to the localized URL (/en/docs/foo), either with /en/ hard-coded or using Netlify's language redirects that I mentioned earlier (so it'd redirect /docs/foo to /<user's preferred language>/docs/foo).

Check it out with Yarn: https://yarnpkg.com/docs/getting-started redirects to the right place. :)

Would you checkin the translated content?

We don't for Yarn. It's pulled in as part of the build process for the site. Since CrowdIn tracks the history for the translations, we didn't really need to also have Git track it.

Some sites do commit the translated files though. Up to you - Either way is fine.

bvaughn commented 6 years ago

Would you checkin the translated content?

No

bvaughn commented 6 years ago

or using Netlify's language redirects that I mentioned earlier (so it'd redirect /docs/foo to /<user's preferred language>/docs/foo).

Cool! 👍

Daniel15 commented 6 years ago

Here's how the Yarn site builds all the redirects from non-localized URLs to localized URLs: https://github.com/yarnpkg/website/blob/master/_redirects#L48-L70 (this would also be how you handle your redirects from 'old' URLs to new localized URLs). Unfortunately the Yarn one is super difficult to understand because it's using the Jekyll templating syntax. Somehow @thejameskyle got it working though. You could likely make a nicer JS script to do the same thing for the React site.

The resulting redirects file looks something like this:

/docs/cli/add/  /en/docs/cli/add/  302  Language=en
/docs/cli/add/  /fr/docs/cli/add/  302  Language=fr
/docs/cli/add/  /pt-BR/docs/cli/add/  302  Language=pt-br
/docs/cli/add/  /zh-Hans/docs/cli/add/  302  Language=zh-cn

Here's the entire language redirects section: https://gist.github.com/Daniel15/d84b39e8a260ee2a7d627751d015181b

Docs: https://www.netlify.com/docs/redirects/#geoip-and-language-based-redirects

Might be able to use a regex or placeholders or something rather than having to generate all the redirects like that.

KyleAMathews commented 6 years ago

@bvaughn wrote a nice redirect API for Gatsby + added it to gatsby-plugin-netlify so this setting up redirects would be pretty easy :-)

smikitky commented 6 years ago

Hi, the official doc in English is being updated frequently, but the contents in Crowdin seems to be out of sync with the newest doc. I think it's still based on the doc several months ago, and I'm seeing no doc for error boundaries ready for translation.

I'm new to Crowdin, but I want to believe there is a way to merge the updates in the original doc and enable people to translate the changed part incrementally. Is there any plan for this? I'd rather not see an overwhelming one-time update of the original doc.

bvaughn commented 6 years ago

Crowdin docs show support in their REST API to add, update, and delete a file- so it seems plausible that we could write a script to automatically sync english content from GitHub to Crowdin. Maybe we could even use CircleCI to run this automatically anytime there are commits to the master branch?

cc @ericnakagawa who has been leading the localization effort until now.

Daniel15 commented 6 years ago

I think we already have something configured with Yarn to automate it - Crowdin is automatically updated when changes are committed to the English versions. I'm not quite sure how it works though.

bvaughn commented 6 years ago

Thanks Daniel. I added a note to the description of this issue to check what Yarn's doing for this.

Daniel15 commented 6 years ago

It seems like Crowdin have some sort of Github integration now. I don't think Yarn is using this though: https://support.crowdin.com/github-integration/.

James Kyle set up the Yarn one, it might be worth reaching out to him.

jamiebuilds commented 6 years ago

Hi hi hi, yes I use https://github.com/thejameskyle/crowdin-sync for Yarn

clemmy commented 6 years ago

I'm interested in helping out with this as well. 🙂 @KyleAMathews @Daniel15 Do you guys have a preference on whether we do this through Github integration or just in time (during the build)?

Daniel15 commented 6 years ago

I don't think it really makes sense to commit translations to the repo, since CrowdIn already tracks changes to them. We should just have a single source of truth. Either way is fine though, I don't have a strong opinion about it.

bvaughn commented 6 years ago

I think we have a couple of options that may be worth exploring. I have been leaning toward using the Crowdin API to pull data at build-time. If this turns out to be slow (either the API or the resulting Gatsby build time) then we can provide a mode to disable/bypass it for local dev use, or to only pull a subset of languages, etc.

Daniel15 commented 6 years ago

For what it's worth, on the Yarn site we only pull down localizations in the prod build.

bvaughn commented 6 years ago

Agreed that we shouldn't do them for yarn dev but- depending on the speed- it might also be nice to disable for yarn build. (It's often useful to test the "production" build locally too.)

bvaughn commented 6 years ago

Here are some thoughts I had about creating localized links that are also backwards-compatible:

Assumption: The path to markdown files synced from Crowdin will contain a language code (eg <lang>/path/to/file.md).

This language code would enable us to easily generate a unique URLs for a given locale (eg ${language}/${slug}). However we will probably also want to handle the case where there is no language code (eg yarn dev local mode).

We could decorate Gatby's createPage action creator, parse this language code from the slug, and if it's English- also automatically create a redirect (using the createRedirect action creator) to a non-localized, backwards compatible URL.

The following Gatsby plug-ins may also be worth checking out:

I'm not sure how tested this plug-in is or how well it would work for us. However it might be worth looking at its implementation for ideas. For example, this plug-in seems to use Gatsby's onCreateNode to create node fields to associate slugs and language codes. They also use the onCreatePage API to just-in-time add language code info to pages.

bvaughn commented 6 years ago

Realized another issue we'll need to solve while iterating on this today: How we should make links defined in Markdown aware of their locales (eg not using a root "/path/to/file")?

The only ways I can think of are:

Require our translators to add a language code prefix before each link (which sucks)
Write our own Remark transform to prepend the language code based on the location of the markdown file.

I think option 2 sounds more likely to work but it also sounds like a lot of effort. Will need to research and think about this more.

This also leaves the question about how we handle Gatsby Links (like in the navigation header). Perhaps we could use a sub-class of Link that also uses withPrefix to prepend the language code based on a property or context variable?

Daniel15 commented 6 years ago

You shouldn't create redirects only for English, you should use Netlify's locale-based redirects too (so if someone's browser or OS is set to Italian and a page has an Italian translation, the local-less URL redirects to the Italian page. This means you'll generate one redirect per page per locale, and one extra one for English as default (if the page hasn't been translated to a language yet).

For locale aware links, do you just mean links on the site? Just use relative links everywhere :) I think that's what we do on the Yarn site.

bvaughn commented 6 years ago

Hey @Daniel15!

If we could use Netlify somehow to do a locale-friendly redirect for old links, that's great. I didn't know about that. (I don't see it mentioned in the Netlify docs for redirects.)

Anyway, Gatbsy's createRedirect method just outputs to a redirect file in the format Netlify uses, so what you're proposing is hopefully not too different from what I mentioned- assuming I can figure out how to do it, and assuming Netlify's language codes are in the same format Crowdin gives us when we create Gatsby URLs (eg "en" vs "en-US"). Then that would work for people visiting Gatsby from a referrer page.

Once they're there, we'll need to make sure links within the site work (eg ones that are defined in Markdown as well as pages using the Gatsby Link component).

For locale aware links, do you just mean links on the site? Just use relative links everywhere :) I think that's what we do on the Yarn site.

The Yarn website and this one are significantly different, so what works for one doesn't necessarily apply to the other. :smile: This is particularly true with markdown links. Gatsby+ReactRouter don't play nicely with relative links.

Let's chat via Messenger about this more if you'd like.

bvaughn commented 6 years ago

Quick update: I wrote a remark transform last night that handles one part of this:

module.exports = (
  {markdownAST, markdownNode, getNode},
  {defaultLanguageCode, defaultSourceName, translationsSourceName},
) => {
  const parentNode = getNode(markdownNode.parent);
  const {relativePath, sourceInstanceName} = parentNode;

  let languageCode = null;

  switch (sourceInstanceName) {
    case defaultSourceName:
      languageCode = defaultLanguageCode;
      break;
    case translationsSourceName:
      languageCode = getLanguageCodeFromPath(relativePath);
      break;
  }

  if (languageCode !== null) {
    // Prepand language code before links with absolute URLs,
    // eg '/path/to/file.html' => '/en/path/to/file.js'
    visit(markdownAST, `link`, node => {
      if (node.url.startsWith('/')) {
        node.url = `/${languageCode}${node.url}`;
      }
    });
  }
};

Some quick testing locally seems to suggest this works well enough. Next up is a decorator for Link.

Daniel15 commented 6 years ago

If we could use Netlify somehow to do a locale-friendly redirect for old links, that's great. I didn't know about that.

I mentioned it in an earlier comment: https://github.com/reactjs/reactjs.org/issues/82#issuecomment-337721594

(I don't see it mentioned in the Netlify docs for redirects.)

It's on the page you linked to, under the "GeoIP and Language-based redirects" heading. 😛

Here's what the language redirects look like for the Yarn site, as an example: https://gist.github.com/Daniel15/d84b39e8a260ee2a7d627751d015181b

bvaughn commented 6 years ago

I guess I misunderstood the section of the docs on the page I linked to. The comment you linked to is a month old. I guess I forgot it. /shrug

Thanks for linking to the Yarn redirects file. The Gatsby redirects plug-in will need an update to support the additional language parameter but that should be pretty straight forward.

The Link component will still need the ability to detect and prepend the language code itself (during the build phase) because of how Gatsby generates and optimizes its content graph.

bvaughn commented 6 years ago

PR to add support to Gatsby + gatsby-plugin-netlify for language and country based redirects: gatsbyjs/gatsby/pull/2890

bvaughn commented 6 years ago

May be worth noting, explicitly, that we don't plan to support localization for pages like the "Blog" landing page. (There's no plans to translate blog post content.) If we did decide to support that in the future, we would need to fork the page it somehow so that its GraphQL query only pulled in localized content.

techhtml commented 6 years ago

Hi there, I have a question about localization. When I end translating Document into Korean, How can I see that in React site?

bvaughn commented 6 years ago

That's not currently possible, @techhtml. Once this issue is done, it will be.

techhtml commented 6 years ago

@bvaughn Thank you! i'll wait that work done.

techhtml commented 6 years ago

@ericnakagawa Hi Eric, I have question about crowdin localization. I translate static-type-checking docs into Korean. but crowdin's document are not updated that only have old one. How can I update this docs?

clemmy commented 6 years ago

Hey @techhtml, I believe your translations need to be reviewed and approved by an admin of the React project on Crowdin before your changes appear.

techhtml commented 6 years ago

@clemmy of course. Other Korean developers will help reviews my translations.

smikitky commented 6 years ago

Hi, I logged in to Crowdin after a while, and found the "translated" percentage of Japanese has dropped to 65%. It was 100% (i.e., translation completed) in September last year, but no one but translators themselves had a chance to read the translated docs :)

So can someone set up a timeline and a guideline for this on which we can rely on? You have already gathered dozens of volunteers at Crowdin even though you have not officially called for volunteers. Has the localization project started officially, in the first place? Are you going to call for more collaborators, in the blog for example? When are we expected to finish the translation? I don't think the entire Crowdin project may be abandoned, but since there is no timeline nor document to guide the translators, I don't know when we can restart translating to keep up with the current doc. I don't even know what those "master" and "test-17" folders mean. (BTW, I just found the translation percentage for the "master" folder is now 1%... Does this mean almost all the effort we already made on the "old" master is going to disappear soon?)

cc @ericnakagawa

smikitky commented 5 years ago

Can we expect some progress before the release of React hooks? For now, articles related to hooks are not even listed in the Crowdin repository, and it appears to me that this effort has been half-abandoned recently. It's sad to see developers in my country have to depend on fairly insufficient personal blog articles when such a big change is approaching to the ecosystem.

By the way, does anyone know why materials in Crowdin are in HTML format rather than Markdown? HTML tags, especially all those <code>s, are very cumbersome to handle.

ping @bvaughn @ericnakagawa

bvaughn commented 5 years ago

Sorry, I don't have any updates/estimates for this. The core team is spread very thin and I don't think anyone owns this effort at the moment 😞

smikitky commented 5 years ago

I see. Meanwhile, is it possible to allow translators to download/export their existing contributions from Crowdin? If I understand correctly, this can be done by changing the project settings. Then each translator may host the translated version elsewhere on the internet, say, my GH pages.

bvaughn commented 5 years ago

That's something I think @ericnakagawa would have to manage. I don't know much about our Crowdin setup. Not even sure if I would have the necessary permissions.

gaearon commented 5 years ago

To give an update — if everything works out, we expect a person to jump on finishing the integration in one of the next weeks. Can't promise to get it done before Hooks are stable though but it's on our radar.

smikitky commented 5 years ago

For the record, here are some problems I'm currently seeing as a translator.

Translations are not Markdown-based but HTML-based. I believe all translated materials should be version-controlled.
There are several "root" directories on the Crowdin repo, and it is not clear which to work on.
New commits to the original doc are not automatically synched. For example there are no articles about Hooks yet in the Crowdin repository.
Some elements in the front matter (document IDs, etc) are wrongly marked as translatable.
Some code examples are marked as translatable and some are not.
This may be specific to CJK languages, but italics are not always rendered properly for those languages, so we need a different CSS styles for <em>'s.

Daniel15 commented 5 years ago

Translations are not Markdown-based but HTML-based.

For Yarn we use Markdown in CrowdIn (at https://i18n.yarnpkg.com/), so this might just be an issue with how it's configured for React. Should be straightforward to fix it 😃

tesseralis commented 5 years ago

@smikitky We've come up with an alternate approach here: https://github.com/reactjs/reactjs.org/issues/1605

Does this resolve any of your issues working with Crowdin, and would you be interested in working with us on maintaining the Japanese fork? We should be able to use the work that you and others have done so far on Crowdin as a baseline.

gaearon commented 5 years ago

https://reactjs.org/blog/2019/02/23/is-react-translated-yet.html

reactjs / react.dev

Integrate with Crowdin localization project #82

Integrate with Crowdin

Misc thoughts and questions