Open jhildenbiddle opened 3 months ago
Thanks for bringing this issue up @jhildenbiddle. As you've identified, up-to-date translation itself is a major on-going effort to create and then maintain - for example, another open source project I have been involved with is https://getgrav.org has a dedicated team of volunteer contributors using Crowdin https://crowdin.com/profile/getgrav.
The out-of-date docs is a major user experience issue in itself. Based on what I see, perhaps option 1 is a beneficial path to take for the initial release of v5 and then re-assess viability of either option 2 or 3? Interested to hear other perspectives.
Browser translation is pretty good I think, especially after many years of improvements.
Does having pre-translated docs help with SEO? For example, do search engines show users in a nother country results for english sites in their language? Or do websites need to have content in the target country's language to become visible there?
If the answer is that pre-translation would be needed for this, I'd say maybe that's the single most reason to do it. But then the question would be how? And another question is, will keeping the current translations even if outdated help with this SEO?
I think we can keep the remained documentation translations, and add the last update date
banner with docsify-top-banner-plugin. i.e
This label could be automate update, when new commits into.
Current site last update date is [xxxx-xx-xx] if you don't find the things you want, try find in latest [EN] site.
Browser translation is always available, every users can use it in our default update-to-date EN docs, if users can not find the target part in their prefer langs, they can go to the EN docs.
@trusktr mentioned the SEO thing is reasonable to me, and I think the Browser translation
still exist the Gap
for the natives.
Our goal should be to provide accurate, useful documentation to Docsify users. If our translations don't do that, then they aren't serving their intended purpose and we should remove them for the benefit of our users and our project maintainers. It should be that simple.
I appreciate the desire to show translations on the site, but the reality is that offering localized content that serves users requires a significant upfront and continued investment that our team simply does not have the capacity to make. Pretending this is not the case by publishing inaccurate and/or outdated documentation that hasn't been updated since 2020 is a disservice to the site visitors that read those translations and, IMHO, reflects poorly on both the project and its maintainers.
As for SEO, I don't think there is any reason to believe our existing translations impact SEO primarily because docsify.js.org serves pages (including all translations) in router: hash
mode which we know generates URLs that are not indexed reliably (if at all) by search engines.
Fortunately, we don't have to speculate about SEO. Our site is configured with Google Analytics so we can see exactly where our traffic is coming from and how many visitors are actually reading out current translations. Assuming that has been capturing data properly, we just need to log in and review the data.
All of this is to say my preference is to remove non-EN translations until we are able to create a plan for if/how we want to support localized documentation and which locals we will support.
Yea, I agree that the out of date docs maybe a disservice when we do have lots of breaking changes. The reality is, those sites' most part of the basic configuration/usage works . So I don't think we need remove it directly.
On the other side, I don't think the exist contents
in search engine from other ref-links sites which doesn't links to non-EN site path should be invalid directly either.
Alternatively, we move to only keep the EN (and CN? @sy-records ) site, we needs some redirect solutions for those non-EN/CN paths directly to the EN
contents (they have the same paths and structures).
Hi, I just landed here by accident and would like to add some of my thoughts.
I think the main problem with the current setup is that the sources and localization files are in separate repositories. So it is a lot of work for a potential contributor to suggest new translations, especially if the sources are not synchronized with the translations (they have to go to the main repo, take the current source files, compare them to the translations from the separate repo, and then suggest new translations).
The community translators can work together in Crowdin, for example, translations would be delivered as PR, and volunteers can raise issues for problematic strings via Crowdin. In addition, everything could be automated via CI/CD (pushing sources to Crowdin, downloading translations).
You can also check out how Docusaurus manages i18n for their website.
I believe that a localized website can attract more users who are not fluent in English. Providing content in their native language makes the site more accessible and user-friendly, and encourages greater engagement from a diverse audience.
By the way, Crowdin is free for OSS. If you decide to improve the 18n flow and update the translation status, I can help with automated integration with Crowdin.
I'll update CN's as they are released.
None of the @docsifyjs/translators team members seem to be active anymore.
I think we can try Crowdin.
Hi @andrii-bodnar , thx for your suggestion. I have a quick shoot on Crowdin. It seems use something like the fine-tuning AI translator or 3rd translation providers to do the job. (correct me if I do misunderstanding something). And I don't much clear understand about the sample you mentioned in Docusaurus.
So, before we consider the efforts about moving to Crowdin.
Could you plz provide more info about What we need prepare before we use the Crowdin :
Hi @Koooooo-7! Currently, Crowdin supports the following AI providers:
So it's possible to use them to pre-translate existing or new content. In addition, community contributors can also suggest their own translations.
And I don't much clear understand about the sample you mentioned in Docusaurus.
This was done to provide an example of how it can be implemented (it is an interesting approach, but more complex to set up).
Do we need have one AI providers key? or order the 3rd translation provider?
You can use your own key. Visit the Crowdin AI article for more details.
Based on our project current i18n things, what the structures of the i18n changes we may need to do?
I'd suggest keeping the source files and translations within the current repository (use some subdirectories for the translated documents).
For example:
.
โโโ docs/locales
โ โโโ en
โ โ โโโ ...
โ โโโ es
โ โ โโโ ...
โ โโโ de
โ โ โโโ ...
โโโ ...
Crowdin supports markdown, so it shouldn't change much for us.
Hi @andrii-bodnar
So it's possible to use them to pre-translate existing or new content. In addition, community contributors can also suggest their own translations.
IIUC, it means that we could choose use own API key, or using the Crowdin's solution directly for free (if we don't do the pre-train stuff)?
I'd suggest keeping the source files and translations within the current repository (use some subdirectories for the translated documents).
For example:
. โโโ docs/locales โ โโโ en โ โ โโโ ... โ โโโ es โ โ โโโ ... โ โโโ de โ โ โโโ ... โโโ ...
Could you show me an vivid PR case of the translation
content via the github integration?
I wanna make sure how to category the contents.
Crowdin supports markdown, so it shouldn't change much for us.
Yes, I believe it can handle multi file types with AI power. I just wanna clarify more about the integration details scope/cost in our side, then, we could set up a planning.
@Koooooo-7 the mentioned AI solutions are not included directly in the platform for free. There is only free "Crowdin Translate MT Engine" - Crowdin Neural Machine Translator uses Crowdin Global TM to produce way better suggestions.
To use the OpenAI, Google Gemini, Microsoft Azure OpenAI, Mistral AI, Anthropic providers you'll need your own key.
Could you show me an vivid PR case of the translation content via the github integration?
Sure. For example:
@Koooooo-7 the mentioned AI solutions are not included directly in the platform for free. There is only free "Crowdin Translate MT Engine" - Crowdin Neural Machine Translator uses Crowdin Global TM to produce way better suggestions.
I see. so it depends on whether we need the AI stuff and the tokens cost. thx.
Sure. For example:
I checked the files in those samples, it seems the translation contents
is only contains the translation json
content instead of return a pure translated pages (markdownEN <-> markdwonCN ). such as
https://github.com/nodejs/nodejs.org/blob/main/i18n/locales/en.json
https://github.com/mastodon/joinmastodon/blob/main/locales/en.json
Does it mean that we need implement the i18n function or relying on some i18n components to make it works? Does Crowdin have the ability to do the markdownEN.md <-> markdownCN.md thing?
It's just an example of the integration. Basically, the process is the same, regardless of the file format, Crowdin is able to do the markdownEN.md
<-> markdownCN.md
synchronization as well.
Here is the example project with the synchronization of md files - https://github.com/lirantal/nodejs-cli-apps-best-practices
The only thing here is that it might be tricky to upload the existing translations, especially in such a case of out-of-sync translations.
For files that do not have a defined structure, translation upload is handled by an experimental machine learning technology.
This includes the following file formats: HTML, Front Matter HTML, Markdown, Front Matter Markdown, TXT, Generic XML, Web XML, DOCX, HAML, IDML, DITA, Wiki, FLSNP, MIF, and ADOC.
https://support.crowdin.com/uploading-translations/#text-and-html-based-formats
There is a helper app for that - https://store.crowdin.com/aligner
It's just an example of the integration. Basically, the process is the same, regardless of the file format, Crowdin is able to do the
markdownEN.md
<->markdownCN.md
synchronization as well.Here is the example project with the synchronization of md files - https://github.com/lirantal/nodejs-cli-apps-best-practices
Oh I see! It seems put all files together is easier to maintain and do the integration. Thx for your patience!
The only thing here is that it might be tricky to upload the existing translations, especially in such a case of out-of-sync translations.
For files that do not have a defined structure, translation upload is handled by an experimental machine learning technology.
I think if we decide to do the translation via Crowdin, we could only rely on the EN docs to do the translation and drop current out of date translations to avoid the structure miss match issue.
Regardless of any additional tools involved, the challenge remains the management and quality control of all supported languages. At a minimum we would need someone to manage and monitor translation efforts/updates and then contributors (or reviewers re: generated translations) interested and be available for on-going updates etc.
Until someone comes forward to take on this significant task, I am still leaning towards removing incorrect/outdated docs and relying on Browser translation (where people then expect non-perfect results) and re-assess from there. If we were to keep these outdated docs available in some manner, then adding some sort of date banner as suggested by @Koooooo-7 might be something to consider so we can proceed with the planned release of v5.
Hmm.. one more alternative might be to (temporarily?) remove those languages most out-of-date (two - de/ru) but keep the two most recently updated (zh/es) with some sort of date banner?
I support exploring localization options, however we have short-term needs that I believe need to take priority over long-term aspirations.
Equally important, this was not intended to be a debate about the value of high-quality localized content. That value is understood. If we could provide Docsify documentation in multiple languages quickly, easily, reliably, and affordably (preferably free) then of course we would do so.
With regards to continuing to serve the existing localizations, "People like reading in their preferred language", "it's [maybe] good for SEO", "some of it is still accurate", and "Project X does it so Docsify should, too" are not valid reasons for continuing to serve inaccurate and outdated documentation to our users. These also aren't good reasons to burden our small team of maintainers with decisions made long ago by other people which may no longer be in our best interests.
About those short-term needs...
We are trying to get Docsify v5 out the door. This is the first major version bump for this project since 2017. We do not want to launch v5 with inaccurate, missing, and outdated documentation in any language. If there's one thing we can learn from v4 and the upcoming v5 transition, it is that problematic documentation can cause signficant problems beyond frustrated users and may require signficant effort to address in the future. We don't want to repeat that mistake.
Given our history it seems highly unlikely that all of our localized documentation will be updated in time for v5's launch. Even if we somehow managed to update all localized documentation in time for the v5 launch, we have no system in place for mainting it so we would quickly find ourselves back in this same position. Let's not do that.
So, I propose the following to address our short-term needs and long-term localization aspirations:
https://docsify.js.org/#/zh-cn/quickstart
will redirect to https://docsify.js.org/#/zh-cn/
. If preferred, we can instead redirect to the EN equivalent content.By doing this, we accomplish the following:
By doing this, we accomplish the following:
If folks are okay with the short-term solution above, I will agree to work on it for v5. If someone else wants to take the lead, I'm happy to assist with the effort if needed.
Also...
@sy-records, I truly appreciate your offer to keep the CN docs up to date as changes are made. To be clear, my goal is not to remove all non-English documentation from the site forever. I just don't think our small team should be burdened with tracking and manually updating markdown across PRs because we know this is problematic despite best intentions. I support providing localized documentation in the future if we determine there is enough interest and we can do so reliably and cost effectively.
Thanks for sharing your thoughts @jhildenbiddle on this issue, and I am in agreement with what you propose overall. I hope we can keep scope small to help drive forward a v5 release and then take next steps as needed etc.
I would be happy to help re: taking a stab at the text for some of the above you describe, here is just an example which tries to address several aspects you highlight so everyone can get a better sense of what might be involved etc.๐๐ผ
Preparing for the release of Docsify v5 it was identified that this available language translation was no longer up-to-date and correct. As the Maintainer Team did not want to launch v5 with inaccurate, missing, and outdated documentation in any language, the decision was made to temporarily remove this localized version and rely on Browser language translation of the English version of the Docsify.js.org documentation in order to provide some time to explore options and offer interested contributors to update specific languages etc.
For additional information about using popular Browsers for language translation, please refer to the following links:
Interested in updating this previously available localized language Docsify documentation? Please join our Discord, https://discord.gg/docsify, and let us know in the new #documentation channel, thank you!
Hi @jhildenbiddle thx for ur clarify. I do agree that we should and we'd better move forward to V5 as a new neat and accurate project. That's y I suggest only tag those docs as Out-of-date instead of try to translate it and sync up-to-date.
For the removal, it's hard to say when and where the multi langs support could come back... So I engaged to do adaption instead of abandon.
I think your short term is a good solution but it seems add more efforts on it๏ผ ๏ผglad to know anything can help๏ผ Maybe we could split those tasks, @sy-records and I could work together on the i18n part of the short term changes, ( nice to have: also we do the sync of zh asap). @sy-records is that okay for u?
If those changes would cost much on the i18n changes, maybe we just add the banner to mention other non-EN docs to make the V5 release go ahead ( not pending on those part too much)๏ผ๏ผAs a worse backup PlanB๏ผ
About the Long term i18n solution. It could start the PoC after we finish the V5 settled to avoid any conflict getting in. (I suppose it may have a project structure changes or other integration stuff needs)
I do agree that we should and we'd better move forward to V5 as a new neat and accurate project. That's y I suggest only tag those docs as Out-of-date instead of try to translate it and sync up-to-date.
Understood.
The issue with just adding banners to our localized documentation is that while they will inform visitors 1) that the documentation is outdated and therefore should not be relied upon and 2) where to find up-to-date documentation, they won't provide any information regarding if, how, and when we plan to address the outdated documentation. These are the things that visitors who want localized documentation will want to know. The landing page described above provides this information. They also provide an opportunity to contribute to our localization discussions and efforts via Discord and GitHub, which we can then consider when evaluating our localization aspirations and commitments in the future.
I think your short term is a good solution but it seems add more efforts on it?
It is more work than simply adding banners, but I believe the benefits justify the effort.
I assume the short-term effort would look something like this:
The majority of the work will be in the initial efforts to create the landing page template. The rest is pretty straight-forward.
@paulhibbitts has kindly drafted some copy for the landing page. Perhaps this is an effort you (and @sy-records?) would like to collaborate on with him?
Maybe we could split those tasks, @sy-records and I could work together on the i18n part of the short term changes, ( nice to have: also we do the sync of zh asap). @sy-records is that okay for u?
My position is that we really want to avoid having maintainers manually sync our localization files--both short- and long-term.
As I mentioned above, even if we get our localized documentation up to date for v5, we don't have a reliable system in place for maintaining it so we'll be almost certainly back in this same position eventually.
Perhaps more importantly, I also believe there is an evaluation to be made regarding the value of offering localized documentation vs. the level of effort required to do so. We can look to our traffic statistics on Google Analytics and the level of interest the new landing pages will (maybe) surface on Discord and GitHub to help inform this evaluation. If we find that an insignificant number of site visitors are viewing our translations and/or that no users have expressed interest in localizations after we launch the new landing pages, we may very well decide offering localized content isn't worth it for our small team. On the other hand, if we determine there's enough interest and we find a free automated localization solution like the ones mentioned above, we can prioritize that work accordingly.
I'll close by saying it's exciting to see some movement in this area as well as Docsify in general. I know these threads can get lengthy and there are lots of opinions and options to sort through, but we're making great progress towards modernizing Docsify and publishing the first major release since v4 in 2017. That's awesome. ๐ฅณ
I'd be happy to contribute more work on the text for the English version landing page template and work with @sy-records (or another Maintainer) on where this should be located in the Docsify repo (for example, /docs/_languages/
) and then work on a draft PR etc.
Just as a quick example, I've done another pass of the example draft text and using Docify-This you can preview what this page might look like ๐ https://docsify-this.net/?basePath=https://raw.githubusercontent.com/paulhibbitts/docsify-translations/main&homepage=translation-template.md&hide-credits=true
UPDATE: As an experiment, here is the same landing page in four different languages as translated by ChatGPT 4.0:
Source repo: https://github.com/paulhibbitts/docsify-translations
UPDATE: As an experiment, here is the same landing page in four different languages as translated by ChatGPT 4.0
Thanks for these, @paulhibbitts!
After some discussion on Discord, it sounds like we may end up with a hybrid solution for the release of v5:
This could change, but for those following I believe this is the direction we are currently moving in.
Bug Report
The documentation translations available on docsify.js.org are outdated and in some cases completely abandoned. Inaccurate documentation in a preferred language is (IMHO) far worse than accurate documentation in a non-preferred language.
Last updated:
Without native language speakers to keep these translations up to date, it seems wise to consider the following options (ordered from most to least preferred):
IMPORTANT: At the time Docsify's documentation translations were created, browsers did not offer built-in language translation functionality. Today, all major browsers offer built-in language translation functionality. This means that non-native English speakers can visit our English docs and have them auto-translated to their native language by the browser automatically if they prefer.
Below are screenshots from taken while viewing docsify.js.org translations as an English-preferred visitor:
Chrome - ็ฎไฝไธญๆ
Safari - Deutsch
Firefox - Spanish