dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.49k stars 10.04k forks source link

Automate localization for resources in templates using Arcade infrastructure #36998

Closed mkArtakMSFT closed 2 years ago

mkArtakMSFT commented 3 years ago

Details on how to onboard can be found at: https://github.com/dotnet/arcade/blob/main/Documentation/OneLocBuild.md

Specific things we'll need to tackle as part of this issue:

BrennanConroy commented 2 years ago

Example of why we need this https://github.com/dotnet/aspnetcore/issues/41802

dougbu commented 2 years ago

I'm curious about a few things

  1. Are we localizing anything beyond the templates other than the templates anytime soon❔
  2. This issue seems to mix a change in how we localize the templates with gaps in what is localized. Which is more important and which is more urgent❔
  3. Using xliff-tasks seems to be a prerequisite for OneLocBuild on-boarding. Is that transition funded❔
dougbu commented 2 years ago

Discussed a bit offline

  1. Not expanding topic beyond templates at this time but push for broader localization remains
  2. The how portion (infrastructure) is more important and urgent because making that change will ease filling the current and future localization gaps
  3. Localization in general is an unfunded mandate. On the other hand, using infrastructure consistent w/ other teams should reduce our costs mid-term
dougbu commented 2 years ago

Note our installers also contain localized strings in files like src\Installers\Windows\SharedFrameworkBundle\1033\thm.wxl Oddly, I don't see anything similar for our Debian or Rpm installers😦 I'll at least check how other teams are localizing their installers.

TanayParikh commented 2 years ago

From Phil:

The localized "name" of the parameter is fixed in NET 6.

What needs to happen to fix the tool tip is one of following:

Update to use the existing strings.json mechanism: Add entries to en\strings.json for both the name and description of the parameter: aspnetcore/strings.json at main · dotnet/aspnetcore (github.com) Note that the localized versions of the strings.json already have the translated name, since I had previously just modified the EN strings.json we had on our end, without it we fallback to the string that is in the template.json: aspnetcore/strings.json at main · dotnet/aspnetcore (github.com)Image After step 1 above, we can grab the updated strings.json into the webtools repo and get the description localized and hand it back after translations happen. Ideally the localization mechanism should be updated to be entirely in the aspnet repo - localizing in webtools was never meant to be the permanent solution, and doesn't solve the problem of localizing the template metadata in the dotnet new. Onboard the ASP.NET repo into its own localization mechanism for the templates, continue to localize strings.json OR - move to the new localization mechanism which involves localizing the template.config files. David has some additional information on this.

https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1075373#2761831

dougbu commented 2 years ago

Is there anything to do for this issue❔ If not, the relevant teams should instead concentrate on the lower-level localization gaps described in the Description. It's only clear so far that OneLocBuild isn't relevant to installer or template localization.

The question depends on whether we want to move forward with "the localization mechanism should be updated to be entirely in the aspnet repo" because that seems to be the main "manual" step. If we want to do this and we need me to do it, I'll need a lot more information on the upsides of either approach @phenning described.

phenning commented 2 years ago

A few things.

  1. Strings.json localization is mostly being deprecated, with the exception of some corner cases in some templates where the CLI symbol display name differs from what is used in Visual Studio. These are mostly due to the symbols which are “invertBoolen” = true in the host file.

  2. All other localization strings need to be done via the templatestrings.json files. If these exist in the template, Visual Studio will not use strings.json for parameter localization unless an override is set in the host file. The repo currently has an example of this.

  3. We are no longer round tripping strings between aspnet and WebTools repos. This was to be a short lived stop gap we did back in 3.1 until the work could be done in the repo. The localization needs to now live with the templates. Winforms, WPF, and other repos containing templates are localizing their templatestrings.json, and in some cases strings.json as well. @bekir-ozturk or @vlada-shubina should be able to either help or get you in touch with people who set set the process up in those repos.

To be clear, without the templatestrings.json being localized and shipped in the templates, the names and descriptions will not be localized in Visual Studio when NET 7 is installed in a customers machine.

My comments quoted above where actually from last year and had more to do at that time where stings.json was the primary localization mechanism. That is no longer that cases. templatestrings.json is used by both VS and the CLI, strings.json was never supported by the CLI.

I hope that clarifies the current state and why this is still important.

dougbu commented 2 years ago

We use both templatestrings.json and strings.json in this repo, with strings.json present only in the Angular-CSharp, React-CSharp, and WebApi-CSharp templates.

The WebApi-CSharp case confuses me because templatestrings.*.json files also exist in that template, I don't know why #40153 was needed, and a few changes have been made to the templatestrings.*.json files w/o touching the strings.json files since @TanayParikh merged #40153.

The other templates were already split into https://github.com/dotnet/spa-templates (see #34366) when @bekir-ozturk did #38867. Probably need a similar PR to #38867 in that repo. No idea how that work was done (e.g. are there tools to help the conversion❔).


Separately, @mkArtakMSFT, @captainsafia, and @TanayParikh do the sub-issues apply throughout the repo or only to specific templates❔ I'm basically wondering if the steps outlined in #38867 have been missed in a few cases or if the problem is continued use of strings.json files.

phenning commented 2 years ago

First I'll summarize, then later give a bit of a history lesson explaining the evolution of the localization story in templates.

To your specific questions:

  1. strings.json is still needed when there is no templatestring..json OR if the template makes use of "invertBoolean" in ide.host.json which requires different text in VS versus the CLI. In these cases "overrideDefaultText" needs to be set in order to force strings.json to be used over templatestring..json for that symbols displayName or description. That is why #40153 is there.

  2. As to why there are only templatestring..json updates and not strings.json, it is because templatestrings..json take precedence over strings.json in cases where overrideDefaultText is not explicitly set in the host file, so there is no point in making the changes to both files in those cases (explained in detail below)

  3. The other submodule with the angular templates do need a similar change to #38867. I believe there is some tooling there, @bekir-ozturk would have more context about the initial generation of these files. See https://github.com/dotnet/templating/wiki/Localization

  4. It appears as though despite @bekir-ozturk's work that Arcade is still not inserting updated templatestrings.*.json into the repo, as you can see here: https://github.com/dotnet/aspnetcore/blob/main/src/ProjectTemplates/Web.ProjectTemplates/content/WebApi-CSharp/.template.config/localize/templatestrings.de.json

  5. I should also point out that the translated strings.json which are in the repo are the result of the manual WebTools roundtripping we used to do. If we need to update these translations in the future - they should be localized as well, for the templates which still required them (see below for details)

Evolution of Template Localization

Historically, there was no localization support in the core template engine for localizing template metadata. The CLI was rather loose in requiring this, but the Visual Studio requirement was stronger, so we initially followed the model that VSTEMPLATES had, by having metadata in the host file which consist of a VS Package Guid and Resource Id. The assembly containing this was produced in the WebTools repo.

This had the problem of decoupling the template content from the localized metadata. So that whenever a change was made to the template metadata in strings.json (like the name/description of the template or a symbol) - we'd need to make the change in WebTools and ship it. Obviously this had its issues.

Around 3.1 timeframe, we tried to solve this problem by introducing strings.json and deprecating the use of the package/resourceIds. The initial attempt here was for us (WebTools) to seed the aspnet repo with the initial translations from our existing translations of the resx files that had the package resources. My hope then was that the repo would localize its own content directly. That never happened, instead when a string change was made in the aspnet repo, we needed to update the strings.json we had in WebTools, generate a build, have a loc handoff/handback then reinsert the updated language strings.json to the aspnet repo. There was no automation around this process, like you said - it was an unfunded.

Mid to late in net6, template engine core team implemented the templatestrings.*.json method which could be used to localize the template metadata for display in dotnet new list. They onboarded the console and class templates and enabled localization for their repo.

In Visual Studio we implemented logic that would prefer templatestrings.json if the template contained it, while falling back to looking for strings.json if the template did not contain the updated localization mechanism. This would allow us to still have localization on older versions of VS which did not recognize templatestrings.json while still trying to move to the future. The goal here is that eventually we would be able to remove strings.json from the templates. Since net7 is only supported in VS 2022+ and the templatestrings have been supported since 17.0 we could do it now, if not for the following issue.

However there was/is a problem. Some templates, like WebAPI have inverted booleans (see https://github.com/dotnet/aspnetcore/blob/10b1fd98b06a4a2c590a14c95ff412227f79010e/src/ProjectTemplates/Web.ProjectTemplates/content/WebApi-CSharp/.template.config/ide.host.json#L42). This requires us to invert things like enable/disable in the parameter name. Unfortunately, templatestrings.json has no way to provide different strings based on where the template is being loaded from. We can get around this in Visual Studio by specifying overrideDefaultText: true. Doing this causes us to fallback to using strings.json for that particular symbol. This was implemented as another transition stop gap. Long term, this should go away after template.json supports per-host displayNames.

Thats the long winded explanation and should cover most of it and give more context explaining why/how we are where we are now.

dougbu commented 2 years ago

long winded explanation

I really appreciate the background❕

dougbu commented 2 years ago

Wow, OneLocBuild stuff does apply here, according to https://github.com/dotnet/templating/wiki/Localization#1st-party-templates-localization.

@bekir-ozturk I see the on-boarding process described at https://ceapex.visualstudio.com/CEINTL/_wiki/wikis/CEINTL.wiki/2973/Onboarding-Readiness?anchor=how-to-check-onboarding-readiness-step-by-step isn't complete (at least w/ respect to csigs or celoc permissions in our AzDO and GitHub repo). What else has not been done❔

vlada-shubina commented 2 years ago

@dougbu afaik, onboarding to OneLocBuild was not completed. Templating team only did the part explained in https://github.com/dotnet/templating/wiki/Localization#1st-party-templates-localization - enabled localization via property and checked-in initial non-translated files.

The scope of this ticket #36998 is to complete onboarding so the templates start to be localized.

dougbu commented 2 years ago

:ok: @vlada-shubina now I'm wondering why we're cleaning this up months later and who was informed of the remaining work❔

phenning commented 2 years ago

@dougbu I did open issue #40666 back in March and it was duped to this issue. Oh and the title of this issue is actually the remaining work. "Automate localization for resources in templates using Arcade infrastructure"

dougbu commented 2 years ago

@dougbu I did open issue #40666 back in March and it was duped to this issue. Oh and the title of this issue is actually the remaining work. "Automate localization for resources in templates using Arcade infrastructure"

There shoudl be no work remaining w.r.t. Arcade and its infrastructure (ignoring the spa-templates bit) and the mention of XLF files in #40666 is just confusing. The interaction between us and the localization team is the important piece and I haven't heard where @bekir-ozturk left that.

phenning commented 2 years ago

I can't really speak to that, but based on Artak's comments in my issue, I suspect that nothing was done in regards to onboarding the repo with the localization team or adding the localization build.

I was curious though and found where WPF team repo had done the pipeline configuration. https://dev.azure.com/dnceng/internal/_git/dotnet-wpf/commit/03d9d5eed18265b3ec566024a70e1e20bc8152e5?refName=refs%2Fheads%2Fmain&path=%2Feng%2Fpipeline.yml&_a=compare

I hope this helps - note that there are some updates to the yml after this initial commit.

It even localizes both templatestrings.json and strings.json. From generate-locproject.ps1:

$jsonFiles += Get-ChildItem -Recurse -Path "$SourcesDirectory" | Where-Object { $_.FullName -Match "\.template\.config\\localize\\en\..+\.json" } # .NET templating pattern
$jsonFiles += Get-ChildItem -Recurse -Path "$SourcesDirectory" | Where-Object { $_.FullName -Match "en\\strings\.json" } # current winforms pattern
vlada-shubina commented 2 years ago

🆗 @vlada-shubina now I'm wondering why we're cleaning this up months later and who was informed of the remaining work❔

As far as I know we informed the team after the template localization onboarding, that's why this issue was created to track outstanding work. @bekir-ozturk and @mkArtakMSFT may provide more details here.

dougbu commented 2 years ago

Thanks again for your offline help @vlada-shubina. I'll follow up w/ @jonfortescue on the submodule wrinkle tomorrow (they had a morale event today 😀).

dougbu commented 2 years ago

Submitted https://ceapex.visualstudio.com/CEINTL/_workitems/edit/650311 for the onboarding. Working on onboarding dotnet/spa-templates as well…

Since the OneLocBuild tasks seem to work fine w/ strings.json files and I can't find tools to convert those files, is it fine to deal with converting one to the other in dotnet/spa-templates later (potentially much later)❔ Or @bekir-ozturk are there tools somewhere❔

dougbu commented 2 years ago

dotnet/arcade#9851 means we could perhaps localize dotnet/spa-templates but not keep onelocbuild.yml current ☹️ That's not sustainable.

dougbu commented 2 years ago

I'm going to walk my "blocking" comment in dotnet/arcade#9851. We could, if necessary, use a GitHub action to synchronize files we need e.g. from dotnet/aspnetcore to dotnet/spa-templates.

dougbu commented 2 years ago

For those playing along at home, a checklist:

dougbu commented 2 years ago

Caught a typo, adding #42590 to the checklist…

dougbu commented 2 years ago

Shoot, hit a problem w/ the Arcade infrastructure for spa-templates. See https://teams.microsoft.com/l/message/19:afba3d1545dd45d7b79f34c1821f6055@thread.skype/1657073140162?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&parentMessageId=1657073140162&teamName=.NET%20Core%20Eng%20Services%20Partners&channelName=First%20Responders&createdTime=1657073140162 for the issue.

dougbu commented 2 years ago

It's difficult to find w/o some hunting or a query like https://github.com/pulls?q=is%3Aopen+is%3Apr+author%3Adotnet-bot+repo%3Adotnet%2Faspnetcore+repo%3Adotnet%2Fspa-templates+sort%3Aupdated-desc+ but #42551 contains the initial swath of translations for dotnet/aspnetcore 🎉 Unfortunately that PR has been open for 4 days w/o action ☹️

dougbu commented 2 years ago

The https://dev.azure.com/dnceng/internal/_build/results?buildId=1863737&view=logs&j=c0636702-cad1-5e4d-7b5b-f15c1cce530a&t=ef3c2779-04f6-5216-b20f-460f31132ece logs show we're at "43% localized for 24 resource files" so far. The task will keep updating the strings until the loc team is done. Or, if we merge the PR, they'll start again w/ a new one. I suggest merging early and often since we're close to the preview7 cutoff but don't feel I'm the right person to approve these changes.

dougbu commented 2 years ago

Things are looking fairly good here. We have almost reached 100% localization of the template strings in dotnet/aspnetcore, completed preparations in dotnet/spa-templates, and opened Loc Request 656173: [SWLOC-ProjectReOnboarding] Onboard dotnet-spa-templates in OneLocBuild - Boards (visualstudio.com).

If someone could check then merge #42636, I'd appreciate it.

Please see checklist above for additional details.

dougbu commented 2 years ago

For dotnet/aspnetcore follow-ups once #42636 is in, I suggest the following

dotnet/spa-templates will likely deserve the same treatment.

dougbu commented 2 years ago

Of course, CTI should also validate the bugs listed in the Description of this issue are resolved once #42636 is in and CTI team has new templates and those templates are in a VS build 😁 The SPA templates will take slighty longer to reach this point.

dougbu commented 2 years ago

We're still waiting for a new Package Id for the spa-templates localization work though I've pinged a couple of times. If CTI had been testing those templates, I suspect the number of localization gaps would have been much higher. Recommend they look closely once we have done the work.

dotnet/aspnetcore has wrapped up for now though processes going forward haven't been written. This means the items in the original Description are all done. (The original issues were all closed anyhow.)

I also have a conversation going w/ Arcade, installer and localization team members about automating localization of strings in our Windows installers. This might be possible and would reduce manual intervention after the next installer strings change.

dougbu commented 2 years ago

workaround (or see loc team fix) a bug I reported about 99% completion due to mishandling of version properties in strings.json files

I'm trying out a workaround for this concern. Will know only after next hand-back from the loc team after I merge that change (still a draft) whether it works for us.

dougbu commented 2 years ago

On spa-templates, loc team is hitting an AzDO issue that is likely slowing things down. Not sure when we'll see dotnet/spa-templates#64 get newly localized strings, though the relevant ICM should be mitigated by the 20th.

dougbu commented 2 years ago

workaround (or see loc team fix) a bug I reported about 99% completion due to mishandling of version properties in strings.json files

Build #20220727.1 artifacts include a LocCompleteness.json file indicating an overall ResourcesToLocalize count of 104 which matches the number of files touched (due to dotnet/aspnetcore#42802) in dotnet/aspnetcore#42928. This is a clear indication my fix attempt in dotnet/aspnetcore#42797 was successful.

This completes dotnet/aspnetcore onboarding.

the relevant ICM should be mitigated by the 20th

Apparently the mitigation didn't occur until today. Loc Request 656173: [SWLOC-ProjectReOnboarding] Onboard dotnet-spa-templates in OneLocBuild - Boards (visualstudio.com) is now unblocked and the loc team has begun their work on translations.

Including translations in dotnet/spa-templates#64 is back on track and will complete dotnet/spa-templates onboarding and resolve this issue.

mkArtakMSFT commented 2 years ago

One more report, which may probably be covered by this change: https://github.com/aspnet/AspNetCore-ManualTests/issues/1107

Rita003 commented 2 years ago

@mkArtakMSFT This issue also repro on 3.1 ASP.NET Core Web App with the latest Dev17.4 Preview 1 CHS OS, the 7.0/6.0 ASP.NET Core Web App is well. Do I need to open a new bug to track it? image

v-elenafeng commented 2 years ago

"Do not use this top-level statements " repro on d17.4-32731.10 from Pull request 414422 when creating an Angular/React project image

phenning commented 2 years ago

@mkArtakMSFT This issue also repro on 3.1 ASP.NET Core Web App with the latest Dev17.4 Preview 1 CHS OS, the 7.0/6.0 ASP.NET Core Web App is well. Do I need to open a new bug to track it? image

@mkArtakMSFT I think this will be by design - 3.1 template never had this string localized, and this option is no longer in the template for 6.x. We would need to update the 3.x templates with the new loc infrastructure.

dougbu commented 2 years ago

We would need to update the 3.x templates with the new loc infrastructure.

According to https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core#lifecycle, we don't have much runway left for 3.x (EOL is 13 December 2022). We could port the infrastructure to 6.0.x but haven't made that call yet. @dotnet/aspnet-admins will discuss the question early next week.

v-elenafeng commented 2 years ago

"Do not use this top-level statements " repro on d17.4-32731.10 from Pull request 414422 when creating an Angular/React project image @dougbu So, localization for this one is still in progress?

dougbu commented 2 years ago

So, localization for this one is still in progress?

No decision has been made on localization in 6.0.x

v-elenafeng commented 2 years ago

So, localization for this one is still in progress?

No decision has been made on localization in 6.0.x

Repro on 7.0 as well for Angular /React templates. image

dougbu commented 2 years ago

Repro on 7.0 as well for Angular /React templates

That's entirely expected given the short time since the localization work was checked into https://github.com/dotnet/spa-templates. That submodule was only updated in https://github.com/dotnet/aspnetcore 2 days ago and the fixes likely haven't made it into the SDK, let alone VS.

If you're installing recent template packages and using them in VS

  1. How are you doing this❔ I've never found out a way
  2. Exactly what template package version were you testing❔
v-elenafeng commented 2 years ago

Repro on 7.0 as well for Angular /React templates

That's entirely expected given the short time since the localization work was checked into https://github.com/dotnet/spa-templates. That submodule was only updated in https://github.com/dotnet/aspnetcore 2 days ago and the fixes likely haven't made it into the SDK, let alone VS.

If you're installing recent template packages and using them in VS

  1. How are you doing this❔ I've never found out a way
  2. Exactly what template package version were you testing❔

I just installed d17.4-32731.10 from Pull request 414422 which templates version are 6.0.8 and 7.0.0-preview.7.22376.6 Not repro on ASP.NET Core web app templates.

dougbu commented 2 years ago

templates version are 6.0.8 and 7.0.0-preview.7.22376.6

Build #20220726.6 back on 26 July produced those 7.0 ASP.NET templates. The localization updates for spa-templates i.e. not the web app templates were not part of any preview 7 build. You need ASP.NET template version 7.0.0-rc.1.22401.3 or newer to see fully localized Angular and React content.

v-elenafeng commented 2 years ago

Build #20220726.6 back on 26 July produced those 7.0 ASP.NET templates. The localization updates for spa-templates i.e. not the web app templates were not part of any preview 7 build. You need ASP.NET template version 7.0.0-rc.1.22401.3 or newer to see fully localized Angular and React content.

@dougbu Thanks for your confirmation, I checked this issue with 7.0 RC1 build, and it is localized on both 6.0 and 7.0 SPA templates.

LuckyKang79 commented 2 years ago

@mkArtakMSFT This issue is also reproduced on 6.0/3.1 All Projects on the latest Dev17.3 GA [rel.d17.3-32803.156] KOR OS. Will it be fixed in the future? If not fix. Do I need to open a new bug to track it? image