MultilanguageLg issue resolving templates using import

[guru98]

Github issues should be used for bugs and feature requests. Use Stack Overflow for general "how-to" questions.

Version

4.9.4

Describe the bug

i have a multilanguage application and i use adaptive cards. To make an adaptivecard generic i have activity and adaptive cards teplates definition in specific lg files. So i have text in language specific lg files where i import lg files with activity that include lg files with cards . In 4.5.X code everything works fine but upgrading to 4.9.4 i have an exception becouse in the card it can't resolve the text templates

To Reproduce

Steps to reproduce the behavior:

1. create an lg file with a template for a language and import a secong lg file. in the second file make a reference to a template in the first file.

Expected behavior

I assume that the structure of the templates is flat so calling a template is always resolved. In multilanguage it's the only way to reduce code duplication

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

[bug]

[jwiley84]

Hi @guru98 I'll try to track down what change broke this, and get back to you. Thank you for reporting this!

[jwiley84]

Hi @guru98 Can you post a limited version of your lg files?

[guru98]

Hi @jwiley84 sure... in attach ResponsesZip.zip

[guru98]

Any news? I have an other app that use adaptive dialog and I used MultiLanguageGenerator in that case and everything works fine. I also notest that in the skill examples the activity files are duplicated (one per language) so I assume that developers have the same problem there.

[jwiley84]

Hi @guru98 I noted the same thing in the samples, it was why I asked for your lg files. I'm trying to repro using the steps you outlined, to see if this occurs without even using languages. Thank you for your patience.

[jwiley84]

I'm not able to repro this issue (mine is not erroring, it's just blank), so I'm going to reach out to to a coworker for this.

[guru98]

Hi @jwiley84 , here the simplest way to reproduce the error. create 2 files first.lg and second.lg (in attach) than you can make 2 calls: Templates lgTemplates = Templates.ParseFile(Path.Combine(".", "Resources", "first.lg")); var t1 = lgTemplates.Evaluate("test2"); var t2 = lgTemplates.Evaluate("second2"); ` lgfiles.zip

the beaviour in my case is that when you call the first Evaluate i receive an error: "[Error] C:\Users\UgoCastellani\Source\Repos\EchoBot2\EchoBot2\Resources\second.lg line 5:2 - line 5:12: [second2]Error occurred when parsing expression 'test1()'. test1 does not have an evaluator, it's not a built-in function or a custom function."

Hope this help

[vishwacsena]

@boydc2014 can you help/ comment?

[jwiley84]

My repro was the same as your second set of files, which is why I am so very confused why I'm getting a different problem all together. Hopefully @boydc2014 will be able to help us untangle this.

[guru98]

Hi, i did some test. My feeling is that the issue is related to importResolver and resourceID. I didn't understand all the implications but the template is in the AllTemplates list but not resolved and this could be an issue with the resourceid. If you use adaptive dialog they use a different approach with a different importResolver.

[boydc2014]

Hey, @guru98 sorry for the late response and the inconvenience, the behavior is expected.

Prior to 4.9.x, LG imports works more like a "flatten" like what you observed, in which import just means concat everything together and treated as one big lg file. This is not the usual pattern (a directed dependency graph) you would see in a common programming language.

This "flatten" model works well and did provide some convenience in certain scenario that you don't have to remember which one import which one. And that time, the API we provided to parse file is called parseFiles which accept a bunch of files, as long as it's refered, it will be treated together.

But the "flatten" model have several issues,

1. it really don't scale very well, when we have more advance\customized reference structure like multi-language, generated dialog, and many other sharing scenario where many lg files are involved.
2. when you have many files and cross refereed, as you would image, its also not friendly to reason about
3. also this flatten model, encourages or at least enable users a pattern that, not spending efforts to organize their lg files in a very well structure, which will bring long-term debts eventually. We are seeing customer doing this and result in more efforts to refactor when there scenario and solution go deep.

So we made the changes to make LG imports works like a normal module import like programming language, in which the lg been imported can not access the one that import it. So the import\dependency is more like a DAG. So in your case, 2.lg is imported by 1.lg, the templates\functions in 2.lg can't access the templates in 1.lg. The error message is a little bit too generic saying saying there is no evaluator for test1, i will see what we can do to make that clearer.

Is this make sense to you?

[guru98]

Hi Dong, Thanks for your answer. What you describe have sense but have a very Bad behavior for multilanguage where you have template for text and common template for activities. In that case you have to duplicate template related to activities for every language that is a no so good behavior. What you suggest in that case. Use of namespace can mitigate the issue? Thanks

Ottieni Outlook per Androidhttps://aka.ms/ghei36

---

From: Dong Lei notifications@github.com Sent: Wednesday, August 19, 2020 4:22:14 PM To: microsoft/botbuilder-dotnet botbuilder-dotnet@noreply.github.com Cc: Ugo Castellani ugoc@hotmail.com; Mention mention@noreply.github.com Subject: Re: [microsoft/botbuilder-dotnet] MultilanguageLg issue resolving templates using import (#4381)

Hey, @guru98https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fguru98&data=02%7C01%7C%7C64632564303f4dbb18c808d8444b45ff%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637334437367187061&sdata=Yi2SVqqQvwS6ratL9SQ7hgUbgCVpA6l2VLkC8ycWOIM%3D&reserved=0 sorry for the late response and the inconvenience, the behavior is expected.

Prior to 4.9.x, LG imports works more like a "flatten" like what you observed, which is not the usual pattern (a directed dependency graph) in normal programming language that.

This "flatten" works well and did provide some convenience in certain scenario that you don't have to remember which one import which one. And that time, the API we provided to parse file is called parseFiles which accept a bunch of files, as long as it's refered, i will be treated toghther.

But the "flatten" model have several issues

1. it really don't scale very well, when we have more advance\customized reference structure like multi-language, generated dialog, and many other sharing scenario where many lg files are involved.
2. when you have many files and cross refereed, as you would image, its also not friendly to reason about
3. also this flatten model, encourages or at least enable users a pattern that, not spending efforts to organize their lg files in a very well structure, which will bring long-term debts eventually.

So we made the changes to make LG imports works like a normal module import like programming language, in which the lg been imported can not access the one that import it. So the import\dependency is more like a DAG. So in your case, 2.lg is imported by 1.lg, the templates\functions in 2.lg can't access the templates in 1.lg.

Is this make sense to you?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fbotbuilder-dotnet%2Fissues%2F4381%23issuecomment-676438914&data=02%7C01%7C%7C64632564303f4dbb18c808d8444b45ff%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637334437367197068&sdata=zJyjHe%2F028M4%2F2tb187Jh%2BOcm77csVPCQdOrs5xABH4%3D&reserved=0, or unsubscribehttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA77TTUXKDV4QOH3OPYHL3LSBPNZNANCNFSM4PURID4Q&data=02%7C01%7C%7C64632564303f4dbb18c808d8444b45ff%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637334437367217046&sdata=EiJeIYJtvEg41DHErNXlKYPRG6yomfHiBqAv1lOwrBA%3D&reserved=0.

[boydc2014]

Hi Dong, Thanks for your answer. What you describe have sense but have a very Bad behavior for multilanguage where you have template for text and common template for activities. In that case you have to duplicate template related to activities for every language that is a no so good behavior. What you suggest in that case. Use of namespace can mitigate the issue? Thanks Ottieni Outlook per Androidhttps://aka.ms/ghei36

Hey @guru98, customized import resolver is what we recommended in your scenario, if i understand your scenario correctly. For example, i assume you have, let's say 3 lg files.

common.lg, hello.en-us.lg, hello.fr-fr.lg

common.lg // in where you defined a shared function, sayHello, that you don't want to duplicate

[import] (hello.lg)
# sayHello
- ${helloText}

hello.en-us.lg // in here you defined a helloText in en-us

# helloText
- hello in english

hello.fr-fr.lg // in here you defined a helloText in en-us

# helloText
- hello in fr-fr

CustomizedResolver means, we allow you to pass in an delegate to customize the interpretation of this import statement [import] (hello.lg) .

In code, it will looks like

// parsing templates
var TemplatesInEN = Templates.parseFile("path to common.lg",  a resolver which will direct hello.lg to hello.en-us.lg)
var TemplatesInFR = Templates.parseFile("path to common.lg",  a resolver which will direct hello.lg to hello.fr-fr.lg)

// run time
switch (incommingLocale) {
   case: "en-us": // use templatesInEN to response
   case: "fr-fr": // use templatesInFR to response
   ...
}

Basically, your assets are well separated, and you are just combining that in different ways for different languages.

Will this help? I can get some concrete coding examples if this is what you are looking for.

[guru98]

Seems ok. Two questions: in.the adaptive dialog language generator my approach still work. It is subject to change? What happen for declarative dialog in that case? I have to use custom dialog? Anyway if you have Samples for custom importresolver are welcome

---

From: Dong Lei notifications@github.com Sent: Wednesday, August 19, 2020 5:01:22 PM To: microsoft/botbuilder-dotnet botbuilder-dotnet@noreply.github.com Cc: Ugo Castellani ugoc@hotmail.com; Mention mention@noreply.github.com Subject: Re: [microsoft/botbuilder-dotnet] MultilanguageLg issue resolving templates using import (#4381)

Hi Dong, Thanks for your answer. What you describe have sense but have a very Bad behavior for multilanguage where you have template for text and common template for activities. In that case you have to duplicate template related to activities for every language that is a no so good behavior. What you suggest in that case. Use of namespace can mitigate the issue? Thanks Ottieni Outlook per Androidhttps://aka.ms/ghei36https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7C%7Cbacf4ed1daa242b1735a08d84450bc88%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637334460828475003&sdata=PctJkaIX7iIZCyPLb60bhfKYn9T5Est6jgZ1PAJdM8w%3D&reserved=0

Hey @guru98https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fguru98&data=02%7C01%7C%7Cbacf4ed1daa242b1735a08d84450bc88%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637334460828484998&sdata=vRZsb1qxOwUI2HeqPCtzh3EudYc0RSdXE5NdC2Ns8HI%3D&reserved=0, customized import resolver is what we recommended in your scenario, if i understand your scenario correctly. For example, i assume you have, let's say 3 lg files.

common.lg, hello.en-us.lg, hello.fr-fr.lg

common.lg // in where you defined a shared function, sayHello

[import] (hello.lg)

sayHello

• ${helloText}

hello.en-us.lg // in here you defined a helloText in en-us

helloText

• hello in english

hello.fr-fr.lg // in here you defined a helloText in en-us

helloText

• hello in fr-fr

CustomizedResolver means, we allow you to pass in an delegate to customize the interpretation of this import statement "importhttps://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhello.lg%2F&data=02%7C01%7C%7Cbacf4ed1daa242b1735a08d84450bc88%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637334460828494999&sdata=gqrZ56XPmSKCd%2FTNgJI6pRmsayBYS%2BMV%2BBIVSlwajY0%3D&reserved=0".

In code, it will looks like

// parsing templates var TemplatesInEN = Templates.parseFile("path to common.lg", a resolver which will direct hello.lg to hello.en-us.lg) var TemplatesInFR = Templates.parseFile("path to common.lg", a resolver which will direct hello.lg to hello.fr-fr.lg)

// run time switch (incommingLocale) { case: "en-us": // use templatesInEN to response case: "fr-fr": // use templatesInFR to response ... }

Will this help? I can get some concrete coding examples if this is what you are looking for.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fbotbuilder-dotnet%2Fissues%2F4381%23issuecomment-676480499&data=02%7C01%7C%7Cbacf4ed1daa242b1735a08d84450bc88%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637334460828494999&sdata=qwhHcPBhhhbHwqnjSrh0SqGfZndwMHDCoWyoT1oZXeg%3D&reserved=0, or unsubscribehttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA77TTT5WLV4ZJERI34BY6LSBPSMFANCNFSM4PURID4Q&data=02%7C01%7C%7Cbacf4ed1daa242b1735a08d84450bc88%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637334460828504992&sdata=n20OQsPDcBPuhoqSOcimYRyRujctjUbXaDp91vE5S3Q%3D&reserved=0.

[boydc2014]

Seems ok. Two questions: in.the adaptive dialog language generator my approach still work. It is subject to change? What happen for declarative dialog in that case? I have to use custom dialog? Anyway if you have Samples for custom importresolver are welcome

Do you mean in AdaptiveDialog's languagGenerator, your 2.lg can still access 1.lg's templates even it's 1.lg importing 2.lg (not 2.lg import 1.lg)? I don't think that's supported, because the generators in adaptiveDialog packages (either "TemplateEngineLanguageGenarator" or "ResourceMultiLanguageGenerator") is an upper layer call of the core lg library api which has already upgraded to this new importing model. So i don't think in adaptive dialog's generators could by pass that limitation. can you check your adapative dialog and lg library are both 4.9.+? I will double confirm the case.

In adaptive and declarative dialog (also in Composer which result in declarative assets), we also used a customized import resolver, but the pattern is a little different than the example i gave you above. The pattern above i gave is for the scenario that don't want to duplicate this "entry lg file" common.lg, which is definite a valid scenario.

The pattern we choose in declarative and composer is more isolated, here is some details, when you sepecify "generator: a.lg" in declartive json, it's actually not referring to a real physical file a.lg, it's a mapping to a set of a.[locale].lg like

• a.en-us.lg
• a.fr-fr.lg

What happened at backed, is we will group all lg files by locales first, then try to do customized importing within each locale group, en-us group, fr-fr group, etc. And there is a one special group, the "empty group" (foo.lg, bar.lg) is shared across all group as a "fallback" when no file is found within this group. And when the bot receive an activity, we will use the locale in the incoming activity to determine which group's lg file we want to pick, pick a.en-us.lg or a.fr-fr.lg like that.

To simplify, the pattern used in declarative and composer requires different lg files for different locales as "entries", not supporting sharing one "entry lg file". Except that difference, the rest is almost identical to the above example.

Lg files, given the way we support functions, modules, imports, customized importing, it will result in many valid patterns and some of them are contradict to the others. In declarative and Composer, right now, we have to pick one pattern as default, and the criteria for that is more about clear structure and clear message, than minimal duplication.

I will get the customized import resolver sample sent to you soon.

[Danieladu]

Here the simple sample for MultiLanguageLG to handler multi lingual cases with custom import resolver. https://github.com/microsoft/botbuilder-dotnet/blob/12a204021c2347e3991b4b3f9f8a182d8d99d911/tests/Microsoft.Bot.Builder.LanguageGeneration.Tests/MultilanguageLGTest.cs#L95

Hope it helps. Thanks.

[guru98]

Hi, i did some test and I think i understand the point... i will update my code soon. Anyway the "flat approach continue to work with adaptivedialog lggenerator updating library to 4.10. Considering that working with adaptivedialog push to use smaller lg files for every dialog or so this approach could be resonable without major impact on performance. Thanks for your amazing support

[boydc2014]

Thanks @guru98, we are closing this for now.