Bertverbeek4PS / bc2adls

Exporting data from Dynamics 365 Business Central to Azure data lake storage or MS Fabric lakehouse
MIT License
49 stars 20 forks source link

Output Enum int-values instead of/as well as strings #55

Closed jespertheil closed 8 months ago

jespertheil commented 11 months ago

I have had a request from one of the consumers of our data lake if we can support outputting the option fields as their numeric value instead of the corresponding text. Has anybody met that requirement and maybe looked at a solution. My thought is that the ideal would be to output both as separate columns.

/Jesper

Bertverbeek4PS commented 11 months ago

Hi @JESPERTHEIL, I haven't heard of it. But if you have the export of the numbers instead of the values. I expect that the customer then also want to have the values but in another file or table?

Why do they want to have the numbers instead of the values?

jespertheil commented 11 months ago

Most consumers are fine with the texts, but one wanted something that wouldnt be affected by translations. I think it should be possible to output both numbers and value as text. I'll look into it when I have a dev resource available, just curious if anyone else had similar requests.

Bertverbeek4PS commented 11 months ago

Well maybe creating a switch if you want to export numbers or text. But if you want to to export only the numbers. We can create a extra table that inserts the enums/options with values and translated text.

jespertheil commented 11 months ago

I added the below few modifications which solves our need and outputs 2 columns for options fields, one column as now named Field-XX with the text-representation and another column named Field-XX-val with the option numeric value. The added column is added both to the csv file and to the manifest.

In Util.Codeunit.al: procedure CreateCsvHeader right after "FieldsAdded += 1" :

            if FieldRef.Type = FieldRef.Type::Option then begin
                FieldTextValue := FieldTextValue + '-val";
                Payload.Append(StrSubstNo(CommaPrefixedTok, FieldTextValue));
                FieldsAdded += 1;
            end;
procedure CreateCsvPayload right after "FieldsAdded += 1" :
             if FieldRef.Type = FieldRef.Type::Option then begin
                Payload.Append(StrSubstNo(CommaPrefixedTok, ConvertOptionFieldToValueText(FieldRef)));
                FieldsAdded += 1;
            end;
New function:
      procedure ConvertOptionFieldToValueText(FieldRef: FieldRef): Text
      var
          DateTimeValue: DateTime;
      begin
          case FieldRef.Type of
              FieldRef.Type::Option:
                  exit(ConvertNumberToText(FieldRef.Value()));
          end;
      end;

In CDMUtil.Codeunit.al : procedure CreateAttributes right before the end; statement in the foreach loop:

  if FieldRef.Type = FieldRef.Type::Option then 
                Result.Add(
                    CreateAttributeJson(
                        ADLSEUtil.GetDataLakeCompliantFieldName(FieldRef.Name, FieldRef.Number)+'-val',
                        'Int32',
                        FieldRef.Name+' value',
                        AppliedTraits));

I've attached the two modified files. If someone were to incorporate this in the BC2ADLS module it should maybe be controlled by an option if you want to output option values like this.

src.zip

/Jesper

Bertverbeek4PS commented 11 months ago

Thanks @jespertheil I will look at it after this week when I'm back from Directions.

Bertverbeek4PS commented 10 months ago

@jespertheil could you look at branche https://github.com/Bertverbeek4PS/bc2adls/tree/55-output-enum-int-values-instead-ofas-well-as-strings I have created a extra table for all the enums in the language that the user is running in. It adds this table also to the ADLSE tables and export it to Fabric or DataLake

jespertheil commented 10 months ago

I can see that would work too, and be a better option for multi-lingual instances. But wouldn't it then make sense to output the enum-id instead of the text value for the other tables? Or make that an option as you also suggested above, that would be more intuitive to me.

I'm sorry I can't test the solution since I don't have a BC instance (I don't code in BC anymore - that's at least 12 years ago :)), but I can ask someone to look at it next week.

Bertverbeek4PS commented 10 months ago

That is maybe also an option. But I thought this is a better design if we want to extend it with multiple languages. In the export are all the necessary fields. Like table id, name, field id, name, enum id and caption.

jespertheil commented 9 months ago

Hi Bert,

I finally got around to testing the change, and it works fine but it solves a different problem than my fix does. It might be a niche issue, but the customers BI department on the project I am on was asking to have Enum VALUES instead of / in addition to the Enum Texts because they don't want to rely on changing translations or different language switches on the system.

So my change outputs the enum value, your change supports choosing which language to use for Enum texts, also useful but different, if I have understood the use correctly.

They work fine together so for now we are using BC2ADLS with both changes.

Best regards, and a merry Christmas to you Jesper Theil


Fra: Bert Verbeek @.> Sendt: 24. november 2023 17:33 Til: Bertverbeek4PS/bc2adls @.> Cc: Jesper Theil Hansen @.>; Mention @.> Emne: Re: [Bertverbeek4PS/bc2adls] Output Enum int-values instead of/as well as strings (Issue #55)

That is maybe also an option. But I thought this is a better design if we want to extend it with multiple languages. In the export are all the necessary fields. Like table id, name, field id, name, enum id and caption.

— Reply to this email directly, view it on GitHubhttps://github.com/Bertverbeek4PS/bc2adls/issues/55#issuecomment-1825890154, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKM5PDBQ53SC6KMT5FCMO6LYGDD6VAVCNFSM6AAAAAA6SZXXB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRVHA4TAMJVGQ. You are receiving this because you were mentioned.Message ID: @.***>

Bertverbeek4PS commented 9 months ago

Sorry @jespertheil I now understand it. Very sorry for the confusion! Lett me think of it. Maybe a switch is a solution that you propose! Also now with the new option of exporting translation it will be a better option to have this one also.

Maybe next week I will take a look and do an proposal.

Thanks! And sorry again for the confusion. And have also a happy Christmas!

Bertverbeek4PS commented 9 months ago

@jespertheil I have created Pull Request: https://github.com/Bertverbeek4PS/bc2adls/pull/71

Can you take a look at it? Tested it and everything went OK.

jespertheil commented 9 months ago

@Bertverbeek4PS From the code, it looks like :

I think this is a good solution, but would prefer being able to output both text and value - that is what we do at the moment. I am off for the holidays so wont be able to test in practice before the new year.

Bertverbeek4PS commented 9 months ago

@jespertheil thanks! But in that case you can get the value from the translation file also in the correct language.

Ok I will wait until you have tested also. Happy holidays!

Bertverbeek4PS commented 8 months ago

Added this in the newest release: https://github.com/Bertverbeek4PS/bc2adls/releases/tag/2.17.0

marl0c0 commented 1 month ago

Hi all, in this release it's possible to add "Export Enum as Integer" option, but Booleans still remain as false/true instead of 0/1. Am I correct? Thanks a lot.

Bertverbeek4PS commented 1 month ago

Hi all, in this release it's possible to add "Export Enum as Integer" option, but Booleans still remain as false/true instead of 0/1. Am I correct? Thanks a lot.

That is indeed correct. The option that is build in is only for the enums and not for the boolean. Booleans still will be true or false.