search

Bertverbeek4PS / bc2adls

Exporting data from Dynamics 365 Business Central to Azure data lake storage or MS Fabric lakehouse
MIT License
60 stars 22 forks source link

Fabric - CopyBusinessCentral.ipynb - multi-company export #62

Closed lhammerbh closed 11 months ago

lhammerbh commented 11 months ago

Dear Bert and Co.

We are testing multi-company export in Fabric mode.

The export of data using the bc2adlse component is working perfectly fine.

When we are running the CopyBusinessCentral python script, we seem to be missing out some of the values. The reason seems to be that in a multi-company export, the system id in the same table for different companies may have the same systemid value.

A possible solution could be add $Company to the df_new.dropDuplicates, so the code could be:

remove duplicates by filtering on systemID, $Company and systemModifiedAt fields

df_new = df_new.orderBy('systemId-2000000000',['$Company'],desc('SystemModifiedAt-2000000003')) df_new = df_new.dropDuplicates(['systemId-2000000000'],['$Company'])

Yours Lars Hammer

Bertverbeek4PS commented 11 months ago

Hi @lhammerbh you are totally right. I though that I have changed that. And Company must be there. I will fix this today!

Bertverbeek4PS commented 11 months ago

Same issue as https://github.com/Bertverbeek4PS/bc2adls/issues/56

Bertverbeek4PS commented 11 months ago

@lhammerbh in the newest version in the repro the bug is fixed with the modified fields and the deleted records. Thanks for reporting!!

lhammerbh commented 11 months ago

Perfect. Bert.

Thanks for quick response.

Best regards, Bunker Holding A/S

@.*** Lars Christian Hvidegaard Hammer Technical Business Intelligence Lead Group IT

M +45 60 37 67 98 @.***<mailto:(Email)>

VAT number: 75266316 [Bunker Holding Group logo] Terms, conditions, and privacy policy of the company can be found at www.bunker-holding.comhttp://www.bunker-holding.com/ From: Bert Verbeek @.> Sent: Wednesday, November 29, 2023 1:13 PM To: Bertverbeek4PS/bc2adls @.> Cc: Lars Christian Hvidegaard Hammer @.>; Mention @.> Subject: Re: [Bertverbeek4PS/bc2adls] Fabric - CopyBusinessCentral.ipynb - multi-company export (Issue #62)

@lhammerbhhttps://github.com/lhammerbh in the newest version in the repro the bug is fixed with the modified fields and the deleted records.

- Reply to this email directly, view it on GitHubhttps://github.com/Bertverbeek4PS/bc2adls/issues/62#issuecomment-1831780069, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQRDX6OL6KZ3JIPDG7H4VO3YG4RF5AVCNFSM6AAAAAA77GNSKOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZRG44DAMBWHE. You are receiving this because you were mentioned.Message ID: @.**@.>>