Multi-company exports and Power Automate flow

[taylor-moran]

Hello,

Very happy to see this project live on here!

For the available Power Automate flow, would there be an issue if multiple companies are being exported simultaneously when it comes to triggering the consolidation pipeline for a table? For example, if the vendor table is being exported for multiple companies at the same, would the consolidation pipeline be triggered multiple times for the vendor table each time a company finishes exporting causing multiple consolidations to happen to the vendor table in Synapse at the same time? The staging process and the deleting deltas by the pipeline if it were to be run simultaneously for a table could cause issues.

Any insight would be greatly appreciated!

[taylor-moran]

I see now in the flow the consolidation pipeline will only allow one run per table/entity at a time. It will not allow the pipeline to run simultaneously for a table being exported by multiple companies.

But would it be possible for a consolidation to kick off for a table after a company finished exporting at the same time the table is being exported by another company to the delta folder?

[Bertverbeek4PS]

Hi @taylor-moran. If you do a multi company export on Azure Data Lake you must first finish all the export in all companies. If that is done you can run the Synapse pipelines. Otherwise you can maybe process the file that is in progress of the export from BC.

[taylor-moran]

@Bertverbeek4PS thanks, Bert! So far I modified the On Export action to include Table Name and will try tracking exports and their status for Company/Table combinations in a SQL table. This way a consolidation for an entity won't begin unless all companies have finished processing the entity first.

The event driven approach using these bc2adls events is fantastic!