search

Bertverbeek4PS / bc2adls

Exporting data from Dynamics 365 Business Central to Azure data lake storage or MS Fabric lakehouse
MIT License
49 stars 20 forks source link

Do multi-company exports still work? #94

Closed taylor-moran closed 4 months ago

taylor-moran commented 6 months ago

Hello,

When trying to start exports of BC companies from Power Automate, I'm getting the following error:

"ErrorCode: Application_DialogException An export data process is already running. Please wait for it to finish."

In past versions we were able to start exports for multiple companies in Power Automate without waiting for each to finish exporting data. It seemed to work when we used the Multi-company export flag which is being deprecated in current versions.

I saw from a previous issue that the flag was removed since it was only a check on if the schema exported has changed or not, but did it also allow for multiple companies to export simultaneously?

From the FAQ:

"A field called Multi- company export was added in https://github.com/microsoft/bc2adls/pull/47 to improve concurrency for parallel exports from different companies."

Please let us know if we are missing something when it comes to multi-company exports.

Taylor

greglong1 commented 6 months ago

If you look at my closed issue #76, my understanding was that the multi-company export flag only worked to prevent schema exports. However I'm interested in how you are using Power Automate to trigger exports (either within the Export to Data Lake window or with Job Queue Entries. Are you able to link to documentation that achieves this?

Bertverbeek4PS commented 6 months ago

Hi @taylor-moran, It is what @greglong1 says. It is only prefent to change the schema export and nothing more. You have to create in each company a job queue to export your data. Or you can create a Power Automate flow to trigger the export.

The error that you get if from the check on the export if there are any export sessions in table: "ADLSE Current Session" And does a check if there is a session active: image

That code isn't changed.

taylormoran commented 6 months ago

Hi @Bertverbeek4PS,

The code isn't exactly the same as version 1.3.15.0 which we are using still using in prod.

The first line in the procedure didn't exist previously: ActiveSession.SetLoadFields("Session Unique ID");

Could this be causing issues when trying to export from multiple companies simultaneously if sessions aren't unique?

@greglong1 this is what the flow basically looks like. It includes multiple Run Actions for BC https://learn.microsoft.com/en-us/connectors/dynamicssmbsaas/#run-action-(v3). One action per company as follows:

image

With the Asynchronous Pattern turned on in the flow actions and Multi-company exports turned on in the extension we haven't had issues in production that I remember. But when testing recent versions with the flag deprecated it seems we get the error now.

image

Taylor

Bertverbeek4PS commented 6 months ago

@taylormoran that line is only to load the data for checking faster. Not anything new in that one.

I need to deepdive into this one. Because it hasn't changed conceptually,

taylormoran commented 6 months ago

Thank you! I appreciate your time and efforts on this project. We uninstalled the extension from the sandbox we are using for this and went back to 1.3.15.0 and the flow is working when exporting multiple companies.

When trying to upgrade to 2.18 it gave us an error about schemas not being exported so ran schema export in Setup page and tried exporting again and received the original error "ErrorCode: Application_DialogException An export data process is already running. Please wait for it to finish." Maybe we aren't doing something right when upgrading from 1.3.15.0?

Bertverbeek4PS commented 6 months ago

@taylormoran sorry for my late answer. But I'm currently on a business trip this week and also next week. So expect some delay.

Bertverbeek4PS commented 6 months ago

@taylormoran I been back from all the events and I have tested it this morning and I don't get the error on the latest release. I run it in parralel: image And everything it OK. Also in Fabric you can see that both are exported the same time: image

I think it is a good idea to also set the concurrency control to yes in degree of Parallism to one. In that case the flow would not run twice the same time. Because then you get the error.

If that isn't working. Maybe we can setup a call?

taylormoran commented 6 months ago

Thank you so much Bert. I will pick up testing again tomorrow with 2.21.

I spoke with a colleague recently and he said he has seen the "An export data process is already running" error before with the legacy version of bc2adls. But it only happened a couple times initially when we started exporting and hasn't returned since. Could be an environmental issue for us.

taylor-moran commented 6 months ago

Had a chance to briefly test exports tonight and didn't run into issues when running the start export action for companies in parallel in the flow. Then I tried running the actions again sequentially like we have in production currently and ran into the error again. The actions have asynchronous pattern turned on, "run after" is set only to successful, so the next action shouldn't start exporting the next company until 200 is returned from the API for the previous action/export. Will dig into it more.

Bertverbeek4PS commented 6 months ago

@taylormoran is it possible to share your flow? Because I'm curious what causes that issue.

y0m0 commented 4 months ago

@Bertverbeek4PS I am facing the same issue as @taylormoran when running export from multiple companies in parallel. image

The flow usually fails when there is a large amount of data to be exported, meaning that the export takes a few seconds instead of only a few milliseconds. image

The same error is being thrown when running the export by using job queues for multiple companies all starting at the same time.

The error comes from the already mentioned procedure IsSessionActive()

rem-bou commented 4 months ago

Apologies for the question that isn't directly relevant to issue. I am new to BC2ADLSE and I am trying to export table from BC to Fabric using this awesome extension.

What do we need to use for the SystemId field in Power Automate task please?

image

y0m0 commented 4 months ago

@rem-bou the systemId of the ADLSE Setup record, you can get it by inspecting the page "Export to Azure Data Lake Storage". That's what I used, and it's working.

Bertverbeek4PS commented 4 months ago

@y0m0 sorry for the delay. A little bit busy preparing all the events. I will dive into it. It is very strange behaviour.

Bertverbeek4PS commented 4 months ago

@y0m0 I have fixed two issues in the following branche: https://github.com/Bertverbeek4PS/bc2adls/tree/94-do-multi-company-exports-still-work Would you mind to test it? Because I don't have currently an environment with that large companies.

rem-bou commented 4 months ago

@rem-bou the systemId of the ADLSE Setup record, you can get it by inspecting the page "Export to Azure Data Lake Storage". That's what I used, and it's working.

@y0m0 Are you talking about the Application ID? If so, I tried this one and can not make it work.

If not, would you be able to share the class it is under? I tried to inspect the page as recommended but couldn't find any guuid or systemId

y0m0 commented 4 months ago

@Bertverbeek4PS

It looks like it now works as intended. I tested it by creating a new lakehouse and running the export in parallel for 5 companies. I tried both via power automate flows and via job queues all starting at the exact same time, and it worked in both cases.

As a double check I reverted to the previous commit, and tested again with power automate and job queues and sure enough the error was being thrown again.

Thanks for the fix :)

y0m0 commented 4 months ago

@rem-bou simply navigate to the page "Export to Azure Data Lake" and inspect the page by either pressing CTRL + ALT + F1 or by clicking on ? -> Help and Support -> Inspect pages and data.

image

scroll down and you should find systemId

image

triple click on the value and it should select it all and then simply ctrl+c

rem-bou commented 4 months ago

@y0m0: Thank you! I didn't think at that - I was looking at inspecting the webpage code using "developer tools". Anyways, this is working for me now. Thank you very much for taking the time to help me out!

Bertverbeek4PS commented 4 months ago

@Bertverbeek4PS

It looks like it now works as intended. I tested it by creating a new lakehouse and running the export in parallel for 5 companies. I tried both via power automate flows and via job queues all starting at the exact same time, and it worked in both cases.

As a double check I reverted to the previous commit, and tested again with power automate and job queues and sure enough the error was being thrown again.

Thanks for the fix :)

Thanks @y0m0 for testing it for me!! I will merge it into the main and releases. Thanks!!

y0m0 commented 4 months ago

@Bertverbeek4PS Thanks to you for the time you are putting in to keep the project alive. If you are at TechDays this year, I'll definitively buy you a beer or two ;)

Bertverbeek4PS commented 4 months ago

@Bertverbeek4PS Thanks to you for the time you are putting in to keep the project alive. If you are at TechDays this year, I'll definitively buy you a beer or two ;)

Thanks @y0m0 . But also the community is really great to help with it! I'm at techdays indeed 😂