Bertverbeek4PS / bc2adls

Exporting data from Dynamics 365 Business Central to Azure data lake storage or MS Fabric lakehouse
MIT License
49 stars 20 forks source link

Last export state stops on "In process" #21

Closed Arthurvdv closed 1 year ago

Arthurvdv commented 1 year ago

Sometimes the Last exported state seems not to be updated to Succes.

image

It's difficult to describe the repro steps to create this situation. The good news is that in a next run it seems that this issue will resolve itself, where it's not blocking issue.

Bertverbeek4PS commented 1 year ago

Never had that experience. Even a refresh doesn;t help?

Arthurvdv commented 1 year ago

The problem seems to be that the Ended field on the ADLSE Run (82566) table isn't populated. It seems we're not reaching the RegisterEnded method in the "ADLSE Run" table.

I was hoping to find a result on the ADLSE-035 Telemetry, but no luck.

image

Bertverbeek4PS commented 1 year ago

With your newest pull request. Is that resolving the issue also?

Arthurvdv commented 1 year ago

That was not the intent the pull-request and unfortunately the issue still persists.

image

I'll try to investigate more in depth if I can find why this is failing sometimes.

Arthurvdv commented 1 year ago

image

Based on the telemetry it seems the SetStateFinished method failed somehow. I'm going to add some more telemetry and monitor the next couple of days if I can pinpoint the exact location where it's failing.

codeunit 82561 "ADLSE Execute"

trigger OnRun()
...

    // update Jsons
    ADLSECommunication.UpdateCdmJsons(EntityJsonNeedsUpdate, ManifestJsonsNeedsUpdate);
    if EmitTelemetry then
        ADLSEExecution.Log('ADLSE-007', 'Jsons have been updated', Verbosity::Normal, CustomDimensions);

    // Finalize
    SetStateFinished(Rec, TableCaption);
    if EmitTelemetry then
        if ExportSuccess then
            ADLSEExecution.Log('ADLSE-005', 'Export completed without error', Verbosity::Normal, CustomDimensions)
        else
            ADLSEExecution.Log('ADLSE-040', 'Export completed with errors', Verbosity::Warning, CustomDimensions);
end;
Arthurvdv commented 1 year ago

Issue has been resolved: https://github.com/Bertverbeek4PS/bc2adls/commit/7653e073438e68805951e530b507f0355b0d5c22