microsoft / bc2adls

Exporting data from Dynamics 365 Business Central to Azure data lake storage
MIT License
92 stars 53 forks source link

Procedure UpdateCdmJsons is failing because of LockTable in a try function #130

Open RonKoppelaar opened 1 year ago

RonKoppelaar commented 1 year ago

When running the tool (at least on BC22.3) its failing on procedure UpdateCdmJsons. This procedure is executed as part of a try function. Below error was found in the eventLog of my local container.

My guess is that the idea behind this lock table is to prevent different background tasks writing simultaneously to the storage account for updating the manifest. But unfortune this lead to below error. Maybe the LockTable inside a try function has changed in recent BC versions? Not sure.

Best option I guess: Not implementing as TryFunction. This might lead to errors on sync process of a table. But with current implementation it will just keep the error away, and only difference in the output is telemetry not shown.

Steps to reproduce:

Table will stay in progress for ever... Check out the eventlog of the container. Below error is reported.

Running the same repro steps in an Online sandbox will succeed. It looks like Try functions with a Lock table are handled different in cloud compared to Onpremis.

EventLog/Error

Server instance: BC Tenant ID: default Environment Name: Environment Type: Sandbox Session type: Background Session ID: 1087 User: Type: Microsoft.Dynamics.Nav.Types.Exceptions.NavNclTryFunctionWriteException ErrorCode: -1 ShouldCalculateALCallStack: True SuppressMessage: False ContainsPersonalOrRestrictedInformation: True DiagnosticsSuppress: False DiagnosticsMessage: Message not shown because the NavBaseException constructor was used without privacy classification MessageWithoutPrivateInformation: Message not shown because the NavBaseException constructor was used without privacy classification SuppressExceptionCreatedEvent: False FatalityScope: None ErrorLevel: Error ALCallStack: "ADLSE Communication"(CodeUnit 82562).UpdateCdmJsons line 20 - Azure Data Lake Storage Export by cegeka "ADLSE Communication"(CodeUnit 82562).TryUpdateCdmJsons line 2 - Azure Data Lake Storage Export by cegeka "ADLSE Execute"(CodeUnit 82561).OnRun(Trigger) line 82 - Azure Data Lake Storage Export by cegeka

Message: Call to the function 'LOCKTABLE' is not allowed inside the call to 'TryUpdateCdmJsons' when it is used as a TryFunction. Source: Microsoft.Dynamics.Nav.Ncl HResult: -2146233088 StackTrace:

RonKoppelaar commented 1 year ago

I got feedback from the MS Server team regarding the difference in behavior: LockTable in a Try function using Docker/Onpremis ==> Will fail LockTable in a Try function in the online service ==> Works (for now...)

Feedback from the server team in regards to this subject: The docker behavior is the correct one – it should raise an error. We want to run with the same config in production, but changing the behavior in PROD would cause disruption for customers, so we are working on getting that code updated

Its a matter of time this will also start to fail in Online. Looking at the code, the updating of changed json CDM definitions is a mandatory step. Skipping it would lead to a successful synchronization of delta's but the cdm definition could be wrong. Proposed solution is to remove the TryFunction mechanism and just perform a normal invocation of the method. If it fails the table is marked as error instead of completed. Action could be redone next time. The lock table in the end is a mandatory step to prevent collisions of sessions writing to the same json file.