search

Bertverbeek4PS / bc2adls

Exporting data from Dynamics 365 Business Central to Azure data lake storage or MS Fabric lakehouse
MIT License
56 stars 22 forks source link

Bad Request - Invalid Verb #23

Closed Arthurvdv closed 1 year ago

Arthurvdv commented 1 year ago

When working with Microsoft Fabric as Storage Type starting a full run of multiple entities, some of them wil receive a HTTP 400 response with the error message Bad Request - Invalid Verb.

image Could not read data on https://onelake.dfs.fabric.microsoft.com/Business Central/BusinessCentral.Lakehouse/Files/PurchaseHeader-38.cdm.json. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd"> <HTML><HEAD><TITLE>Bad Request</TITLE> <META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD> <BODY><h2>Bad Request - Invalid Verb</h2> <hr><p>HTTP Error 400. The request verb is invalid.</p> </BODY></HTML> "ADLSE Gen 2 Util"(CodeUnit 82568).GetBlobContent line 22 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Communication"(CodeUnit 82562).CheckEntity line 14 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Execute"(CodeUnit 82561).TryExportTableData line 13 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Execute"(CodeUnit 82561).OnRun(Trigger) line 48 - Azure Data Lake Storage Export by The bc2adls team\

In the example above 41 tables where only 3 of them receives this error. On a next run these failed entities are then successful, which points me in the direction that maybe exceeding a threshold of API calls within a certain timespan?

Bertverbeek4PS commented 1 year ago

Moslty the error message is that the HTTP request is not supported by the server. But in this case it is a GET request and that is supported by the server. image

Bertverbeek4PS commented 1 year ago

Maybe it is good to split the metadata creation and delta into two seperate functions and not into one.

Arthurvdv commented 1 year ago

Another error message, probably created with the same use case; When removing all the files in the Lakehouse and do a full reset in Business Central, there's a reasonable certainty you're going to have an error on some entities.

For example the error below, where the file is present in the Lakehouse, it fails on the GetBlobContentLength method.

image Could not read data on https://onelake.dfs.fabric.microsoft.com/Business Central/BusinessCentral.Lakehouse/Files/deltas/SalesCrMemoLine-115/87A25E82-F28A-4A27-B308-D3F6C527646D.csv. "ADLSE Gen 2 Util"(CodeUnit 82568).GetBlobContentLength line 12 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Communication"(CodeUnit 82562).FlushPayload line 35 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Communication"(CodeUnit 82562).CollectAndSendRecord line 18 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Communication"(CodeUnit 82562).TryCollectAndSendRecord line 6 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Execute"(CodeUnit 82561).ExportTableUpdates line 44 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Execute"(CodeUnit 82561).TryExportTableData line 15 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Execute"(CodeUnit 82561).OnRun(Trigger) line 48 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Execute"(CodeUnit 82561).ExportTableUpdates line 48 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Execute"(CodeUnit 82561).TryExportTableData line 15 - Azure Data Lake Storage Export by The bc2adls team\"ADLSE Execute"(CodeUnit 82561).OnRun(Trigger) line 48 - Azure Data Lake Storage Export by The bc2adls team\

image

I'm doubting if we could handle these exceptions in the ADSLE extensions itself. Knowing that Microsoft Fabric is still in preview, it could be that the API endpoint will improve and these issues will be resolved?

To investigate we could implement Telemetry on the InvokeRestApi method when the .Send returns a failure? Ideally all the details like the Method, RequestUri, Payload, etc. Note sure on howto handle possible sensitive information on this.

Bertverbeek4PS commented 1 year ago

I think it is the OneLake API. And if you implement a retry if we got that message? Resolve that?

Arthurvdv commented 1 year ago

I didn't encounter these error any more for the past two weeks. I believe this is indeed the OneLake API and combined with a initial run of all the entities these error could occur.

I'm a bit hesitated to implementing a retry functionality, because of the increasing complexity of the API handeling. For now I'll close this issue.