search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.12k stars 21.19k forks source link

Edge Blob Storage: deviceToCloudUpload not robust #85217

Closed fswane closed 2 years ago

fswane commented 2 years ago

Good morning

We are using Edge Blob Storage extensively in production but are running into robustness/reliability issues with regards to uploading of the blobs from the edge to the cloud.

Our use case is as follows:

1) At the edge, we have applications that generate CSV files and JPEG files. (The products are machine vision based smart sensors.)

2) These files are uploaded to the Edge Blob Storage using the Python SDK.

3) The Edge Blob Storage is configured to have two containers:

i) A blob container for the CSV files.

ii) A blob container for the JPEG files.

4) Both edge containers are configured to uploaded blobs to the same cloud Blob Storage.

5) The JPEG and CSV files are uploaded to the cloud, BUT...

6) PROBLEM: Not all files are uploaded to the cloud Blob Storage.

i) Almost 100% of the JPEG files are uploaded successfully.

ii) Only about 70% of the CSV files are uploaded successfully. THIS IS THE MAJOR PROBLEM.

More context:

  1. On the edge, blobfuse is used to access blobs on the Edge Blob Storage for analytics. (https://github.com/Azure/azure-storage-fuse)
  2. CSV files are created at the edge at about 2 CSV files per second. Average CSV size is 2.5 kbytes.
  3. JPEG files are created at the edge at about 1 JPEG per second. Average JPEG size is: 80 kbytes
  4. Azure Functions with a blobtrigger are used to process the CSV and JPEG files as they arrive in the cloud Blob Storage.
  5. The operating system of the edge device hosting the Edge Blob Storage: Ubuntu 18.04.4 LTS
  6. The Iot Edge architecture of the device hosting the Edge Blob Storage: iotedge 1.0.8-2 amd64 Azure IoT Edge Security Daemon
  7. Edge Blob Storage container: mcr.microsoft.com/azure-blob-storage:latest (running version is: 1.4.1.0 ) (I just see a 1.4.2 container is now available, will this help perhaps with this problem?)
  8. Edge Blob Storage configuration:

"deviceToCloudUploadProperties": { "uploadOn": true, "uploadOrder": "NewestFirst", "cloudStorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=s3analyticsimages;AccountKey=HIDDEN;EndpointSuffix=core.windows.net", "storageContainersForUpload": { "imagestore": { "target": "imagestore" }, "csvstore": { "target": "csvstore" } }, "deleteAfterUpload": false }, "deviceAutoDeleteProperties": { "deleteOn": true, "deleteAfterMinutes": 14400, "retainWhileUploading": true },

9) Edge Blob Storage create options:

{ "Env": [ "LOCAL_STORAGE_ACCOUNT_NAME=s3analyticsimages", "LOCAL_STORAGE_ACCOUNT_KEY=HIDDEN" ], "HostConfig": { "Binds": [ "/blobroot:/blobroot" ], "PortBindings": { "11002/tcp": [ { "HostPort": "11002" } ] } } } 10) A portion of a log file of the Edge Blob Storage is attached. I don't see anything in the logs related to this problem. edge-blob-store_2021_11_15.log

Questions:

1) Do you have any suggestions on improving the robustness of edge to cloud sync/upload using the Edge Blob Storage?

2) Is this a known issue?

3) Is there a way to instruct the Edge Blob Storage to refresh is database state? I.e. validate that its database and filesystem storage are in sync.

4) Is there a way to instruct the Edge Blob Storage to re-upload to cloud some or all blobs?

Any feedback or ideas to improve the robustness of edge to cloud upload of blobs would be much appreciated.

Regards

Francois

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

shashishailaj commented 2 years ago

@fswane We are sorry that you have to face this . We generally use this repo for issues related to existing documentation . In this case the issue will require considerable troubleshooting . I have assigned it to the team and they will try to help however if it requires deeper investigation they will help you with alternate support options. In future you can always open a thread on Microsoft QnA where we have a vibrant community who can help you on these kind of issues.

Thank you.

fswane commented 2 years ago

Thanks @shashishailaj for the response. I am more than happy to take this query to a more appropriate channel (I have tried various emails to absiotfeedback@microsoft.com which I thought was the correct place without any response).

Should I now post it here

https://docs.microsoft.com/en-us/answers/topics/25493/azure-iot-edge.html

or wait on feedback from you?

Regards Francois

AshokPeddakotla-MSFT commented 2 years ago

@fswane As mentioned by Shashi, We try to reserve these issues for feedback directly related to the tutorials: like if there are errors that block tutorial completion or if there are updates/changes required, suggestions for improvement, etc. If this is not the case I would suggest you open an issue with your question on the Edge repository or Microsoft QnA or StackOverflow forums where there is a community that can help (plus the engineering team).

Since you are having issues with data and required further troubleshooting to find out the root cause, Please file a support request @ https://aka.ms/azsupt? If you do not have access to a support plan, please reach out @ AZCommunity@microsoft.com with a link to this Issue as well as your subscription ID and we can help get the support ticket opened for this issue.

Let me know if you need further help.

AshokPeddakotla-MSFT commented 2 years ago

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.