search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.29k stars 21.47k forks source link

Container not found. Error : 39987 #44894

Closed tanieee28 closed 4 years ago

tanieee28 commented 4 years ago

Hi, I was deploying azure storage on iot edge via marketplace image of azure storage (mcr.microsoft.com/azure-blob-storage) After deploying the container on iotedge, it was showing running but when i checked the logs i found out this :

Starting Azure Blob Storage on IoT Edge, version 1.3.0.0.
[2019-12-18 14:42:09.284] [info    ] [tid 1] [MetaStore.cc:558] [Initialize] Existing DB version:1, current version:1
[2019-12-18 14:42:09.288] [info    ] [tid 40] [main.cc:77] [GCUncommittedBlocks] GCUncommittedBlocks thread starts. UncommittedBlocksExpiryTimeInMinutes:1440
[2019-12-18 14:42:09.288] [info    ] [tid 40] [main.cc:89] [GCUncommittedBlocks] GCUncommittedBlocks thread sleeps. UncommittedBlocksExpiryTimeInMinutes:1440
[2019-12-18 14:42:10.351] [info    ] [tid 1] Microsoft.Azure.Devices.BlobStorage.Common.EventGridConfiguration: Event Grid endpoint not specified, events turned off.
[2019-12-18 14:42:10.457] [info    ] [tid 1] Microsoft.AzureStack.Services.Storage.EntryPoint.BlobService: BlobService - In StartAsync
[2019-12-18 14:42:10.968] [info    ] [tid 1] Info_debug: Loading config Param {0} ({1}) read: {2}, p0="NephosAllowPathStyleUris", p1="Allow path-style URIs", p2="true"
[2019-12-18 14:42:10.969] [info    ] [tid 1] Info_debug: Successfully loaded {0}: {1}, p0="NephosAllowPathStyleUris", p1="True"
[2019-12-18 14:42:10.980] [info    ] [tid 1] Info_debug: Loading config Param {0} ({1}) read: {2}, p0="Nephos.MaskClientIPAddressesInLogs", p1="Indicates whether client IP addresses should be masked in logs.", p2="false"
[2019-12-18 14:42:10.981] [info    ] [tid 1] Info_debug: Successfully loaded {0}: {1}, p0="Nephos.MaskClientIPAddressesInLogs", p1="False"
[2019-12-18 14:42:10.981] [info    ] [tid 1] Info_debug: Loading config Param {0} ({1}) read: {2}, p0="NephosIncludeInternalDetailsInErrorResponses", p1="Include internal details in error responses", p2="true"
[2019-12-18 14:42:10.981] [info    ] [tid 1] Info_debug: Successfully loaded {0}: {1}, p0="NephosIncludeInternalDetailsInErrorResponses", p1="True"
[2019-12-18 14:42:10.994] [info    ] [tid 1] Info_debug: Loading config Param {0} ({1}) read: {2}, p0="StampName", p1="Stamp Name", p2="Default Stamp"
[2019-12-18 14:42:11.893] [info    ] [tid 1] Microsoft.AzureStack.Services.Storage.EntryPoint.BlobService: BlobService - StartAsync completed
[2019-12-18 14:42:11.922] [info    ] [tid 1] Microsoft.Azure.Devices.BlobStorage.Tiering.BlobTieringService: Starting service...
[2019-12-18 14:42:12.123] [info    ] [tid 1] [BlobInterface.cc:1490] [ListBlobsInOrder] ListBlobsInOrder received. Container:imageblob BlobNameStart:null MaxBlobNames:1 OrderType:0 Flags:1
[2019-12-18 14:42:12.123] [error   ] [tid 1] [MetaStore.cc:1928] [ListBlobsInOrder] Container not found. Name:imageblob
[2019-12-18 14:42:12.123] [error   ] [tid 1] [BlobInterface.cc:1528] [ListBlobsInOrder] ListBlobsInOrder failed. Container:imageblob Error:39987
[2019-12-18 14:42:12.159] [info    ] [tid 1] Microsoft.Azure.Devices.BlobStorage.Ttl.BlobTtlService: Starting service...
[2019-12-18 14:42:12.326] [info    ] [tid 34] [ContainerInterface.cc:148] [ListBlobContainers] ListContainers received. ContainerNamePrefix:null ContainerNameStart:null MaxContainerNames:10
[2019-12-18 14:42:12.327] [info    ] [tid 34] [ContainerInterface.cc:168] [ListBlobContainers] ListContainers completed. ContainerEntryCount:0 NextContainerName:null
Application started. Press Ctrl+C to shut down.
Hosting environment: Production
Content root path: /azure-blob-storage/
[2019-12-18 14:42:13.147] [info    ] [tid 20] [BlobInterface.cc:1490] [ListBlobsInOrder] ListBlobsInOrder received. Container:imageblob BlobNameStart:null MaxBlobNames:1 OrderType:0 Flags:1
[2019-12-18 14:42:13.147] [error   ] [tid 20] [MetaStore.cc:1928] [ListBlobsInOrder] Container not found. Name:imageblob
[2019-12-18 14:42:13.147] [error   ] [tid 20] [BlobInterface.cc:1528] [ListBlobsInOrder] ListBlobsInOrder failed. Container:imageblob Error:39987
[2019-12-18 14:42:14.149] [info    ] [tid 20] [BlobInterface.cc:1490] [ListBlobsInOrder] ListBlobsInOrder received. Container:imageblob BlobNameStart:null MaxBlobNames:1 OrderType:0 Flags:1
[2019-12-18 14:42:14.149] [error   ] [tid 20] [MetaStore.cc:1928] [ListBlobsInOrder] Container not found. Name:imageblob
[2019-12-18 14:42:14.149] [error   ] [tid 20] [BlobInterface.cc:1528] [ListBlobsInOrder] ListBlobsInOrder failed. Container:imageblob Error:39987
[2019-12-18 14:42:15.149] [info    ] [tid 20] [BlobInterface.cc:1490] [ListBlobsInOrder] ListBlobsInOrder received. Container:imageblob BlobNameStart:null MaxBlobNames:1 OrderType:0 Flags:1
[2019-12-18 14:42:15.150] [error   ] [tid 20] [MetaStore.cc:1928] [ListBlobsInOrder] Container not found. Name:imageblob
[2019-12-18 14:42:15.150] [error   ] [tid 20] [BlobInterface.cc:1528] [ListBlobsInOrder] ListBlobsInOrder failed. Container:imageblob Error:39987
[2019-12-18 14:42:16.151] [info    ] [tid 20] [BlobInterface.cc:1490] [ListBlobsInOrder] ListBlobsInOrder received. Container:imageblob BlobNameStart:null MaxBlobNames:1 OrderType:0 Flags:1

Now, i am not sure why this is happening after following all steps from below azure docs correctly

Container Create Options :

{
    "HostConfig": {
        "Binds": [
            "local-image-repository:/blobroot"
        ],
        "PortBindings": {
            "11002/tcp": [
                {
                    "HostPort": "11002"
                }
            ]
        }
    }
}

Module Twin Settings :

{
    "deviceAutoDeleteProperties": {
        "deleteOn": true,
        "deleteAfterMinutes": 7200,
        "retainWhileUploading": false
    },
    "deviceToCloudUploadProperties": {
        "uploadOn": true,
        "uploadOrder": "OldestFirst",
        "cloudStorageConnectionString": "SENSITIVE CONTENT",
        "storageContainersForUpload": {
            "imageblob": {
                "target": "imageblob"
            }
        },
        "deleteAfterUpload": false
    }
}

Can somebody help here ?

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

AshokPeddakotla-MSFT commented 4 years ago

@tanieee28 Please help with the link of document on which above document feedback request is created. Under this repository we want to make sure any issue is linked to a doc so we can help others that may be having the same issue.

tanieee28 commented 4 years ago

@AshokPeddakotla-MSFT I have followed the below document for deploying blob storage on IoT Edge :

https://docs.microsoft.com/en-us/azure/iot-edge/how-to-deploy-blob

AshokPeddakotla-MSFT commented 4 years ago

@tanieee28 I followed the documentation and the config which you have provided and able to deploy module successfully without any issues. Could you please provide more detailed steps to repro your issue? Where are you seeing this error message? @arduppal Do you have any suggestions for this issue?

AshokPeddakotla-MSFT commented 4 years ago

@tanieee28 Have you had a chance to see my response? Do let us know the requested details for further help.

tanieee28 commented 4 years ago

Hi @AshokPeddakotla-MSFT . Sorry for delayed response.

I have also followed the same document and still when deploying the module, after module start the above mentioned error logs are coming ( Error 39987)

Let me explain the scenario again,

  1. I am deploying the blob on iotedge with the configuration mentioned above in the issue.
  2. I have an azure blob storage on AZURE whose connection string i have given in the same configuration.
  3. After successful deployment, the module is showing running but in the logs i am seeing that the module is not able to find the local container named as imageblob, and it is showing the error as "Container Not Found"
 "storageContainersForUpload": {
            "imageblob": {      //local container name
                "target": "imageblob"   //remote container name
            }
}

More Info: I am doing this on Raspberry Pi 3 ( OS - raspbian stretch) iotedge -- latest version

AshokPeddakotla-MSFT commented 4 years ago

@tanieee28 Thanks for the details. I am assigning this to the author @arduppal for further assistance.

AshokPeddakotla-MSFT commented 4 years ago

@arduppal Please check this issue and provide your suggestions.

mbialecka commented 4 years ago

@tanieee28 Have you created storage container "imageblob"? This error is to be expected on a fresh deployment when you haven't created any containers/blobs yet. The upload module is trying to push out blobs based on the configuration you specified but it looks like there is nothing to push out yet.

Let me know if you have any questions.

AshokPeddakotla-MSFT commented 4 years ago

@tanieee28 Does the above suggestions helped you resolve the issue? Do let us know if you need further help.

tanieee28 commented 4 years ago

@mbialecka Sorry for delayed response.

Yes, its a fresh deployment and what i was expecting is that local blob module should create the local blob container by itself and should not through above logs stating that it is not able to find it.

However, With some help i was able to resolve this by using Azure storage explorer. So, i connected the storage explorer with the local blob storage using the connection string and later created the blob container named "imageblob" from the storage explorer functionalities.

But, i am still concerned about the local blob module as its the module's work to create an internal blob container from given configs. Above solution can be a workaround but its not a solution and it will be helpful if the owner of this module fixes this in future.

Thanks

arduppal commented 4 years ago

@tanieee28- Thanks for the feedback, will look at it in our next update.

arduppal commented 4 years ago

@AshokPeddakotla-MSFT : Can you close this issue?

AshokPeddakotla-MSFT commented 4 years ago

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

0xarmingithub commented 3 years ago

Hi. I have the same problem on a fresh deployment. I have created a docker volume called "storagevolume". Then below is my deployment settings:

Container create options: { "Env": [ "LOCAL_STORAGE_ACCOUNT_NAME=localstorage", "LOCAL_STORAGE_ACCOUNT_KEY=KEY" ], "HostConfig": { "Binds": [ "storagevolume:/blobroot" ], "PortBindings": { "11002/tcp": [ { "HostPort": "11002" } ] } } }

Module twin settings: { "deviceAutoDeleteProperties": { "deleteOn": false, "deleteAfterMinutes": 35791, "retainWhileUploading": true }, "deviceToCloudUploadProperties": { "uploadOn": true, "uploadOrder": "OldestFirst", "cloudStorageConnectionString": "DefaultEndpointsProtocol=AZURE-STORAGE-ENDPOINT", "storageContainersForUpload": { "localblobstorage": { "target": "edgelocalstorage" } }, "deleteAfterUpload": false } }

The error I see is: [2021-06-29 10:02:00.719] [error ] [tid 1838] [MetaStore.cc:1953] [ListBlobsInOrder] Container not found. Name:localblobstorage [2021-06-29 10:02:00.719] [error ] [tid 1838] [BlobInterface.cc:1535] [ListBlobsInOrder] ListBlobsInOrder failed. Container:localblobstorage Error:39987 [2021-06-29 10:02:01.722] [info ] [tid 1838] [BlobInterface.cc:1494] [ListBlobsInOrder] ListBlobsInOrder received. Container:localblobstorage BlobNameStart:null MaxBlobNames:1 OrderType:0 Flags:1

I can clearly see that module cannot find the source container. But isn't it module's responsibility to create that container upon start?

tanieee28 commented 3 years ago

Yes, i agree and as from the comments from @arduppal it was expected to be added in next update. Till that time, you can create a container with the name specified ("localblobstorage") in your azureblobstorage module using azure storage explorer App. Here is the link to get the app, https://azure.microsoft.com/en-in/features/storage-explorer/

0xarmingithub commented 3 years ago

Thanks @tanieee28. Actually I was not successful in manually creating local storage via storage explorer. I will redo and if I face same issue will write here.

@arduppal, any timeline on the updated version? I'm using version 1.4.1.0.

0xarmingithub commented 3 years ago

I can't manually create container using the storage explorer. I'm using the following connection string to connect to storage account:

DefaultEndpointsProtocol=http;BlobEndpoint=http://192.168.0.102:11002/localstorage;AccountName=localstorage;AccountKey=<account-key>==;

I face the following error:

`Unable to retrieve child resources.

Details: { "name": "RestError", "message": "One of the request inputs is not valid.\nRequestId:3e3f5d3b-3fc0-4baf-9965-931bb3f569f8\nTime:2021-06-29T21:27:05.4881822Z", "stack": "RestError: One of the request inputs is not valid.\nRequestId:3e3f5d3b-3fc0-4baf-9965-931bb3f569f8\nTime:2021-06-29T21:27:05.4881822Z\n at new RestError (C:\Program Files (x86)\Microsoft Azure Storage Explorer\resources\app\node_modules\@azure\core-http\dist\index.js:2362:28)\n at handleErrorResponse (C:\Program Files (x86)\Microsoft Azure Storage Explorer\resources\app\node_modules\@azure\core-http\dist\index.js:3246:17)\n at C:\Program Files (x86)\Microsoft Azure Storage Explorer\resources\app\node_modules\@azure\core-http\dist\index.js:3182:18\n at processTicksAndRejections (internal/process/task_queues.js:97:5)", "code": "InvalidInput", "statusCode": 400, "request": { "streamResponseStatusCodes": {}, "url": "http://192.168.0.102:11002/?include=metadata&comp=list", "method": "GET", "headers": { "_headersMap": { "x-ms-version": { "name": "x-ms-version", "value": "2020-04-08" }, "user-agent": { "name": "user-agent", "value": "Microsoft Azure Storage Explorer, 1.19.1, win32 azsdk-js-storageblob/12.4.1 (NODE-VERSION v12.16.3; Windows_NT 10.0.18363)" }, "x-ms-client-request-id": { "name": "x-ms-client-request-id", "value": "203d8317-1183-47d7-9d6f-26b9b4f7beec" }, "x-ms-date": { "name": "x-ms-date", "value": "Tue, 29 Jun 2021 21:27:05 GMT" }, "authorization": { "name": "Authorization", "value": "SharedKey localstorage:qAgjrg6rMCNEFxQsg+cVT8czp72qnPZvl0zpffeev7Y=" }, "cookie": { "name": "Cookie", "value": "" } } },`

tanieee28 commented 3 years ago

Yes, I checked this in my environment and saw the same error. let me try to debug this. @AshokPeddakotla-MSFT Request your help on this incase its an issue from some new updates.

0xarmingithub commented 3 years ago

Any update on this from MS? @AshokPeddakotla-MSFT @arduppal

SatishBoddu-MSFT commented 3 years ago

@arminghasemazar Could you please let us know what is the error you were facing when using Azure Storage Explorer to create the container manually?

0xarmingithub commented 3 years ago

@SatishBoddu-MSFT Already wrote here.

I can't manually create container using the storage explorer. I'm using the following connection string to connect to storage account:

DefaultEndpointsProtocol=http;BlobEndpoint=http://192.168.0.102:11002/localstorage;AccountName=localstorage;AccountKey=<account-key>==;

I face the following error:

`Unable to retrieve child resources.

Details: { "name": "RestError", "message": "One of the request inputs is not valid.\nRequestId:3e3f5d3b-3fc0-4baf-9965-931bb3f569f8\nTime:2021-06-29T21:27:05.4881822Z", "stack": "RestError: One of the request inputs is not valid.\nRequestId:3e3f5d3b-3fc0-4baf-9965-931bb3f569f8\nTime:2021-06-29T21:27:05.4881822Z\n at new RestError (C:\Program Files (x86)\Microsoft Azure Storage Explorer\resources\app\node_modules\@Azure\core-http\dist\index.js:2362:28)\n at handleErrorResponse (C:\Program Files (x86)\Microsoft Azure Storage Explorer\resources\app\node_modules\@Azure\core-http\dist\index.js:3246:17)\n at C:\Program Files (x86)\Microsoft Azure Storage Explorer\resources\app\node_modules\@Azure\core-http\dist\index.js:3182:18\n at processTicksAndRejections (internal/process/task_queues.js:97:5)", "code": "InvalidInput", "statusCode": 400, "request": { "streamResponseStatusCodes": {}, "url": "http://192.168.0.102:11002/?include=metadata&comp=list", "method": "GET", "headers": { "_headersMap": { "x-ms-version": { "name": "x-ms-version", "value": "2020-04-08" }, "user-agent": { "name": "user-agent", "value": "Microsoft Azure Storage Explorer, 1.19.1, win32 azsdk-js-storageblob/12.4.1 (NODE-VERSION v12.16.3; Windows_NT 10.0.18363)" }, "x-ms-client-request-id": { "name": "x-ms-client-request-id", "value": "203d8317-1183-47d7-9d6f-26b9b4f7beec" }, "x-ms-date": { "name": "x-ms-date", "value": "Tue, 29 Jun 2021 21:27:05 GMT" }, "authorization": { "name": "Authorization", "value": "SharedKey localstorage:qAgjrg6rMCNEFxQsg+cVT8czp72qnPZvl0zpffeev7Y=" }, "cookie": { "name": "Cookie", "value": "" } } },`

kgalic commented 3 years ago

Also having the same problem here. Creating manually the container via Storage Explorer is not an option.

ozgurcetinok commented 2 years ago

Any update on this issue?

mserigo commented 2 years ago

Same issue here... since the IoT device is not on my local network (it´s an Edge) I cannot use the Storage Explorer to create the container. Is there any update on this issue?

raza707mirza commented 2 years ago

I had similar issue with storage explorer. Then i figured out the issue in my connection string, I was missing account name in blob endpoint inside connection string Following is my connection string DefaultEndpointsProtocol=http;BlobEndpoint=http://127.0.0.1:11002/**gatewayftpfiles**;AccountName=gatewayftpfiles;AccountKey=<ACCOUNTKEY>

dalmasj commented 2 years ago

Same issue for me even with account anme in blob endpoint : DefaultEndpointsProtocol=http;BlobEndpoint=http://127.0.0.1:11002/devstoreaccount11;AccountName=devstoreaccount11;AccountKey=*****

Is there any update about this issue?

AnnaGuzy commented 1 year ago

@arduppal @AshokPeddakotla-MSFT

Any updates on this issue?

surajgampa95 commented 1 year ago

Any updates on this issue? @AshokPeddakotla-MSFT

tynevi commented 5 months ago

@tanieee28 - When you originally filed this issue, the info you provided contained an Azure Storage account access key. The key has been edited out of the issue and replaced with the <account-key> placeholder, but we recommend you revoke/rotate account keys to ensure that key cannot be used.