search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.29k stars 21.48k forks source link

Cannot send any data to blob storage - "Server encountered an internal error. Please try again after some time." #37980

Closed m3nowak closed 5 years ago

m3nowak commented 5 years ago

While using this container I've encountered an issue preventing me from uploading files. It is present when i try to upload file using Azure Storage Explorer and python library (azure-storage-blob==2.1.0).

Log from error in Azure Storage Explorer:

{
  "name": "StorageError",
  "message": "Server encountered an internal error. Please try again after some time.\nRequestId:aaabece5-c7a2-43d6-ac52-5138d5546ee0\nTime:2019-08-29T08:20:32.1070553Z",
  "stack": "StorageError: Server encountered an internal error. Please try again after some time.\nRequestId:aaabece5-c7a2-43d6-ac52-5138d5546ee0\nTime:2019-08-29T08:20:32.1070553Z\n    at Function.StorageServiceClient._normalizeError (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\azure-storage\\lib\\common\\services\\storageserviceclient.js:1205:23)\n    at Object.StorageServiceClient._processResponse (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\azure-storage\\lib\\common\\services\\storageserviceclient.js:751:50)\n    at Request.processResponseCallback [as _callback] (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\azure-storage\\lib\\common\\services\\storageserviceclient.js:319:37)\n    at Request.self.callback (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\request\\request.js:185:22)\n    at emitTwo (events.js:126:13)\n    at Request.emit (events.js:214:7)\n    at Request.<anonymous> (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\request\\request.js:1157:10)\n    at emitOne (events.js:116:13)\n    at Request.emit (events.js:211:7)\n    at IncomingMessage.<anonymous> (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\request\\request.js:1079:12)",
  "code": "InternalError",
  "exceptiondetails": {
    "ExceptionMessage": "No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed",
    "StackTrace": "Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStoreException: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.Blob.BlobClientException: Exception of type 'Microsoft.AzureStack.Services.Storage.Blob.BlobClientException' was thrown.\n   at Microsoft.AzureStack.Services.Storage.Blob.WBlobClient.<>c__DisplayClass8_0.<PutBlobAsync>b__0()\n   at System.Threading.Tasks.Task`1.InnerInvoke()\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobSingleChunk()\n   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobAsync()\n   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition)\n   --- End of inner exception stack trace ---\n   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition)\n   at Microsoft.AzureStack.Services.Storage.Util.Await(Task t)\n   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult)\n   --- End of inner exception stack trace ---\n   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.ExceptionManager.ReThrowIfWStoreException(Exception e)\n   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult)\n   at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.PutBlobImpl(IAccountIdentifier identifier, String account, String container, String blob, Stream inputStream, Int64 contentLength, IPutBlobProperties putBlobProperties, Boolean supportCrc64, Boolean calculateCrc64, Boolean storeCrc64, Boolean verifyCrc64, Boolean calculateMd5, Boolean storeMd5, Boolean verifyMd5, Boolean generationIdEnabled, Boolean isLargeBlockBlobRequest, Boolean is8TBPageBlobAllowed, BlobObjectCondition condition, OverwriteOption overwriteOption, TimeSpan timeout, RequestContext requestContext, AccessTier accessTier, String blobSasUrl, AsyncIteratorContext`1 context)+MoveNext()\n   at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)\n   --- End of inner exception stack trace ---\n   at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.EndPutBlob(IAsyncResult asyncResult)\n   at Microsoft.Cis.Services.Nephos.Storage.Service.Protocols.Rest.HttpRestProcessor.PutBlobImpl(AsyncIteratorContext`1 async)+MoveNext()\n   at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)\n   --- End of inner exception stack trace ---\n   at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.EndPerformOperation(IAsyncResult ar)\n   at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.ProcessImpl(AsyncIteratorContext`1 async)+MoveNext()"
  },
  "statusCode": 500,
  "requestId": "aaabece5-c7a2-43d6-ac52-5138d5546ee0"
}

Log from when using python library:

Traceback (most recent call last):

File "(...)", line 15, in put_data

BLOB_SERVICE.create_blob_from_bytes(date_str, data_id_str, data)

File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/blockblobservice.py", line 803, in create_blob_from_bytes

standard_blob_tier=standard_blob_tier, cpk=cpk)

File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/blockblobservice.py", line 633, in create_blob_from_stream

timeout=timeout)

File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/blockblobservice.py", line 1258, in _put_blob

return self._perform_request(request, _parse_base_properties)

File "/usr/local/lib/python3.6/site-packages/azure/storage/common/storageclient.py", line 446, in _perform_request

raise ex

File "/usr/local/lib/python3.6/site-packages/azure/storage/common/storageclient.py", line 374, in _perform_request

raise ex

File "/usr/local/lib/python3.6/site-packages/azure/storage/common/storageclient.py", line 360, in _perform_request

HTTPError(response.status, response.message, response.headers, response.body))

File "/usr/local/lib/python3.6/site-packages/azure/storage/common/_error.py", line 115, in _http_error_handler

raise ex

azure.common.AzureHttpError: Server encountered an internal error. Please try again after some time. ErrorCode: InternalError

<?xml version="1.0" encoding="utf-8"?><Error><Code>InternalError</Code><Message>Server encountered an internal error. Please try again after some time.

RequestId:78dfb400-40ae-4d7f-92af-dca3c9a95da1

Time:2019-08-29T08:31:09.9013502Z</Message><ExceptionDetails><ExceptionMessage>No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed</ExceptionMessage><StackTrace>Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStoreException: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.Blob.BlobClientException: Exception of type 'Microsoft.AzureStack.Services.Storage.Blob.BlobClientException' was thrown.

at Microsoft.AzureStack.Services.Storage.Blob.WBlobClient.<>c__DisplayClass8_0.<PutBlobAsync>b__0()

at System.Threading.Tasks.Task`1.InnerInvoke()

at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

--- End of stack trace from previous location where exception was thrown ---

at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)

--- End of stack trace from previous location where exception was thrown ---

at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobSingleChunk()

at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobAsync()

at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition)

--- End of inner exception stack trace ---

at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition)

at Microsoft.AzureStack.Services.Storage.Util.Await(Task t)

at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult)

--- End of inner exception stack trace ---

at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.ExceptionManager.ReThrowIfWStoreException(Exception e)

at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult)

at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.PutBlobImpl(IAccountIdentifier identifier, String account, String container, String blob, Stream inputStream, Int64 contentLength, IPutBlobProperties putBlobProperties, Boolean supportCrc64, Boolean calculateCrc64, Boolean storeCrc64, Boolean verifyCrc64, Boolean calculateMd5, Boolean storeMd5, Boolean verifyMd5, Boolean generationIdEnabled, Boolean isLargeBlockBlobRequest, Boolean is8TBPageBlobAllowed, BlobObjectCondition condition, OverwriteOption overwriteOption, TimeSpan timeout, RequestContext requestContext, AccessTier accessTier, String blobSasUrl, AsyncIteratorContext`1 context)+MoveNext()

at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)

--- End of inner exception stack trace ---

at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.EndPutBlob(IAsyncResult asyncResult)

at Microsoft.Cis.Services.Nephos.Storage.Service.Protocols.Rest.HttpRestProcessor.PutBlobImpl(AsyncIteratorContext`1 async)+MoveNext()

at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)

--- End of inner exception stack trace ---

at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.EndPerformOperation(IAsyncResult ar)

at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.ProcessImpl(AsyncIteratorContext`1 async)+MoveNext()</StackTrace></ExceptionDetails></Error>

Client-Request-ID=5a57e88c-ca37-11e9-affe-b24446ff1e0f Retry policy did not allow for a retry: Server-Timestamp=Thu, 29 Aug 2019 08:32:05 GMT, Server-Request-ID=e4df871b-770a-4f3a-b394-c193351171b6, HTTP status code=500, Exception=Server encountered an internal error. Please try again after some time. ErrorCode: InternalError<?xml version="1.0" encoding="utf-8"?><Error><Code>InternalError</Code><Message>Server encountered an internal error. Please try again after some time.RequestId:e4df871b-770a-4f3a-b394-c193351171b6Time:2019-08-29T08:32:05.9130943Z</Message><ExceptionDetails><ExceptionMessage>No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed</ExceptionMessage><StackTrace>Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStoreException: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.Blob.BlobClientException: Exception of type 'Microsoft.AzureStack.Services.Storage.Blob.BlobClientException' was thrown. at Microsoft.AzureStack.Services.Storage.Blob.WBlobClient.<>c__DisplayClass8_0.<PutBlobAsync>b__0() at System.Threading.Tasks.Task`1.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)--- End of stack trace from previous location where exception was thrown --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)--- End of stack trace from previous location where exception was thrown --- at Microsoft
.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobSingleChunk() at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobAsync() at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition) --- End of inner exception stack trace --- at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition) at Microsoft.AzureStack.Services.Storage.Util.Await(Task t) at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult) --- End of inner exception stack trace --- at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.ExceptionManager.ReThrowIfWStoreException(Exception e) at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult) at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.PutBlobImpl(IAccountIdentifier identifier, String account, String container, String blob, Stream inputStream, Int64 contentLength, IPutBlobProperties putBlobProperties, Boolean supportCrc64, Boolean calculateCrc64, Boolean storeCrc64, Boolean verifyCrc64, Boolean calculateMd5,
Boolean storeMd5, Boolean verifyMd5, Boolean generationIdEnabled, Boolean isLargeBlockBlobRequest, Boolean is8TBPageBlobAllowed, BlobObjectCondition condition, OverwriteOption overwriteOption, TimeSpan timeout, RequestContext requestContext, AccessTier accessTier, String blobSasUrl, AsyncIteratorContext`1 context)+MoveNext() at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin) --- End of inner exception stack trace --- at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.EndPutBlob(IAsyncResult asyncResult) at Microsoft.Cis.Services.Nephos.Storage.Service.Protocols.Rest.HttpRestProcessor.PutBlobImpl(AsyncIteratorContext`1 async)+MoveNext() at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin) --- End of inner exception stack trace --- at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.EndPerformOperation(IAsyncResult ar) at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.ProcessImpl(AsyncIteratorContext`1 async)+MoveNext()</StackTrace></ExceptionDetails></Error>.

Error in container log:

[2019-08-29 08:34:47.076] [info    ] [tid 488] Info: Exception is raised after sending the response. Exception: {0}, Status code: {1}, IsFatal: {2} IsResponseClosed:{3} IsErrorSerialized:{4}, p0="Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStoreException: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.Blob.BlobClientException: Exception of type 'Microsoft.AzureStack.Services.Storage.Blob.BlobClientException' was thrown. 
   at Microsoft.AzureStack.Services.Storage.Blob.WBlobClient.<>c__DisplayClass8_0.<PutBlobAsync>b__0() 
   at System.Threading.Tasks.Task`1.InnerInvoke() 
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) 
--- End of stack trace from previous location where exception was thrown --- 
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) 
--- End of stack trace from previous location where exception was thrown --- 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobSingleChunk() 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobAsync() 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition) 
   --- End of inner exception stack trace --- 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition) 
   at Microsoft.AzureStack.Services.Storage.Util.Await(Task t) 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult) 
   --- End of inner exception stack trace --- 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.ExceptionManager.ReThrowIfWStoreException(Exception e) 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult) 
   at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.PutBlobImpl(IAccountIdentifier identifier, String account, String container, String blob, Stream inputStream, Int64 contentLength, IPutBlobProperties putBlobProperties, Boolean supportCrc64, Boolean calculateCrc64, Boolean storeCrc64, Boolean verifyCrc64, Boolean calculateMd5, Boolean storeMd5, Boolean verifyMd5, Boolean generationIdEnabled, Boolean isLargeBlockBlobRequest, Boolean is8TBPageBlobAllowed, BlobObjectCondition condition, OverwriteOption overwriteOption, TimeSpan timeout, RequestContext requestContext, AccessTier accessTier, String blobSasUrl, AsyncIteratorContext`1 context)+MoveNext() 
   at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin) 
   --- End of inner exception stack trace --- 
   at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.EndPutBlob(IAsyncResult asyncResult) 
   at Microsoft.Cis.Services.Nephos.Storage.Service.Protocols.Rest.HttpRestProcessor.PutBlobImpl(AsyncIteratorContext`1 async)+MoveNext() 
   at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin) 
   --- End of inner exception stack trace --- 
   at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.EndPerformOperation(IAsyncResult ar) 
   at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.ProcessImpl(AsyncIteratorContext`1 async)+MoveNext()", p1="InternalError", p2="False", p3="True", p4="False" 
[2019-08-29 08:34:48.736] [info    ] [tid 28] Info: Accepting {0} request for '{1}'. RemoteEndPoint='{2}', Via='{3}', Expect='{4}', Connection='{5}', Version='{6}', ClientRequestId='{7}', UserAgent='{8}' DurationStayedInQueueInMs='{9}', RemainingRequestsInQueueAtEnqueue='{10}', RemainingRequestsInQueueAtDequeue='{11}', WasInHighPriorityQueue='{12}', ToSetKeepAliveToFalse='{13}', ActiveWorkerThreads='{14}', ActiveCompletionThreads='{15}' PassedInActivityId={16}, EnqueueTime={17}, DequeueTime={18}, RequestTraceActvityId = '{19}', p0="PUT", p1="http://blob-storage:11002/uniesb/2019-08-29/5d360072-0435-44bf-995d-ffe5d1f476d2", p2="10.42.0.21:49460", p3="", p4="", p5="keep-alive", p6="2019-02-02", p7="c7e6b888-ca37-11e9-bae1-aea9aca50ec3", p8="Azure-Storage/2.1.0-2.1.0 (Pytho", p9="0.0308", p10="0", p11="0", p12="False", p13="False", p14="0", p15="0", p16="null", p17="2019-08-29T08:34:48.7364616Z", p18="2019-08-29T08:34:48.7364850Z", p19="ee1ba80e-58ae-456a-9182-0d3587e71ed1" 
[2019-08-29 08:34:48.737] [info    ] [tid 506] WossProvider : Setting PutBlob timeout to '{0}'(initial timeout was '{1}')., p0="119.9989355", p1="119.9989355" 
[2019-08-29 08:34:48.738] [info    ] [tid 506] WossProvider : PutBlobAsync >> Account = {0}, Container = {1}, Blob = {2}, Timeout = {3}, ContentLength = {4}, OverwriteOption = {5}, Condition = {6}., p0="uniesb", p1="2019-08-29", p2="5d360072-0435-44bf-995d-ffe5d1f476d2", p3="00:01:59.9989355", p4="410", p5="CreateNewOrUpdateExisting", p6="[includeDisabledContainers = False, includeExpiredBlobs = False, ifModifiedSinceTime = <null>, ifNotModifiedSinceTime = <null>, leaseId = <null>, isIncludingUncommittedBlobs = True, isFetchingMetadata = False, isIncludingSnapshots = False, isIncludingPageBlobs = False, isIncludingAppendBlobs = False, isDeletingOnlySnapshots = False, ifLastModificationTimeMatch = <null>, ifLastModificationTimeMismatch = <null>, isRequiringNoSnapshots = False, isSkippingLastModificationTimeUpdate = False, sequenceNumberOperator = <null>, sequenceNumber = <null>, requiredBlobType = None, isMultipleConditionalHeaderEnabled = False, IsOperationOnAppendBlobsAllowed = True, IsRequiringLeaseIfNotModifiedSince = <null>, isDiskMountStateConditionRequired = False, ExcludeSoftDeletedBlobs = True, ExcludeAutomaticSnapshots = True, IncludeBlobVersionId = False, EnsureBlobIsAlreadySoftdeleted = False, isOperationAllowedOnArchivedBlobs = False, skipLeaseCheck = False, internalArchiveBlobLMT = <null>, internalArchiveBlobGenerationId = <null>, internalArchvieRequestId = <null>, includeSoftDeletedBlobsOnly = False, enforceWormPolicy = True, updateSequenceNumber = False, blobDeletePermissionLevel = RootBlobAndUserSnapshotOnly, ActiveNamespaceTransactionId = , excludeDFSCapacity = False, RequireFileEntity = False]" 
[2019-08-29 08:34:48.738] [info    ] [tid 506] [ContainerInterface.cc:64] [GetBlobContainerProperties] Get container properties. Name:2019-08-29 
[2019-08-29 08:34:48.738] [info    ] [tid 506] [BlobInterface.cc:73] [PutBlob] PutBlob received. Container:2019-08-29 Blob:5d360072-0435-44bf-995d-ffe5d1f476d2 BlobType:block_blob ContentLength:410 Overwrite:create_new_or_update_existing SvcMetadataSize:36 AppMetadataSize:0 DataSize:410 
[2019-08-29 08:34:48.738] [error   ] [tid 506] [MetaStore.cc:1338] [GetBlobRecord] Blob not found. ContainerName:2019-08-29 BlobName:5d360072-0435-44bf-995d-ffe5d1f476d2 
[2019-08-29 08:34:48.738] [error   ] [tid 506] [MetaStore.cc:1338] [GetBlobRecord] Blob not found. ContainerName:2019-08-29 BlobName:5d360072-0435-44bf-995d-ffe5d1f476d2 
[2019-08-29 08:34:48.738] [info    ] [tid 506] [DataStoreCommon.cc:163] [GetBlobFilePath] Insert BlobFilePath record. key:2019-08-29.5d360072-0435-44bf-995d-ffe5d1f476d2 value:2019-08-29/5d360072-0435-44bf-995d-ffe5d1f476d2 
[2019-08-29 08:34:48.739] [error   ] [tid 506] [DataStore.cc:378] [GetFileOffset] fallocate failed with error: Not supported 
[2019-08-29 08:34:48.739] [info    ] [tid 506] [DataStore.cc:395] [GetFileOffset] AllocateRange completed. next ChunkId:0 
[2019-08-29 08:34:48.739] [error   ] [tid 506] [BlobInterface.cc:204] [PutBlob] PutBlob failed. Container:2019-08-29 Blob:5d360072-0435-44bf-995d-ffe5d1f476d2 Status:1359 
[2019-08-29 08:34:48.740] [error   ] [tid 506] WossProvider : PutBlobAsync failed with error {0}, p0="1359" 
[2019-08-29 08:34:48.742] [error   ] [tid 506] WossProvider : Exception in EndPutBlob {0}, p0="Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStoreException: PutBlobAsync failed ---> Microsoft.AzureStack.Services.Storage.Blob.BlobClientException: Exception of type 'Microsoft.AzureStack.Services.Storage.Blob.BlobClientException' was thrown. 
   at Microsoft.AzureStack.Services.Storage.Blob.WBlobClient.<>c__DisplayClass8_0.<PutBlobAsync>b__0() 
   at System.Threading.Tasks.Task`1.InnerInvoke() 
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) 
--- End of stack trace from previous location where exception was thrown --- 
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) 
--- End of stack trace from previous location where exception was thrown --- 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobSingleChunk() 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobAsync() 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition) 
   --- End of inner exception stack trace --- 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationMetadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumberUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition) 
   at Microsoft.AzureStack.Services.Storage.Util.Await(Task t) 
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult)"

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

AshokPeddakotla-MSFT commented 5 years ago

@menmikimen Thanks for the feedback! We are currently investigating and will update you shortly.

mbialecka commented 5 years ago

@menmikimen The culprit here is "fallocate failed with error: Not supported". Please share your OS version, create options used, and type of storage/file system you are using.

Alberto-Vega commented 5 years ago

Hi @menmikimen would you please share the info requested on the previous comment by @mbialecka ?

AshokPeddakotla-MSFT commented 5 years ago

@menmikimen Did you get a chance to see the above response? Please share the requested details.

AshokPeddakotla-MSFT commented 5 years ago

@menmikimen Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly reopen the issue and continue the discussion. You can also open a new forum thread in MSDN forum or Stack Overflow as it could be better place to get help on your scenario. These forum community members could provide their expert guidance on your scenario based on their experience.

manuelsujith commented 1 year ago

@AshokPeddakotla-MSFT Is it possible to re-open this case? We are facing the exact same issue in our setup. I am attaching the log file once again for your checks. this container in a k8s cluster with a volume mount with a persistent volume which is based on SMB share.

The error is: [2023-01-09 15:52:47.946] [error ] [tid 1838] [DataStore.cc:389] [GetFileOffset] fallocate failed with error: Not supported

The host system is Windows Server 2022 Datacenter Azure Edition the container is running on a Linux worker node which is based on ubuntu and its mounted with a persistent volume as mentioned above

logsblob5.txt