search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.83k stars 325 forks source link

There was an error performing a read operation on the Blob Storage Secret Repository. Please ensure the 'AzureWebJobsStorage' connection string is valid. #1179

Open calloncampbell opened 2 years ago

calloncampbell commented 2 years ago

Which service(blob, file, queue, table) does this issue concern?

My guess is all of them the error message, but I'm only using queue from my app.

Which version of the Azurite was used?

3.14.3

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

Came with VS2022. I then uninstall from NPM and reinstall from NPM to v3.14.3

What's the Node.js version?

v14.18.1.

What problem was encountered?

Trying to debug my azure function app in VS2022 and on startup I get the above error. I don't get this error on any other function app debug session.

[2021-11-18T20:06:09.561Z] There was an error performing a read operation on the Blob Storage Secret Repository. Please ensure the 'AzureWebJobsStorage' connection string is valid.
[2021-11-18T20:06:09.611Z] A host error has occurred during startup operation 'e8b3d28c-2c65-4fc6-9c7e-69cd82eda265'.
[2021-11-18T20:06:09.613Z] Azure.Storage.Blobs: Service request failed.
[2021-11-18T20:06:09.615Z] Status: 500 (Internal Server Error)
[2021-11-18T20:06:09.617Z]
[2021-11-18T20:06:09.618Z] Headers:
[2021-11-18T20:06:09.619Z] Server: Azurite-Blob/3.14.3
[2021-11-18T20:06:09.620Z] Date: Thu, 18 Nov 2021 20:06:09 GMT
[2021-11-18T20:06:09.620Z] Connection: keep-alive
[2021-11-18T20:06:09.621Z] Keep-Alive: REDACTED
[2021-11-18T20:06:09.621Z] Content-Length: 0

Steps to reproduce the issue?

Press F5

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "FunctionStorageBindings-StorageConnectionString": "REDACTED",
    "AzureAppConfiguration.ConnectionString_PrimaryStore": "REDACTED",
    "AzureAppConfiguration.ConnectionString_SecondaryStore": "REDACTED",
    "AzureAppConfiguration.CacheExpirationTimeInSeconds": 120,
    "AzureAppConfiguration.EnvironmentLabel": "LabA",
    "DurableTaskHubName": "BulkEditProcessorDEV2",
    "DurableManagementStorage": "UseDevelopmentStorage=true",
    "AzureWebJobs.ProcessBulkEditQueue.Disabled": 0
  }
}

Here are the versions of the packages being used:

  <ItemGroup>
    <PackageReference Include="Azure.Identity" Version="1.5.0" />
    <PackageReference Include="Azure.Storage.Blobs" Version="12.10.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.6.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid" Version="3.0.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="4.0.1" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Azure" Version="1.1.1" />
    <PackageReference Include="Microsoft.Extensions.Configuration.AzureAppConfiguration" Version="4.5.1" />
    <PackageReference Include="Microsoft.FeatureManagement" Version="2.4.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.13" />
  </ItemGroup>

If possible, please provide the debug log using the -d parameter, replacing \<pathtodebuglog> with an appropriate path for your OS, or review the instructions for docker containers:

-d "<pathtodebuglog>"

Please be sure to remove any PII or sensitive information before sharing!
The debug log will log raw request headers and bodies, so that we can replay these against Azurite using REST and create tests to validate resolution.

Have you found a mitigation/solution?

No

blueww commented 2 years ago

@calloncampbell

Would you please share the Azurite debug log, we need it for investigation?

calloncampbell commented 2 years ago

Hi @blueww I cleared my debug.log file in C:\Users\ccampbell\AppData\Local\Temp\Azurite and restarted VS2022 and tried to debug the function app, I got the same error. Here is the log file.

debug.log

I was able to get it to work magically last night on this project, but then encountered the same issue with another function app and then again with the reported function app just now. It doesn't seem to be an issue in VS2019.

calloncampbell commented 2 years ago

Just an update, if I delete or rename this folder C:\Users\ccampbell\AppData\Local\Temp\Azurite I'm able to get past this issue and the function app works locally again.

blueww commented 2 years ago

@calloncampbell Thanks for the reply! From the debug log, we see the error is "Error: ENOENT: no such file or directory, open 'C:\Users\ccampbell\AppData\Local\Temp\Azurite\blobstorage\93e1dd6c-5675-44be-a20f-7168e3618dcc'".

But the debug log only contains get the file fail, not contain how the file is created/removed. Would you give the full debug log from: delete the folder C:\Users\ccampbell\AppData\Local\Temp\Azurite, to the issue repro? So we can see why Azurite need the file, but the file doesn't exist.

calloncampbell commented 2 years ago

@blueww that is what I did. The log file attached is after I deleted the temp Azurite directory and then reproducing the issue. I'm still getting the issue once and while, but deleting the folder helps reset for sure until the issue happens again.

blueww commented 2 years ago

@calloncampbell Thanks for the confirmation! It's interesting we didn't see the file created in the log, but azurite try to get it. We will continue look at it.

calloncampbell commented 2 years ago

@blueww let me know if there is anything I can do to help.

blueww commented 2 years ago

@calloncampbell Would you please confirm do you use SQL server for Azurite？(Have you set the environment variable "AZURITE_DB"?) And could you share the command you used to start Azurite? (hide any credential.)

a99cl208 commented 2 years ago

@calloncampbell I ran into the same issue. It killed the azurite process that is started with visual studio, then restarted it using a command line with admin privilege. Then it worked, so it is maybe a security related issue.

OzBob commented 2 years ago

i have the same issue. If I create a new v3 AzFn solution and project with VS2022, and copy all the code from my 'broken' project, it works perfectly. I tried deleting the .vs folder from the 'broken' project but still same issue.

I do note that when the sln is opened there is a Services Dependencies message:

Ensuring Azure Functions Core Tools are up to date. This may take a few minutes...
AzFn2: Azurite emulator cannot be started because port 10000 is already in use. Another instance of the Azurite emulator or Azure Storage emulator might be already running on your machine.
Unable to start dependency 'storage1'.
Azure Functions Core Tools are up to date.
AzFn2: Azurite emulator cannot be started because port 10000 is already in use. Another instance of the Azurite emulator or Azure Storage emulator might be already running on your machine.
Unable to start dependency 'storage1'.

When debug VS the azfn breaks in ExtensionConfigContext

namespace Microsoft.Azure.WebJobs.Host.Config

    ...
 [Obsolete("preview")]
        public Uri GetWebhookHandler()
        ...
            return _webHookProvider.GetUrl(this.Current);
mkellyelumus commented 2 years ago

@calloncampbell I ran into the same issue. It killed the azurite process that is started with visual studio, then restarted it using a command line with admin privilege. Then it worked, so it is maybe a security related issue.

FWIW, this fix/work-around resolved the issue for me as well.

Note if I simply run "./azurite.exe" in the elevated console, all works fine. But if I run it with the same args VS uses when launching it (./azurite.exe --location "[etc.]\AppData\Local\Temp\Azurite" --debug "[etc.]\AppData\Local\Temp\Azurite\debug.log"), the [OP] issue resurfaces.

yhs666 commented 2 years ago

I found some process used network port 10000 maybe impacted azurite start.

image

Close SangForSP service issue fixed. image

blueww commented 2 years ago

@yhs666 The 10000 port is occupied is different from the original issue. And Azurite will use 10000 by default for blob endpoint. User need make sure 10000 port is not occupied to start Azurite successfully. Or you can also customize the blob listening port per your requirements in the extension settings.

calloncampbell commented 2 years ago

@blueww The issue I originally reported hasn't happened in a while but every now and then I do get the port in use issue that has also been reported above. For me, the port is usually in use by NodeJs. After terminating the process Azurite starts up correctly.

For context Visual Studio 2022 is the one starting up Azurite storage emulator when working with Azure Functions app.

calloncampbell commented 2 years ago

@calloncampbell Would you please confirm do you use SQL server for Azurite？(Have you set the environment variable "AZURITE_DB"?) And could you share the command you used to start Azurite? (hide any credential.)

No I'm not using SQL Server for Azurite.

Azurite is starting with Visual Studio 2022. Here is what shows up in the Output window for "Service Dependencies":

Ensuring Azure Functions Core Tools are up to date. This may take a few minutes...
Azure Functions Core Tools are up to date.
DotNetCore31-FunctionApp: azurite.cmd --location "C:\Users\ccampbell\AppData\Local\Temp\Azurite" --debug "C:\Users\ccampbell\AppData\Local\Temp\Azurite\debug.log"
DotNetCore31-FunctionApp: Azurite Blob service is starting at http://127.0.0.1:10000
DotNetCore31-FunctionApp: Azurite Blob service is successfully listening at http://127.0.0.1:10000
DotNetCore31-FunctionApp: Azurite Queue service is starting at http://127.0.0.1:10001
DotNetCore31-FunctionApp: Azurite Queue service is successfully listening at http://127.0.0.1:10001
DotNetCore31-FunctionApp: Azurite Table service is starting at http://127.0.0.1:10002
DotNetCore31-FunctionApp: Azurite Table service is successfully listening at http://127.0.0.1:10002
blueww commented 2 years ago

@calloncampbell

Thanks for the information! Good to know the original issue not repro now. For the port occupied issue, Azurite need to list on specific port, and the port need be not occupied. So it's expected that Azurite report error if the port is occupied. Please make sure the port Azurite use is not occupied before start Azurite.

calloncampbell commented 2 years ago

@blueww

Could we not change the default ports to something not being used by such a common framework like NodeJS.

blueww commented 2 years ago

@calloncampbell The default port has been used for a long time from the before version of storage emulator. We don't plan to change it, or might block many existing customer.

If you can input parameter when start Azurite, you can use the customized port to avoid the port been occupied issue. https://github.com/Azure/Azurite#listening-port-configuration

zakkra commented 2 years ago

How is durable function going to know about new port used for storage? Is there a way to specify this new port in the storage connection string i.e. "AzureWebJobsStorage": "UseDevelopmentStorage=true"

blueww commented 2 years ago

@zakkra

For connection string, you can also use a connection string like following, and update the endpoint port to the real port you are using, see more details in https://github.com/Azure/Azurite#connection-strings: "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;"

Asier-Villanueva commented 1 year ago

I have the same issue. As a workaround I have added in my local.settings.json a new setting "AzureWebJobsSecretStorageType" setting the value to "files":

"AzureWebJobsSecretStorageType": "files"