VS 2022 Local Debug Durable Functions Issues "Error in LeaseManagerStarter task. Exception: Microsoft.WindowsAzure.Storage.StorageException: The response ended prematurely, with at least 128 additional bytes expected."

ahawes-clarity commented 2 years ago

Which service(blob, file, queue, table) does this issue concern?

N/A

Which version of the Azurite was used?

Latest Visual Studio 2022 Enterprise

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

Visual Studio Code Extension

What's the Node.js version?

N/A

What problem was encountered?

I have an Azure Functions v3 application that I have been developing and running for several years. In VS 2019 I could debug locally without issue. When running the application in VS 2022 and debugging I ran into several issues.

There was one issue related to AzureWebJobsSecretStorageType that I solved by adding "AzureWebJobsSecretStorageType": "files" to my local.settings.json based on this post. https://stackoverflow.com/questions/67728499/azure-functions-there-was-an-error-performing-a-read-operation-on-the-blob-stor

But after that I am still seeing errors that seem related to Durable Functions and I am not finding any answers online so far.

One issue is a timer function listener that triggers a Durable Functions clean up is unable to start due to Azure.Storage.Blobs: Service request failed. But then right after it seems to be able to start just fine. Why would it not start and then start?

Second is an "Error in LeaseManagerStarter task. Exception: Microsoft.WindowsAzure.Storage.StorageException: The response ended prematurely, with at least 128 additional bytes expected." which then leads to a "Error in LeaseManagerStarter task. Exception: System.InvalidOperationException: AppLeaseManager has already started"

I had previously posted about this in the Azure Functions Durable Function discussion which has more details: https://github.com/Azure/azure-functions-durable-extension/discussions/2025

@davidmrdavid suggested that I bring the discussion here.

The problem started when I moved the solution to VS 2022. The project works fine in VS 2019. I understand that is because VS 2019 uses the Azure Storage Emulator and VS 2022 uses Azurite. So, either VS 2022 should just work or there should be some migration steps documented someplace on how to convert a project from Azure Storage Emulator to Azurite, or there is something else going wrong.

Steps to reproduce the issue?

Debug my project in VS 2022.

Have you found a mitigation/solution?

It was suggested that if I run Azure Storage Emulator before launching VS 2022 that I can debug the project without issue but that it will be using Azure Storage Emulator rather than Azurite. Since Azure Storage Emulator is deprecated I would like to get things working with Azurite.

Thanks

edwin-huber commented 2 years ago

Hi Andrew, Thanks for raising this. Would you be able to start Azurite manually and share a debug log? You can find the details on how to activate it here: https://github.com/Azure/Azurite#debug-log-configuration That will show us which requests are failing and why. Alternatively, if you can share a sample which reproduces the issue, I can run it locally and debug the issue. Many thanks!

ahawes-clarity commented 2 years ago

Hi Edwin. I can even seem to run Azurite from the command line. I get this error:

C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator>azurite -d Azurite_ACv3.log Azurite Blob service is starting at http://127.0.0.1:10000 Exit due to unhandled error: Error: EPERM: operation not permitted, open 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator__azurite_db_blob__.json~'

edwin-huber commented 2 years ago

If visual studio and another instance of Azurite are running, then it can't get a second handle to open the files used to persist the data. Please first reboot, then start Azurite from Command Line, then start your VS Code / Visual Studio project.

ahawes-clarity commented 2 years ago

I rebooted and started up Teams, Outlook, and Chrome and the opened a new Command Prompt and changed to the "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator" folder and then ran "azurite -d C:\Users\Andrew.Hawes\DownloadsAzurite_ACv3.log" and got the same error.

Azurite Blob service is starting at http://127.0.0.1:10000 Exit due to unhandled error: Error: EPERM: operation not permitted, open 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator__azurite_db_blob__.json~'

From: Edwin Huber @.> Sent: Wednesday, April 27, 2022 2:05 PM To: Azure/Azurite @.> Cc: Andrew Hawes @.>; Author @.> Subject: Re: [Azure/Azurite] VS 2022 Local Debug Durable Functions Issues "Error in LeaseManagerStarter task. Exception: Microsoft.WindowsAzure.Storage.StorageException: The response ended prematurely, with at least 128 additional bytes expected." (Issue #1447)

If visual studio and another instance of Azurite are running, then it can't get a second handle to open the files used to persist the data. Please first reboot, then start Azurite from Command Line, then start your VS Code / Visual Studio project.

- Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2FAzurite%2Fissues%2F1447%23issuecomment-1111375106&data=05%7C01%7CAndrew.Hawes%40ClarityMIS.com%7C48239debc40f4129048408da2880cb22%7C1f87f22d99f94bc9b392bb0e2bb46d4c%7C0%7C0%7C637866830874140464%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=topokxasJEab2Wpf7cUuTXhOvteKshlnuARNU9CVUPA%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABN4FEFG5M35OEOVLB5Q4B3VHGFU3ANCNFSM5UMBBMPQ&data=05%7C01%7CAndrew.Hawes%40ClarityMIS.com%7C48239debc40f4129048408da2880cb22%7C1f87f22d99f94bc9b392bb0e2bb46d4c%7C0%7C0%7C637866830874140464%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6NZQyUz1RSgVe%2BAw1U8gl06OugeyhAhB%2FzBCLw1kLuQ%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.**@.>>

ahawes-clarity commented 2 years ago

@edwin-huber, any feedback on my comments.

Sti2nd commented 2 years ago

I also have this problem, cannot run Azure Functions locally with Visual Studio 2022. Seems like VS is using Azurite 3.14.1. Running Azurite 3.16.0 (and newer) outside Visual Studio first fixes the problem.

edwin-huber commented 2 years ago

@ahawes-clarity this looks like a permissions issue, the database engine cannot open the DB location. You could try running in an administrative window, but this is an unusual problem. We are looking at long term options for this type of issue. @Sti2nd Regarding the functions project, there is a release pending to resolve this, you need to upgrade Visual Studio to upgrade Azurite.

ahawes-clarity commented 2 years ago

Thanks @edwin-huber that seemed to work. Running Azurite in a administrative command window before launching VS removed the errors I was seeing. I have a log file I can send you but am concerned about posting it here. How can I get it to you if you feel it might be helpful?

Does this give you more insight into what the problem is and a possible resolution. I really don't want to have to follow these steps to get things working each time.

Sti2nd commented 2 years ago

@edwin-huber I am on Visual Studio 2022 version 17.1.6. For some reason my VS can't check for updates right now.

@ahawes-clarity Since running another version of Azurite outside of Visual Studio fixes the problem for me, did you try running Azurite without administrative privileges before opening Visual Studio?

ahawes-clarity commented 2 years ago

@Sti2nd , yes I did try running Azurite before VS but it only worked when running Azurite as an administrator.

edwin-huber commented 2 years ago

@ahawes-clarity I don't think a debug log will show us anything we don't already know. The symptoms point to a permission issue on the file system. The error states that it is unable to open the file at the location 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator\ You could check with windows explorer what permissions are set there. I ran the following PowerShell commands in Windows Terminal / PowerShell:

cd C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft
get-childitem . | where-object {($_.name -eq "Azure Storage Emulator" -and $_.PSIsContainer)} | get-ACL | fl

My permissions are:

Path   : Microsoft.PowerShell.Core\FileSystem::C:\Program Files\Microsoft Visual
         Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator
Owner  : BUILTIN\Administrators
Group  : DESKTOP\<my local username>
Access : NT SERVICE\TrustedInstaller Allow  FullControl
         NT SERVICE\TrustedInstaller Allow  268435456
         NT AUTHORITY\SYSTEM Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  268435456
         BUILTIN\Administrators Allow  FullControl
         BUILTIN\Administrators Allow  268435456
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
         BUILTIN\Users Allow  -1610612736
         CREATOR OWNER Allow  268435456
         APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES Allow  ReadAndExecute, Synchronize
         APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES Allow  -1610612736
         APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES Allow  ReadAndExecute, Synchronize
         APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES Allow  -1610612736

ahawes-clarity commented 2 years ago

@edwin-huber mine look the same.

Path   : Microsoft.PowerShell.Core\FileSystem::C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator
Owner  : BUILTIN\Administrators
Group  :
Access : NT SERVICE\TrustedInstaller Allow  FullControl
         NT SERVICE\TrustedInstaller Allow  268435456
         NT AUTHORITY\SYSTEM Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  268435456
         BUILTIN\Administrators Allow  FullControl
         BUILTIN\Administrators Allow  268435456
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
         BUILTIN\Users Allow  -1610612736
         CREATOR OWNER Allow  268435456
         APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES Allow  ReadAndExecute, Synchronize
         APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES Allow  -1610612736
         APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES Allow  ReadAndExecute, Synchronize
         APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES Allow  -1610612736

edwin-huber commented 2 years ago

Hi @ahawes-clarity did you remove / redact the "Group" entry, or is it blank on export?

ahawes-clarity commented 2 years ago

@edwin-huber it was blank on export

edwin-huber commented 2 years ago

In that case, please use the explorer to do the following:

Browser to this folder: C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft
right click on the folder Azure Storage Emulator and choose "properties"
choose the "Security" tab
and click the "advanced" button
at the top, where it says owner, click the "Change" button
add yourself back as "owner" of the folder by searching for your local user name, or domain user name depending on how you log into Windows
click "check names"
click "OK"
Click "Apply "

ahawes-clarity commented 2 years ago

I can do that but first please tell me why I would need to do that. Currently the folder is owned by the local computer's Administrators Group and I am an administrator on the machine.

edwin-huber commented 2 years ago

The idea is that it will allow access for the process without needing to elevate the permissions to administrator.

ahawes-clarity commented 2 years ago

Sorry for being stubborn. I really do appreciate your help.

Is the change you are suggesting just to test something out or are you proposing it as a solution?

Is there a reason that it is not working as is? I would imagine that my setup is very similar to most peoples. Shouldn't it just work in this scenario?

And, btw, I can't select my current user to be owner because I can only set the ownership to a local user account it seems and my user is a network user. I could set the ownership to the Users group instead of Administrators if you really think we need to test this.

edwin-huber commented 2 years ago

No need to be sorry, I understand your caution, better safe than sorry! Yes, I think that using the Users group should also work. My logic is as follows: It works when started in the context of administrator. (raising the security context and rights of the process) The account that you use to log in is not the owner of the folder. Goal: Lower the restrictions on the folder to provide the necessary access for the process to access the database files.

I can't give you a 100% guarantee, as I do not know what is blocking the access to the folder, but visual studio (devenv.exe) is running under my local user account, which is why I think that I do not get this error.

Once we have a short term solution we can investigate it's root cause, and potentially addressing this via other means in visual studio installer etc.

ahawes-clarity commented 2 years ago

I set the Owner of the folder to the Users Group

I then opened up VS 2022 and ran my Azure Functions project.

I received the same errors as shown in the initial post of this thread.

I closed down VS 2022.

I opened a command window in normal mode (not administrator) and tried to start azurite and received the following error:

Azurite Blob service is starting at http://127.0.0.1:10000
Exit due to unhandled error: Error: EPERM: operation not permitted, open 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator\__azurite_db_blob__.json~'

I ran the same powershell command as before to verify the permissions.

Path   : Microsoft.PowerShell.Core\FileSystem::C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator
Owner  : BUILTIN\Users
Group  :
Access : NT SERVICE\TrustedInstaller Allow  FullControl
         NT SERVICE\TrustedInstaller Allow  268435456
         NT AUTHORITY\SYSTEM Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  268435456
         BUILTIN\Administrators Allow  FullControl
         BUILTIN\Administrators Allow  268435456
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
         BUILTIN\Users Allow  -1610612736
         CREATOR OWNER Allow  268435456
         APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES Allow  ReadAndExecute, Synchronize
         APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES Allow  -1610612736
         APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES Allow  ReadAndExecute, Synchronize
         APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES Allow  -1610612736
Audit  :
Sddl   : O:BUG:DUD:AI(A;ID;FA;;;S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464)(A;CIIOID;GA;;;S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464)(A;ID;FA;;;SY)(A;OICIIOID;GA;;;SY)(A;I
         D;FA;;;BA)(A;OICIIOID;GA;;;BA)(A;ID;0x1200a9;;;BU)(A;OICIIOID;GXGR;;;BU)(A;OICIIOID;GA;;;CO)(A;ID;0x1200a9;;;AC)(A;OICIIOID;GXGR;;;AC)(A;ID;0x1200a9;;;S-1-15-2-2)(A;OICIIOID;GXGR;;;S-1-15-2-2)

I then opened a new command window as administrator and started azurite successfully.

So, it seems that setting the folder ownership to the Users group did not solve the problem.

edwin-huber commented 2 years ago

OK, many thanks for taking the time to try it. I have just reproduced this on my own system. I am always working from the main branch, and so don't start it from the visual studio download location.

Looking at the folder, it is marked "read-only", which would explain why this only works when starting as administrator. However, I am unable to change this restriction.

I need to research and follow up with my colleagues who own the VS distribution mechanism, to see what the expected behavior and folders in this path should look like. Please bear with us.

ahawes-clarity commented 2 years ago

Great! Glad you found something. It is not a blocker for me since I have a work around but would love to see this resolved soon. Thanks

XiaoningLiu commented 2 years ago

@ahawes-clarity

"C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator" seems to be read only be design. I have same read only settings cross my different Visual Studio 2022 installations.

If you want to start 'azurite.exe', please make a directory like C:\Azurite to store data for Azurite. Refer to https://github.com/microsoftdocs/azure-docs/issues/83901

cd C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator
azurite --location c:\azurite --debug c:\azurite\debug.log

XiaoningLiu commented 2 years ago

@ahawes-clarity Can you share the steps to reproduce the issue in Visual Studio 2022 (instead of starting azurite from the extension path)? I'll forward the steps to Visual Studio team for trouble shoot.

ahawes-clarity commented 2 years ago

@XiaoningLiu the steps are quite simple.

Run my Azure Functions v3 project locally in debug mode from VS 2022.

I am not sure if there is something about my project that may be a factor. It is a project that has evolved over the years. Works fine in VS 2019 and Azure Storage Emulator. It does use Durable Functions.

    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="4.0.3" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="3.1.9" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.13" />

@edwin-huber says he was able to reproduce the issue so maybe he has more info.

edwin-huber commented 2 years ago

Hi @ahawes-clarity, thanks, yes, Xiaoning and I have spoken.

I was able to reproduce the error in my environment, by trying to start the executable azurite.exe from the command line.

@XiaoningLiu the steps to reproduce the error are here: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=visual-studio The VS documentation specifically states to start from the command line, which does not work without the --location parameter.

When I start one of my function apps, the location parameter is automatically included, which stores the data files in a different location, this would then also solve the issue for people starting the emulator from command line:

azurite.exe --location "C:\Users\mylocaluser\AppData\Local\Temp\Azurite" --debug "C:\Users\mylocaluser\AppData\Local\Temp\Azurite\debug.log"

We need to check why the location parameter is not being used in the migrated / upgraded function apps...

zpfaltersack commented 1 year ago

Hi, I know this issue has been around a bit. I just ran into it myself and wasn't happy with having to run azurite as admin so I kept digging around and found another thread that solved the issue more permanently for me. It's adjacent to the conversation on this thread. https://github.com/Azure/Azurite/issues/1309

My situation was a functions v3/.netcore 3.1/VS2019 upgrade to v4/.net6/VS2022. Everything went smoothly for all functions projects except the one that uses durable functions.

I don't think it's so much that the location flag isn't being migrated or upgraded. It seems to me that azurite must be securing the db files based on VS project/process. So VS2022 didn't have access to the data files that had been created when previously running in VS2019. Obviously I know nothing and that's just a hunch. The long and short of it was clearing out that temp directory completely reset VS2022 enough that it could start azurite by itself without admin permissions for the files.

edwin-huber commented 1 year ago

Many thanks for the tip, we shall look into it! @XiaoningLiu FYI!

zpfaltersack commented 1 year ago

I randomly reproduced this issue again by clearing out my NuGet cache. After getting everything rebuilt I was getting errors and had to delete those temp files again. Not sure it's relevant but figured I'd mention it just in case. I didn't approach it very scientifically because I was in the middle of something, but it did appear that this time it affected every azure functions project, not just the durable functions.

Sti2nd commented 1 year ago

Following @zpfaltersack' suggestion https://github.com/Azure/Azurite/issues/1447#issuecomment-1309222760 of renaming the temp folder seems to be working! 😍 As first mentioned in https://github.com/Azure/Azurite/issues/1309

Sti2nd commented 1 year ago

But it quickly stopped working for me 😠 I guess I could delete the folder again, but getting seriously tired of the hassle with Azure Functions now

scienide commented 1 year ago

OK, so I get the same issue and it seems that it's due to another Azurite process running on the same port but it seems isolated from any builds or something. Still, running the below steps seem to work:

Close Visual Studio
Try the following PS Code (sorry for the crappy code!): cd "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator" $azStop = .\azurite.exe stop ## Find processes running on port 10000 $portUse = Get-Process -Id (Get-NetTCPConnection -LocalPort 10000).OwningProcess; $portUse ## Kill processes' on this port foreach($port in $portUse) { if($portUse.Id -ne 0) { $x = taskkill /F /PID $portUse.Id $x } }
run .\azurite.exe start then reopen VS. It should now build & serve your app with no errors.

vladnega commented 1 year ago

I have encountered the same error today, and after trying the suggestions in this thread it still didn't work.

I then tried updating to the latest version of Visual Studio (Microsoft Visual Studio Community 2022 (64-bit) - Preview, Version 17.6.0 Preview 6.0), and then reinstalling the Azure development feature from the Visual Studio installer.

Everything seems to be working fine now.

oa-baski commented 1 year ago

All these issues can be fixed by following these steps ( Running in admin mode is not an option)

Clearing the NPM cache %APPDATA%\npm-cache, delete everything in this folder
npm install azure
Then start azurite it should work

For more details refer to his article https://learn.microsoft.com/en-us/archive/blogs/matt-harrington/how-to-fix-node-js-npm-permission-problems

Azure / Azurite