search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.75k stars 311 forks source link

Event Hub blob partition lease conflict #263

Open gabrieljoelc opened 4 years ago

gabrieljoelc commented 4 years ago

This webjob works fine running on Windows with the Windows Storage Emulator.

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

v2

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub

What's the Node.js version?

6.11.2

What problem was encountered?

Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'RouterHost.Route' was unable to start. ---> Microsoft.Azure.EventHubs.Processor.EventProcessorRuntimeException: Out of retries creating lease for partition ---> Microsoft.Window
sAzure.Storage.StorageException: Conflict
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromStreamAsyncHelper(Stream source, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgress`1 progressHandler, CancellationToken cancella
tionToken)
   at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager.CreateLeaseIfNotExistsAsync(String partitionId)
   at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager.CreateLeaseIfNotExistsAsync(String partitionId)
   at Microsoft.Azure.EventHubs.Processor.PartitionManager.RetryAsync(Func`1 lambda, String partitionId, String retryMessage, String finalFailureMessage, String action, Int32 maxRetries)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.EventHubs.Processor.PartitionManager.RetryAsync(Func`1 lambda, String partitionId, String retryMessage, String finalFailureMessage, String action, Int32 maxRetries)
   at Microsoft.Azure.EventHubs.Processor.PartitionManager.InitializeStoresAsync()
   at Microsoft.Azure.EventHubs.Processor.PartitionManager.StartAsync()
   at Microsoft.Azure.EventHubs.Processor.EventProcessorHost.RegisterEventProcessorFactoryAsync(IEventProcessorFactory factory, EventProcessorOptions processorOptions)
   at Medumo.WebJobs.Extensions.EventHub.Binding.EventHubListener.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(CancellationToken cancellationToken, Boolean allowRetry) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 69

Steps to reproduce the issue?

Have you found a mitigation/solution?

No

gabrieljoelc commented 4 years ago

~I wonder if it's because my application is written to use the same storage URL for blob and table.~

I just switched the connection strings to use UseDevelopmentStorage=true;, but I get the same failure. 😞

XiaoningLiu commented 4 years ago

Hi @gabrieljoelc Azurite V2 blob going deprecated. Can you try v3 and share debug.log of Azurite when issue happens?

gabrieljoelc commented 4 years ago

@XiaoningLiu unfortunately, there is a limitation in one of the libraries i need to use that that uses the same storage account connection string for accessing blob and table storage. The lack of table storage support in V3 means I can't use V3. :(

kjohann commented 4 years ago

@XiaoningLiu I am having a very similar issue on v 3.2.0-preview (both using the docker image and running via node on Windows). I have an Azure Function with an EventHub-trigger (Azure Function SDK 1.0.29). I'm using version 3.0.6 of Microsoft.Azure.WebJobs.Extensions.EventHub (I see there's an v4 out of this, but I cannot get my project to run with it). My app starts just fine, and it can process messages. However, when it tries to write checkpoints it fails:

ConsoleError

Attaching an outtake of debug.log from around the time of execution: outtake.log

Never mind the differences in timestamps from the screenshot btw. Separate runs, same errors!

It may be something related to an older API-version being used. My project also uses Microsoft.Azure.WebJobs.Extensions.Storage (version 3.0.8). This package has a dependency on WindowsAzure.Storage which is now deprecated.

edwin-huber commented 4 years ago

@gabrieljoelc You are using the old project's Docker image, which is not coming from the Microsoft Repo, and is very outdated.
Assuming that we can solve the issue with the lease conflict message, please do consider using a different repo for the Docker image.
We are currently prevented from adding the legacy branch to the Microsoft Container Repo, so I would recommend that you create your own image for that.

@XiaoningLiu : We must update the documentation for the table API to indicate that those wishing to use Docker images, should build their own, and not use the arafato registry.
The alternative would be that I host under my Docker alias, which is not ideal, as we could hit the same issue we had with the arafato registry entry.
I'll open a separate issue for the legacy Docker Usage.

The actual root cause here, just looks like it might be how we are handling leases on the containers / blobs, which is behaving differently to our service.

ahacking commented 4 years ago

Similar problem here running azurite@3.2.0-preview when trying to test Azure Event Hubs locally.

I started azurite in debug mode ans then ran my function using func start. My event hub uses multiple partitions which all report the same logs and errors as below:

127.0.0.1 - - [04/Nov/2019:06:10:15 +0000] "HEAD /devstoreaccount1/azure-webjobs-eventhub/xxx.servicebus.windows.net/test/$Default/0 HTTP/1.1" 200 156
127.0.0.1 - - [04/Nov/2019:06:10:15 +0000] "HEAD /devstoreaccount1/azure-webjobs-eventhub/xxx.servicebus.windows.net/test/$Default/0 HTTP/1.1" 200 156
127.0.0.1 - - [04/Nov/2019:06:10:15 +0000] "PUT /devstoreaccount1/azure-webjobs-eventhub/xxx.servicebus.windows.net/test/$Default/0?comp=lease HTTP/1.1" 201 -
127.0.0.1 - - [04/Nov/2019:06:10:15 +0000] "PUT /devstoreaccount1/azure-webjobs-eventhub/xxx.servicebus.windows.net/test/$Default/0 HTTP/1.1" 201 -

127.0.0.1 - - [04/Nov/2019:06:10:25 +0000] "PUT /devstoreaccount1/azure-webjobs-eventhub/xxx.servicebus.windows.net/test/$Default/0?comp=lease&timeout=10 HTTP/1.1" 409 -

[11/4/19 6:10:25 AM] EventProcessorHost error (Action=Renewing Lease, HostName=xxxxx, PartitionId=0)
[11/4/19 6:10:25 AM] Microsoft.Azure.EventHubs.Processor: Conflict. Microsoft.WindowsAzure.Storage: Conflict.

Hope this helps narrow down the problem a bit.

gabrieljoelc commented 4 years ago

@gabrieljoelc You are using the old project's Docker image, which is not coming from the Microsoft Repo, and is very outdated.

@edwin-huber yes I am using arafato/azurite. here's the command:

docker run -d -t -p 10000:10000 -p 10001:10001 -p 10002:10002 -v /my/localy/volume/path:/opt/azurite/folder arafato/azurite

Assuming that we can solve the issue with the lease conflict message, please do consider using a different repo for the Docker image.

which repo? https://hub.docker.com/_/microsoft-azure-storage-azurite doesn't support blob and table storage and i need both.

i suppose i could run new image for just blob and then the old one for table. i feel like i tried that but can't remember. i'll try that again.

The actual root cause here, just looks like it might be how we are handling leases on the containers / blobs, which is behaving differently to our service.

that's my guess as well.

XiaoningLiu commented 4 years ago

@ahacking Can you try latest released 3.3.0-preview? We did some refactor for lease implementation for blobs and containers. If the issue still happens for 3.3.0-preview, please share your environments, example code, and steps for the reproduce. @blueww is helping with Azure Function comptiability invesitgation with Azurite, and we'd like more information.

gabrieljoelc commented 4 years ago

I tried this again to no avail. Please, see context below.

Application description

My application is a c# console app that runs as a webjob in Azure and relies on table storage. It has a custom Event Hub trigger that relies on Blob and Table storage through 2 app settings:

What I tried

Start blob storage with Azurite V3:

# Terminal A
docker run --rm -p 10000:10000 -p 10001:10001 -v ~/azurite:/data mcr.microsoft.com/azure-storage/azurite

Start table storage with Azurite V2 because V3 doesn't support table storage:

# Terminal B
docker run --rm -e executable=table -t -p 10002:10002 -v ~/azurite:/opt/azurite/folder arafato/azurite

Set application settings:

{
  "AzureWebJobsStorage": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;",
  "StorageConnectionString": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;",
}

Error I receive in application when it tries to start consuming from event hub:

Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'MyCustomHost.ProcessEvent' was unable to start. ---> Microsoft.Azure.EventHubs.Processor.EventProcessorConfigurationException: Encountered error while fetching the list of EventHub PartitionIds ---> System.Net.Sockets.SocketException: Operation timed out

Azurite V3 output:

➜  ~ docker run --rm -p 10000:10000 -p 10001:10001 -v ~/azurite:/data mcr.microsoft.com/azure-storage/azurite
Azurite Blob service is starting on 0.0.0.0:10000
Azurite Blob service successfully listens on 0.0.0.0:10000
172.17.0.1 - - [06/Feb/2020:00:04:35 +0000] "HEAD /devstoreaccount1/event-metadata?restype=container HTTP/1.1" 404 -
172.17.0.1 - - [06/Feb/2020:00:04:35 +0000] "HEAD /devstoreaccount1/event-metadata?restype=container HTTP/1.1" 404 -
172.17.0.1 - - [06/Feb/2020:00:04:35 +0000] "PUT /devstoreaccount1/event-metadata?restype=container HTTP/1.1" 201 -
172.17.0.1 - - [06/Feb/2020:00:04:35 +0000] "PUT /devstoreaccount1/event-metadata?restype=container HTTP/1.1" 409 -
172.17.0.1 - - [06/Feb/2020:00:43:19 +0000] "HEAD /devstoreaccount1/event-metadata?restype=container HTTP/1.1" 200 -
172.17.0.1 - - [06/Feb/2020:00:43:19 +0000] "HEAD /devstoreaccount1/event-metadata?restype=container HTTP/1.1" 200 -

Azurite V2 output:

➜  ~ docker run --rm -e executable=table -t -p 10002:10002 -v ~/azurite:/opt/azurite/folder arafato/azurite
Azure Table Storage Emulator listening on port 10002

It's weird there was no output for the Table storage service (arafato/azurite). I haven't tried to use the npm version of the service. I suppose I can try that.

edwin-huber commented 4 years ago

Hi, the arafato repository is not maintained, and is a legacy artifact.
I've just updated the docs to remove this properly and hopefully avoid confusion.

We are not investing in maintaining the v2 version of the emulator, I understand that a lot of people are still relying on this to provide table storage support until we have a new cross platform emulator for table storage, and this requires building and running the v2 emulator yourself.
The v2 emulator should only be used to provide table storage, as the other services are maintained in the v3 emulator.

I am able to set up an Event Hub Function using the local storage without issues when using the locally running v3 Emulator: 

127.0.0.1 - - [06/Feb/2020:17:02:11 +0000] "PUT /devstoreaccount1/azure-webjobs-hosts/locks/erazer-402578966/host?comp=lease HTTP/1.1" 200 -

I'm not using tablestorage, So my recommendation is to check your bindings and ensure that you connect to 2 different emulators, 1 for table, 1 for webjob storage.

HTH.

gabrieljoelc commented 4 years ago

@edwin-huber thanks for the response.

this requires building and running the v2 emulator yourself. The v2 emulator should only be used to provide table storage, as the other services are maintained in the v3 emulator.

I understand that V2 is deprecated and that there's still no support for Table storage in V3. As I explained in https://github.com/Azure/Azurite/issues/263#issuecomment-582687766, I already am pulling the V2 image for Table and the V3 image for Blob.

my recommendation is to check your bindings and ensure that you connect to 2 different emulators, 1 for table, 1 for webjob storage

As I showed in https://github.com/Azure/Azurite/issues/263#issuecomment-582687766, I attempt to connect to 2 different emulators. I was hoping someone here could sanity check what I set up and spot any reason why it's not working.

edwin-huber commented 4 years ago

Maybe I am misunderstanding you.
When you say "pulling" the v2 image... what exactly are you doing?
There is no v2 Docker image that we provide.
This is the only thing that looks wrong with what you described above.

gabrieljoelc commented 4 years ago

I am pulling from arafato/azurite as described at the bottom of https://github.com/Azure/Azurite/issues/263#issuecomment-582687766:

Azurite V2 output:

➜  ~ docker run --rm -e executable=table -t -p 10002:10002 -v ~/azurite:/opt/azurite/folder arafato/azurite
Azure Table Storage Emulator listening on port 10002
edwin-huber commented 4 years ago

that arafato Docker repo is very outdated, an not maintained by us, so I can't give you any promises on it's functionality.
I'm also not using MacOS, but I don't think that should make a difference.
I setup my EventHub function project using the latest VSCode Azure Tools, which bootstrapped / scaffolded the function project for me.

I would try running the v3 emulator from gitrepo & command line :

npm run blob

If that works, it sounds like we need to update the Docker repo.

acd-hfi commented 4 years ago

I had a similar-ish issue recently. I had to switch to loose mode to resolve it; have you tried that yet?

gabrieljoelc commented 4 years ago

@ADrews-HF interesting. I've never heard of "loose mode". how do you set that?

acd-hfi commented 4 years ago

@gabrieljoelc It seems that it was a param that was only added in v3. It is used to ignore unsupported headers in requests (which was my issue using the VSCode extension w/an Event Hub trigger). You would just add the --loose param when starting the v3 Azurite blob.

acd-hfi commented 4 years ago

Hrm. I was able to get the Azurite plugin working, but it appears to have some limitations around locking records. When using the vscode extension, locking errors always eventually lead to what looks like a deadlock situation.

edwin-huber commented 3 years ago

Hi,
we have had a lot of releases since this issue was reported. Is this still a problem for anyone on the thread?
If not I shall close this issue in a week.
Thanks, Edwin

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.