search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.83k stars 325 forks source link

table upsert is very slow. 100k records takes more than 1 hour #2321

Open RudiSzalai opened 11 months ago

RudiSzalai commented 11 months ago

Which service(blob, file, queue, table) does this issue concern?

table

Which version of the Azurite was used?

3.28

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm

What's the Node.js version?

20.9

What problem was encountered?

slow performance

Steps to reproduce the issue?

upsert 100k records. This takes more than 1 hour.

Have you found a mitigation/solution?

no. Running it in memory does not make the upsert any faster.

EmmaZhu commented 11 months ago

Hi @RudiSzalai ,

Current Azurite is only an emulator for test environment to see whether an application works before connecting to Azure. There's no performance and TPS guarantee.

We'd love to let Azurite to be able to help in more scenarios. Could you share more details about your scenario, like under what situation you'd need Azurite to handle larger scales of operations with much better performance?

Thanks Emma

RudiSzalai commented 11 months ago

Hi @EmmaZhu, I'm working with Azure. I pull some information from it and process it. There are a few corner cases in the ~100k records I collect which require different processing, so when I try to truncate the input data I put into the azurite table I miss some of these cases. So I need to do the non-production testing on the same amount of data as it will run in Azure.