search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.85k stars 325 forks source link

Is table storage emulation ever coming? #428

Closed onionhammer closed 3 years ago

onionhammer commented 4 years ago

Its a pretty glaring hole, and it don't actually see anything in this repository tracking this issue. Can someone please reference the tracking issue? If there isn't one, there definitely should be.

Being able to develop things that depend on azure tables, like durable functions for example, on any Mac and Linux with a local storage emulator is currently not possible.

kosmiq commented 4 years ago

I'm hoping that table emulation will come around sooner rather than later.

However I have a solution for you @onionhammer , if you really need it, on a non Windows machine that allows you to dev locally. I'm a Mac user and this is my current setup allowing me to use Azure Storage Emulator running on VirtualBox with a Windows VM in it.

(As a bonus I'm also running the Cosmos DB emulator)

You'll need:

Setup:

(bonus round: Cosmos DB Emulator)

XiaoningLiu commented 4 years ago

Thanks guys! Appreciated to hear from your guys. Currently our main focus is on catching up feature parities for blob & queue storage, there is no short term plan to add table support for Azurite V3. We are also internally discussing with team and PM about table, will update if there is any news. Please leverage CosmosDB or Azurite V2 for Table emualtion for the time being.

onionhammer commented 4 years ago

@kosmiq that sounds a lot more complex than my current docker-compose up -d to spin up a sql database and azure storage from inside msbuild locally.

gunzip commented 4 years ago

a little addendum:

leverage CosmosDB

cosmosdb emulator runs on windows only. if you have windows you can use the azure storage emulator, you don't need azurite. I guess the main selling point of azurite is to have a cross platform emulator that enables non windows user to develop locally

leverage ... Azurite V2 for Table emualtion

azure durable functions need table emulation. but it looks like it's broken on azurite v2, see

https://github.com/Azure/Azurite/issues/115 https://github.com/Azure/Azurite/issues/268

so by no chance today you can run azure functions without the windows version of azure storage emulator.

including table emulation in v3 and make durable functions work looks like an important feature

atrauzzi commented 4 years ago

These are the kinds of gotchas and limitations that keep cropping up when people are trying to use Azure related tools and services.

kosmiq commented 4 years ago

@kosmiq that sounds a lot more complex than my current docker-compose up -d to spin up a sql database and azure storage from inside msbuild locally.

In my case I'm running MacOS and the docker image is only compatible with windows as far as I can tell.

MatejQ commented 4 years ago

From time to time, we need to copy all blobs from real Azure to local storage emulator. But it's not possible using azcopy which fails with nonsense exceptions. I have read somewhere that it's simply not possible because of the windows azure storage emulator which is legacy, but it is possible with Azurite which is the future. Now I read that Table storage isn't supported at all in Azurite V3 and broken in V2. Come on, MS - this is not the first time you deprecate something and don't have a usable replacement.

ayeltsov commented 4 years ago

Adding 2 cents on behalf of SQL team.

One of our smaller projects only has 1.5 people working on it, mostly in free time (read evenings/weekends).

Project uses legacy emulator for unit tests. Want to migrate unit tests to Azurite but lack of table support is blocking.

We don't extra have time or desire to use multiple products to get unit tests to work.

Something simple, lightweight, easy to install and upgrade is what such projects need.

kosmiq commented 4 years ago

What is the official way of doing local development since Azure Storage Emulator is going to be deprecated soon? Azurite v2 is old and marked as legacy, and Azurite v3 does not support Table as of yet (with no "short term plans to add it"). (https://github.com/Azure/Azurite/issues/253).

We are using blob, queue and table storage on our end and have a team running MacOS and Windows.

gabrieljoelc commented 4 years ago

Currently our main focus is on catching up feature parities for blob & queue storage, there is no short term plan to add table support for Azurite V3. We are also internally discussing with team and PM about table, will update if there is any news. Please leverage CosmosDB or Azurite V2 for Table emulation for the time being.

@XiaoningLiu this is unfortunate and makes it very difficult to develop Azure Functions locally on non-windows without a real resource, since all Azure Functions require table storage to run. I end up doing a bunch of the coding on my Mac and then running the app on my PC. :(

See my other open issue #263 where I need blob and table storage in a similar scenario so I have to try and use V3 and V2 because the V2 blob has issues. 💫

@XiaoningLiu Is there any way I could convince you to increase the priority of adding table storage to V3 (e.g. makes Azure Functions much more appealing to non-Windows peeps since there is no Docker image for Table Storage they can use)? 😅

SeanFeldman commented 4 years ago

@XiaoningLiu Is there any way I could convince you to increase the priority of adding table storage to V3 (e.g. makes selling Azure Functions to non-Windows peeps much more appealing since there is no Docker image for Table Storage we can use)? 😅

I'll ask perhaps a bit too direct question, but still - is Table Storage service deprecated?

If "yes", understandable why there's a push back to support it with Azurite. If "no", then it's weird that it's not supported, no?

Fully understand the challenges of any emulator - it adds to the product/service team to technically own and support a parallel product. At the same time, if the emulator is already there and service (Table Storage) is not deprecated, then it's putting customers in an awkwardly strange situation.

onionhammer commented 4 years ago

I really think the team needs to re-prioritize this effort. Table storage support is critical for a lot of different scenarios like durable functions

hoshauch commented 4 years ago

how should we test locally on linux? Please do it :)

XiaoningLiu commented 4 years ago

@XiaoningLiu Is there any way I could convince you to increase the priority of adding table storage to V3 (e.g. makes selling Azure Functions to non-Windows peeps much more appealing since there is no Docker image for Table Storage we can use)? 😅

I'll ask perhaps a bit too direct question, but still - is Table Storage service deprecated?

If "yes", understandable why there's a push back to support it with Azurite. If "no", then it's weird that it's not supported, no?

Fully understand the challenges of any emulator - it adds to the product/service team to technically own and support a parallel product. At the same time, if the emulator is already there and service (Table Storage) is not deprecated, then it's putting customers in an awkwardly strange situation.

Table Storage is not deprecated. It's branding movs from Azure Storage Table to Azure Table. We are internally discussing Table support in Azurite V3 with CosmosDB team. We know many are aksing Table support. The challenge is we need more resources for contribution to Azurite Table. Open source contributions are very welcome. If anyone wants to contribute to Azurite Table, like implementing some APIs, please leave message here : )

onionhammer commented 4 years ago

@XiaoningLiu maybe if the team at Microsoft could start a branch with table support and just stub out the method signatures with "TODO"s the rest could be filled in somewhat idiomatically by the community? Putting such a large task out to the community without guidance is an invitation to failure or duplication of effort

hugoeustaquio commented 4 years ago

Can you fix table storage support on V2?

takekazuomi commented 4 years ago

Is this story the same as the CosmosDB emulator's cross-platform support?

XiaoningLiu commented 4 years ago

@XiaoningLiu maybe if the team at Microsoft could start a branch with table support and just stub out the method signatures with "TODO"s the rest could be filled in somewhat idiomatically by the community? Putting such a large task out to the community without guidance is an invitation to failure or duplication of effort

In recent discussions, we will prepare guidelines, project skeletons and open details tasks. Each task will have detailed description and should be fine greained for anyone to pick up. I'm busy with new features for Azurite blob/queue, and cannot invest too much on Table. We already have some Microsoft internal devs willing to contribute, and here like more contributions from open source to speed up the project.

XiaoningLiu commented 4 years ago

Can you fix table storage support on V2?

V2 table has different structure with v3. @edwin-huber is helping with V2 releasing and reviewing, Open source contribution to table v2 is also welcome.

XiaoningLiu commented 4 years ago

Is this story the same as the CosmosDB emulator's cross-platform support?

What's the CosmosDB emulator's cross-platform support you are referring to? In my recent talk with CosmosDB team, they are generally agree to have Table in Azurite V3 for better cross-platform support.

takekazuomi commented 4 years ago

@XiaoningLiu CosmosDB Emulator is under development. I have been waiting for more than 3 years. I'm tired of waiting. https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/18533509-add-documentdb-emulator-support-for-mac-os-x-and

edwin-huber commented 4 years ago

Hi,
I appreciate your pain.
Maintaining v2 table is not very practicable any more as I can't keep up with the number of features that need backporting to keep up with the current versions of our APIs.
We are working out the details for v3 and will update as soon as we can with something more concrete.

maartenba commented 4 years ago

Would also like to see this supported, especially to be able to work with durable functions locally (without having to depend on connectivity to an actual storage account to do development). We have a lot of developers on macOS and taking the connectivity constraint into account, they currently have no option other than booting into Windows and using the "old" storage emulator

oising commented 4 years ago

Need to add the point that alongside azure functions, Orleans also requires table storage for some important scenarios... :/

XiaoningLiu commented 4 years ago

Hi guys, after serveral rounds of discussion and review. Azurite V3 is going to support Table. It's great if anyone likes to contribute to Azurite V3 Table. Use following queries for all opening TODO items:

Other references: Azurite V3 Table GitHub Branch: https://github.com/Azure/Azurite/tree/table Azurite V3 Table GitHub Project: https://github.com/Azure/Azurite/projects/2 Azurite V3 Table Wiki (under development): https://github.com/Azure/Azurite/wiki/Azurite-V3-Table

edwin-huber commented 3 years ago

Table support added in preview.
GA soon.