Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.8k stars 320 forks source link

Artifically limit IOPS for blob storage with Azurite #2084

Open twhiting opened 1 year ago

twhiting commented 1 year ago

Which service(blob, file, queue, table) does this issue concern?

blob, also files?

Feature Request - Configurable IOPS limits

In azure a storage account can be limited by IOPS (IO operations per second). Azurite does not appear to support configuring an IOPS limit.

Currently the only way of testing IOPS limitations (to my knowledge) is by using a real storage account & accruing the associated costs. Alot of the IOPS limits in azure blob storage are quite high, and difficult to hit in a pre-production environment.

Is this possible now with Azurite or can it be added in the future?

blueww commented 1 year ago

@twhiting

Thanks for raising this request!

Azurite is an Azure Storage rest API emulator; we still not support IOPS limitation. And we are working on some new storage feature implementation recently, so this might won't be out priority recently.

Azurite welcome contribution! It would be greate if you could raise a PR to implement this! (Please first review design with us than raise PR, to make a smooth PR review.)

If you would like to test that on product Azure account, and would like your azure account has a low IOPS limitation, you might can raise a help ticket on Azure portal and see if server team can help you on that. image

twhiting commented 1 year ago

@blueww Thanks for the reply, I see and agree that actually testing this issue is beyond the scope of this project.

Is there some mechanism to tell azurite to return the proper throttling error code within the scope of the API?

Do you think that this is beyond the scope of this project?


I want to essentially write a test that runs entirely within azurite that exposes the same error code that would be returned by the azure blob storage api for this throttling scenario.

blueww commented 1 year ago

@twhiting

Currently Azurite doesn't return throttling error.

However, you can modify the code a little to return the error.

  1. Clone the Azurite repo to local.
  2. Define the throttling error in https://github.com/Azure/Azurite/blob/main/src/blob/errors/StorageErrorFactory.ts
    public static getThrottling(
    contextID: string = DefaultID
    ): StorageError {
    return new StorageError(
      503,
      "ServerBusy",
      "Egress is over the account limit.",
      contextID
    );
    }
  3. Add the following code at the begin of the API that you would like it always fail with throttling. Like for Get Blob properties, should add to https://github.com/Azure/Azurite/blob/9e221953fbaf809315dcc8bd1225a7e0a4dab760/src/blob/handlers/BlobHandler.ts#L106
      throw StorageErrorFactory.getThrottling(
        context.contextId
      );

Then follow https://github.com/Azure/Azurite/#github to build and run Azurite from you local code, then the API will report throttling error.