search

microsoft / PlanetaryComputer

Issues, discussions, and information about the Microsoft Planetary Computer
https://planetarycomputer.microsoft.com/
MIT License
185 stars 9 forks source link

rate limits #191

Closed mhungen closed 1 year ago

mhungen commented 1 year ago

has there been a change in the request limit since 2023-02-23?

when I acquire a new SAS token and repeatedly (~4-8) request 30KB of an image with curl:

curl -r 30720-61440 "https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/33/U/WP/2023/02/09/S2A_MSIL2A_20230209T100151_N0400_R122_T33UWP_20230210T042342.SAFE/GRANULE/L2A_T33UWP_A039873_20230209T100209/IMG_DATA/R10m/T33UWP_20230209T100151_TCI_10m.tif?st=2023-02-22T11%3A50%3A36Z&se=2023-02-23T12%3A35%3A36Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2023-02-23T09%3A45%3A47Z&ske=2023-03-02T09%3A45%3A47Z&sks=b&skv=2021-06-08&sig=ae2HGGdy%2Bpm6Lxpy0A%2B7pnHUTYiOlmE1A/HZMgzcbFg%3D"

i get the following error:

<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>ServerBusy</Code>
    <Message>
        Egress is over the account limit.
        RequestId:67a2f8ed-501e-0088-6e7d-470fa7000000
        Time:2023-02-23T11:54:29.1447165Z
    </Message>
</Error>

i ran the same test with a token obtained with a subscription key. the limits are slightly higher but still much lower than a few days ago.

are these limits intentional? if so, where can i get more information on the limits? the documentation does not specify exact limits, only that they are generous.

TomAugspurger commented 1 year ago

I'm looking into this, but just a quick note: the rate limits discussed at https://planetarycomputer.microsoft.com/docs/concepts/sas/#supplying-a-subscription-key refer to rate limits on the SAS API (so getting the token in the first place).

Once you have the token, there's a second set of limits / performance targets on the Storage Account itself. We recommend that your applications accessing the data watch for and retry on certain response codes. See https://learn.microsoft.com/en-us/azure/storage/blobs/scalability-targets for the codes to watch out for and https://learn.microsoft.com/en-us/azure/architecture/patterns/retry for background.

TomAugspurger commented 1 year ago

Closing this, since the issue was with the storage account. Let us know if you keep seeing errors even with retries set.