search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.08k stars 21.13k forks source link

Need more information on Azure AI Search partitions. #123580

Closed ayushcodes closed 1 day ago

ayushcodes commented 2 days ago

Hi Team

I am working on Azure AI Service and we are using the Standard tier (S1).

As per the document each of the partitions will have 160GB of storage so if we create 12 partitions the storage will be 1.92 TB per index.

I have the following questions.

Is my understanding correct? Will we get the 1.92 TB per index for all 50 indexes( total limit in S1 SKU)?

Please add more information regarding partitions and index size if I missed any.

Document referred:- https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity#service-limits

Thanks in advance Ayush Garg

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

TPavanBalaji commented 2 days ago

@ayushcodes Thank you for bringing this to our attention. I've delegated this to content author, who will review it and offer their insightful opinions.

HeidiSteen commented 1 day ago

Hi @ayushcodes, on S1, if you create an index that uses ALL of the partitions, you will have one index per search service. An index of that size will use up all available storage for that service.

This is just my opinion, but I think you should try to work with somebody from MSFT if your indexes are going to be that big. Starting a new index at the maximum limits seems risky. There are other limits (vector index size, API request), and performance issues you might run into, and you might need advice to land a good experience.

There are things you can do to make your index smaller, like excluding fields that don't add value, and using compression options. Or, if you don't want to work with MSFT directly just yet, try reaching out on Stack Overflow to see if you can get advice from people who have very large indexes. That's the best forum for community advice and its very good. https://stackoverflow.com/questions/tagged/azure-cognitive-search Just remember the new SKUs and larger/faster partitions just launched this year (April and May) and so old comments probably don't apply.

HeidiSteen commented 1 day ago

Also, the section on index size limits says that on tiers S1-S3, the maximum index size is all available storage on the service, so I don't believe I need a documentation update. For this reason, I'm going to close the issue. Feel free to reply back if you disagree.

please-close