dosubot[bot]
commented
1 month ago To resolve the "PropertyValueTooLarge" error when trying to index large files using Azure Table Storage, you need to split the large property value into multiple parts. Here is a function split_large_property_value that can help with this:
def split_large_property_value(num_parts: int, bytes_val: str, key: str) -> dict:
"""Split a large property value into multiple parts."""
parts = {}
for i in range(num_parts):
start_index = i * STORAGE_MAX_ITEM_PROPERTY_VALUE_SIZE
end_index = start_index + STORAGE_MAX_ITEM_PROPERTY_VALUE_SIZE
# Convert back from UTF-16 bytes to str after slicing safely on character boundaries
serialized_part = bytes_val[start_index:end_index].decode(
"utf-16", errors="ignore"
)
parts[f"{key}{STORAGE_PART_KEY_DELIMITER}{i + 1}"] = serialized_part
return partsTo use this function, follow these steps:
- Serialize the value to a JSON string and encode it to UTF-16 bytes.
- Compute the number of parts required to split the value.
- Split the value into parts using the
split_large_property_valuefunction. - Store the parts in Azure Table Storage.
Here is an example of how you can do this:
def serialize_and_split_large_property(key: str, value: Any) -> dict:
serialized_val, bytes_val, val_length = serialize_and_encode(value)
num_parts = compute_table_property_part_count(val_length)
return split_large_property_value(num_parts, bytes_val, key)
# Example usage
large_value = "..." # Your large value here
key = "largeProperty"
split_parts = serialize_and_split_large_property(key, large_value)
# Now you can store `split_parts` in Azure Table StorageAdditionally, if you are using the AzureDocumentStore from the LlamaIndex framework, it handles splitting nodes across multiple properties to stay within Azure Table Storage limits. This can be particularly useful for large documents, as it allows you to store data up to the 1MiB "Item" limit by splitting the data across multiple properties [1][2].
To continue talking to Dosu, mention @dosu.
prabhuperumalofficial
Question Validation
Question
We are trying to use Azure Table Storage to store indexes as below
But it throws the following error when trying to index a larger file
2024-08-09 12:24:55,996 - ERROR - The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less. RequestId:5897b37c-a002-000e-5257-ea4e3b000000 Time:2024-08-09T12:24:55.9923447Z ErrorCode:PropertyValueTooLarge Content: {"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:5897b37c-a002-000e-5257-ea4e3b000000\nTime:2024-08-09T12:24:55.9923447Z"}}} Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/azure/data/tables/_table_client.py", line 658, in upsert_entity self._client.table.update_entity( File "/usr/local/lib/python3.12/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/azure/data/tables/_generated/operations/_operations.py", line 1280, in update_entity raise HttpResponseError(response=response, model=error) azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Bad Request' Content: {"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:5897b37c-a002-000e-5257-ea4e3b000000\nTime:2024-08-09T12:24:55.9923447Z"}}}