search

milvus-io / milvus-haystack

Apache License 2.0
6 stars 12 forks source link

write_documents failing for bigger messages #5

Open akshaykarle opened 8 months ago

akshaykarle commented 8 months ago

When running write_documents I get the following error:

grpc RpcError: [query], <_MultiThreadedRendezvous: StatusCode.RESOURCE_EXHAUSTED, grpc: trying to send message larger than max (69004574 vs. 67108864)>, <Time:{'RPC start': '2023-10-16 11:46:26.850932', 'gRPC error': '2023-10-16 11:46:38.551411'}>

I noticed in _create_collection we set the max_length. is there a reason we need to set that? Is that possibly causing this to fail? If so, can we remove and/or allow overriding it?

xiaofan-luan commented 8 months ago

the batch size is too large.

try to reduce the batch to less than 4M, currently it paased the default 64M setting

akshaykarle commented 8 months ago

Okay, that's what I thought. Will try reducing the batch_size and see if it works. Thanks for your quick reply. Will get back to you once I've tested it and we can possibly close this of

apohllo commented 7 months ago

It's a pitty this is not a parametr I can change in the call. Changing the library's code is the only way to achieve that?

apohllo commented 7 months ago

Ok. So it is a parameter, my bad. But... get_all_documents does not use a batch size, so if the size of docs is larger than the message, there's the same error.