search

explosion / srsly

🦉 Modern high-performance serialization utilities for Python (JSON, MessagePack, Pickle)
MIT License
432 stars 31 forks source link

Add numbin as another serialization library #88

Open kemingy opened 1 year ago

kemingy commented 1 year ago

Hi, I'm the author of numbin (An efficient binary serialization format for numpy data.) I just found this repo and I wonder if you will accept numbin as another serialization library.

Check the benchmark here.

BTW, I don't know there is a library called msgpack-numpy before I develop numbin. After explore the source code, I think numbin would provide better performance and flexibility.

adrianeboyd commented 1 year ago

Thanks for the pointer! To a large extent srsly is developed for the formats we're using for Explosion packages/projects and my initial feeling is that we're unlikely to replace msgpack-numpy with another option unless there are large advantages because it could be complicated to handle a lot of the existing saved data.

We do occasionally run into the size limits for items saved with msgpack, but it looks like this would be the same for numbin?

kemingy commented 1 year ago

We do occasionally run into the size limits for items saved with msgpack, but it looks like this would be the same for numbin?

That depends. Numbin doesn't require msgpack if your data is just NumPy array. It's only required when you have the mixed data type.

I'm not sure about the msgpack size limitation. Will take a look.