search

CAVEconnectome / CAVEclient

This is the python client for accessing REST APIs within the Connectome Annotation Versioning Engine.
https://caveconnectome.github.io/CAVEclient/
MIT License
21 stars 11 forks source link

relax pyarrow version requirements? #116

Open jefferis opened 1 year ago

jefferis commented 1 year ago

can we allow pyarrow>=12 now? https://github.com/seung-lab/CAVEclient/blob/7fbc0e20c3b42236989e4f103411454b92111b81/requirements.txt#L2

It would require additional logic for the deserialize step here (to check the version of pyarrow / if pa.deserialize is available) :

https://github.com/seung-lab/CAVEclient/blob/7fbc0e20c3b42236989e4f103411454b92111b81/caveclient/materializationengine.py#L41

so perhaps you don't want to do it yet.

ceesem commented 1 year ago

The logic is already in there to not use the older pyarrow code path if you don't need to — if both the server and the client are running up to date versions, everything should be compatible with the more recent versions of pyarrow. However, we haven't fully tested it in that situation, and more importantly we also wanted to make sure that there is a period where we keep things overlapping with a warning while some projects we know of (not the ones managed by us) have not yet updated their materialization server.

If you want to try to force upgrade pyarrow beyond 12 in your own environment, we think it should work (and we should make sure of that as well).

jefferis commented 1 year ago

Thanks @ceesem. Yes forcing pyarrow to 12 is what I am currently doing. The reason I care is that I see crashes in R when running a separate R arrow-based process for those with different pyarrow and R arrow major versions. I haven't tracked down why but right now force upgrading pyarrow is the only solution I can find. Arrow 11 displays a different bug with querying feather files on disk.