search

mverleg / pyjson_tricks

Extra features for Python's JSON: comments, order, numpy, pandas, datetimes, and many more! Simple but customizable.
Other
154 stars 23 forks source link

Serialize scalars and 0-dimensional arrays #99

Open hmaarrfk opened 11 months ago

hmaarrfk commented 11 months ago

closes https://github.com/mverleg/pyjson_tricks/issues/98

hmaarrfk commented 11 months ago

The test you added in https://github.com/mverleg/pyjson_tricks/commit/ac5c8bea5e78315a1036738baeab5bbe631f1794 is in direct conflict with some of the spirits here of serializing numpy dtypes in a round trip way.

mverleg commented 11 months ago

The test you added in ac5c8be is in direct conflict with some of the spirits here of serializing numpy dtypes in a round trip way.

Yeah I get that, but I think it could be useful to preserve backwards compatibility (and test to ensure we do so). That's what I meant in the previous review.

We've been doing it this way, it's be a (slightly) breaking change to start doing it differently.

Maybe it's useful enough to break compatibility but I'm not sure. I guess people who really like the old format could use the primitives=True flag. I'll think about it.

mverleg commented 10 months ago

I think it's best to keep this one for consideration in version 4.0 because of the backwards compatibility concerns

hmaarrfk commented 10 months ago

honestly, I think the challenges with float vs np.float64 occur because the default json parser is used as a base. If you want, we can have a heavy handed approach, effectively taking _make_iterencode and adding a stricter check against float type.

ultimately, you could "solve" this issue by trying to use an other json parser like orjson or something other.

Your choice.

This isn't a deal breaker for me just at this exact moment, however, round tripping is something that is quite important to us and this might make me look elsewhere as a user as the challenges increase.

Understanding your timeline for 4.0 would be helpful in us making our decision to stick with pyjson_tricks and continue to contribute to this wonderful project!

hmaarrfk commented 3 months ago

This somewhat came up as we are reviewing our test suite and trying to cleanup our warnings. It seems that the main thing that was left is trying to decide on how to deal with

Maybe it's useful enough to break compatibility but I'm not sure. I guess people who really like the old format could use the primitives=True flag. I'll think about it.

Is that in fact the only outstanding issue? it might be possible if I create enough test cases to find a pattern for how to address these old datasets.