search

icecube / skyreader

An API for Results Produced by SkyDriver & the Skymap Scanner
MIT License
0 stars 0 forks source link

Writing JSON from `npz` fails because `int64 is not JSON serializable` #14

Closed mlincett closed 1 year ago

mlincett commented 1 year ago

I am trying to convert old npz test data to JSON in order to make CI testing of skymap_scanner more transparent. Although such conversion seems to fail.

python examples/npz_to_json.py --input-file skymap_scanner/tests/data/results_npz/millipede_wilks/run00127907.evt000020178442.HESE_1.npz

[...]

TypeError: Object of type int64 is not JSON serializable
kjmeagher commented 1 year ago

json doesn't understand numpy types, you can call array.tolist() before trying to serialize

mlincett commented 1 year ago

json doesn't understand numpy types, you can call array.tolist() before trying to serialize

Makes sense. I was just wondering whether or not this used to work in the past, considering that the CI test seems to output JSON just fine.

I guess when we build SkyScanResult from the runtime scanner output it does not use the numpy type for the internal representation but it does when loading a npz.

mlincett commented 1 year ago

The python exception is annoyingly cryptic about where the problem is. However since the JSON file is sequentially written just up to where the error occurs, it appears that the problem are the metadata (likely the nside field).

Edit: this is due to us storing metadata in the dtype.metadata of numpy, something I'd like to move away from (see new issue #15).