Closed loichuder closed 3 years ago
I've added a query option "ignore_nan" for GET values that will return null rather than NaN in the json response. Try out the nanvalue branch. Commit is here: https://github.com/HDFGroup/hsds/commit/66a0dc0bef166eb78149b952a601171222528d82.
Haven't added support for sql query operations or attribute data yet, but let me know if this resolves the immediate issue.
BTW - Is there a mean to use binary responses from javascript? (guess you would need the javascript equivalent of numpy). Binary vs JSON would be more efficient esp. for larger responses.
I've added a query option "ignore_nan" for GET values that will return null rather than NaN in the json response. Try out the nanvalue branch. Commit is here: 66a0dc0.
Haven't added support for sql query operations or attribute data yet, but let me know if this resolves the immediate issue.
Yup, it fixes the issue for NaN/Infinity
in dataset data. However, as you said, NaN
in attribute data or in fillValue
(which get serialized in the metadata) will still produce invalid JSON.
BTW - Is there a mean to use binary responses from javascript? (guess you would need the javascript equivalent of numpy). Binary vs JSON would be more efficient esp. for larger responses.
This is something that we are actively exploring right now as it would indeed be more efficient and solve such issues.
For the record, in h5web, we are using ndarray as the "JS numpy" which make projects like js-numpy-parser especially interesting to us.
I've updated the branch to return the string "nan" in the fillValue for GET dataset. As with values, you'll need to add "ignore_nan" in the query option. See the commit here: https://github.com/HDFGroup/hsds/commit/641eb6b59892c2bdf2ef7e196116e2fd74dd5c31.
I'll look at attribute support next.
And added attribute support here: https://github.com/HDFGroup/hsds/commit/e8441629fabe3ac83c6c7ff0a778e33a75566f30.
I think this should be it for supporting possible NaN data responses.
Merged to master. Will close issue - please reopen if you run into any problems.
Sorry for not following this up earlier. It works fine :+1: !
The only missing piece would be that the JSON is still invalid when fillValue
is set to (-)Infinity
.
To be completely upfront, we changed our approach on h5web: we search for NaN/Infinity
in the text response to replace them by their string equivalent before JSON deserialization. This allows to have valid JSON and still distinguish between NaN
and Infinity
, which is something we cannot do with ignore_nan
as it converts both to null
.
Say a dataset was created by h5py in the following fashion:
The dataset info created by GET_Dataset produces then a JSON with a field:
"creationProperties": {"fillValue": NaN}
.Unquoted
NaN
is not supported by the JSON spec so this cannot be parsed:The presence of this unsupported value is explained the JSON serialization which is made by
aiohttp.web.json_response
which in turn use json.dumps. The default behaviour ofjson.dumps
is to allow unquotedNaN
.A possible way to solve this would be to parametrize
aiohttp.web.json_response
to quoteNaN
or to sendnull
instead:Edit: This issue also pops when getting data that contains NaN