search

sassoftware / python-sasctl

Python package and CLI for user-friendly integration with SAS Viya
https://sassoftware.github.io/python-sasctl
Apache License 2.0
46 stars 41 forks source link

write_var_json generates NaN values in the JSON files, which is not a valid JSON value. #160

Closed ATFischer closed 1 year ago

ATFischer commented 1 year ago

Describe the issue write_var_json generates NaN values in the JSON files, which is not a valid JSON value. As a result, the imported models can't be compared in SAS Model Manager.

To Reproduce Train a model and use the write_var_json method.

Expected behavior All missings should have the value null.

Stack Trace If you're experiencing an exception, include the full stack trace and error message.

Version 1.9.0

smlindauer commented 1 year ago

I caught the issue within the calculate_model_stats function of write_json_files. Unfortunately, the Python json package does not include a method or option to adjust the float NaN's to null. I know that the simplejson package includes a way to adjust the values, but I want to avoid adding additional dependency requirements.

I found what I believe is the crux point of where the NaN values are showing up (caused by translation of the SAS dataframe output from the ASSESS action) and put in a check with pandas to replace float NaN's with Nonetype objects. These will convert to null when dumped to the json object.

Commit 4684052 should fix the issue.