search

cdisc-org / cdisc-rules-engine

Open source offering of the cdisc rules engine
MIT License
46 stars 12 forks source link

Fix JSON reader to Parquet #701

Closed nhaydel closed 4 months ago

nhaydel commented 4 months ago

This PR fixes the to_parquet method in the json reader class.

It now writes the raw dataset to the tempfile instead of converting to dask first.

Steps to test:

  1. Force the engine to use dask datasets. you can do this by setting the DATASET_SIZE_THRESHOLD environment variable to 0
  2. Run a validation with dataset json. (I used this one for testing: TS_neg.json
  3. Verify the validation runs completely.