Json files too large to parse

GSA / datagov-wptheme

Data.gov WordPress Theme (obsolete)

https://www.data.gov

Other

1.88k stars 411 forks source link

Json files too large to parse #922

Closed HVoltBb closed 2 years ago

HVoltBb commented 2 years ago

https://labs.data.gov/dashboard/offices/qa For example, the second row of this table is blank due to the very large json file. As a solution to this issue, you can parse the json file as a stream instead of as a single block, which is already too large for the current system.

As more and more datasets are posted on data.gov, we would expect the json files to continue to grow, and see more blank rows in the future, so please fix this issue now.

mogul commented 2 years ago

Thanks for your report! Unfortunately the logic for doing validation on JSON files currently depends on everything being loaded at once and it would be a major rewrite to have the validation operate in a streaming fashion.

We're planning to replace the dashboard as a whole using other functionality; see https://github.com/GSA/datagov-deploy/issues/1431. (You can also see all the other issues associated with that one if you install the ZenHub browser plugin.)