search

isamplesorg / isamples_inabox

Provides functionality intermediate to a collection and central
0 stars 1 forks source link

Exporting json result sets lacks the ability to append #357

Closed dannymandel closed 3 months ago

dannymandel commented 4 months ago

In the petl API, there is no way to append to an existing file when exporting JSON. Unlike (CSV), the JSON doesn't support this mode of operation. Additionally, JSON doesn't stream the output, it is all read into memory.

For this reason, we should just limit the JSON result sets to 100k or 200k or whatever we decide the max single request should be. If petl ever changes this, we can change the JSON behavior.

dannymandel commented 4 months ago

This limitation also applies to JSON lines, as it uses the same petl API.

dannymandel commented 3 months ago

Not to be fixed.