Response MediaType CSV failing

[jonaraphael]

Sometimes the CSV format download seems to fail. f=geojson does NOT fail in these same circumstances.

Here it works (4 rows of data): https://api.cerulean.skytruth.org/collections/public.aoi_type/items https://api.cerulean.skytruth.org/collections/public.aoi_type/items?f=csv

Here it fails (expect 141 rows of data, but returns 2 rows of garbled data): https://api.cerulean.skytruth.org/collections/public.slick_plus/items?sortby=slick_timestamp&datetime=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z&filter=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000 https://api.cerulean.skytruth.org/collections/public.slick_plus/items?sortby=slick_timestamp&datetime=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z&filter=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000&f=csv

[vincentsarago]

@jonaraphael thanks for the report,

It's interesting because using other streaming output format like ndjson seems to work fine

$ curl https://api.cerulean.skytruth.org/collections/public.slick_plus/items?sortby=slick_timestamp&datetime=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z&filter=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000&f=ndjson | wc -l 
10

$ curl https://api.cerulean.skytruth.org/collections/public.slick_plus/items?sortby=slick_timestamp&datetime=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z&filter=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000&f=csv | wc -l 
2

[vincentsarago]

Found the bug 🙏

https://github.com/developmentseed/tipg/blob/8483d5451a9f8ce5b171e120aeb338c2e2c30ea8/tipg/factory.py#L766

curl https://api.cerulean.skytruth.org/collections/public.slick_plus/items\?sortby\=slick_timestamp\&datetime\=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z\&filter\=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000\&f\=csv\&offset\=2 | wc -l    

[vincentsarago]

Note: https://github.com/developmentseed/tipg/pull/160 will be merged an released into a 0.6.0 release but I'll also backport this to a 0.5.* version so you can redeploy cerulean_cloud

[jonaraphael]

Thank you!

[aemonm]

@vincentsarago can you confirm this fix was backported into 0.5.*. I'm still seeing it on our latest deployments.

[vincentsarago]

@aemonm 🤔 I think I did https://github.com/developmentseed/tipg/compare/0.5.7...0.5.8

I'm not remembering well what was the issue to be honest