Air-Sea CO₂ Flux dataset throws 500 on the `/statistics` endpoint

[dzole0311]

I noticed that choosing Delaware, Arizona, Indiana and a few other presets and running the analysis, the Air-Sea CO₂ Flux dataset errors out with a 500 on the statistics endpoint. It works though if I choose California as a preset.

It can be reproduced here.

Maybe related: 232

[j08lue]

@amarouane-ABDELHAK or @slesaad - would you be able to check what is going wrong in the backend here, please?

[slesaad]

@j08lue will do!

[slesaad]

The errors we're getting in the titiler API lambda:

GDAL signalled an error: err_no=1, msg='PROJ: cea: Invalid latitude'

and

ZeroDivisionError: Weights sum to zero, can't be normalized

trace

``` [ERROR] 2024-05-29T18:50:59.623Z d5663d57-9e91-40f7-b656-37761227c4bb An error occurred running the application. Traceback (most recent call last): File "/tmp/pip-target-r4n_td5g/lib/python/anyio/streams/memory.py", line 98, in receive File "/tmp/pip-target-r4n_td5g/lib/python/anyio/streams/memory.py", line 93, in receive_nowait anyio.WouldBlock During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/base.py", line 78, in call_next File "/tmp/pip-target-r4n_td5g/lib/python/anyio/streams/memory.py", line 118, in receive anyio.EndOfStream During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/pip-target-r4n_td5g/lib/python/mangum/protocols/http.py", line 66, in run File "/tmp/pip-target-r4n_td5g/lib/python/fastapi/applications.py", line 1054, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/applications.py", line 122, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/errors.py", line 184, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/errors.py", line 162, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/base.py", line 108, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/src/app.py", line 195, in add_correlation_id File "/tmp/pip-target-r4n_td5g/lib/python/aws_lambda_powertools/tracing/tracer.py", line 581, in decorate File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/base.py", line 84, in call_next File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/base.py", line 70, in coro File "/tmp/pip-target-r4n_td5g/lib/python/starlette_cramjam/middleware.py", line 112, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette_cramjam/middleware.py", line 142, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/titiler/core/middleware.py", line 63, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/cors.py", line 83, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/exceptions.py", line 79, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/exceptions.py", line 68, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/routing.py", line 718, in __call__ File "/tmp/pip-target-r4n_td5g/lib/python/starlette/routing.py", line 276, in handle File "/tmp/pip-target-r4n_td5g/lib/python/starlette/routing.py", line 66, in app File "/tmp/pip-target-r4n_td5g/lib/python/src/monitoring.py", line 41, in route_handler File "/tmp/pip-target-r4n_td5g/lib/python/fastapi/routing.py", line 299, in app File "/tmp/pip-target-r4n_td5g/lib/python/fastapi/routing.py", line 294, in app File "/tmp/pip-target-r4n_td5g/lib/python/fastapi/routing.py", line 193, in run_endpoint_function File "/tmp/pip-target-r4n_td5g/lib/python/starlette/concurrency.py", line 41, in run_in_threadpool File "/tmp/pip-target-r4n_td5g/lib/python/anyio/to_thread.py", line 33, in run_sync File "/tmp/pip-target-r4n_td5g/lib/python/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread File "/tmp/pip-target-r4n_td5g/lib/python/anyio/_backends/_asyncio.py", line 807, in run File "/tmp/pip-target-r4n_td5g/lib/python/titiler/core/factory.py", line 490, in geojson_statistics File "/tmp/pip-target-r4n_td5g/lib/python/rio_tiler/models.py", line 776, in statistics File "/tmp/pip-target-r4n_td5g/lib/python/rio_tiler/utils.py", line 191, in get_array_statistics File "/tmp/pip-target-r4n_td5g/lib/python/rio_tiler/utils.py", line 52, in _weighted_stdev File "/tmp/pip-target-r4n_td5g/lib/python/numpy/lib/function_base.py", line 550, in average ZeroDivisionError: Weights sum to zero, can't be normalized ```

@vincentsarago might need your intervention here

Request URL: https://ghg.center/api/raster/cog/statistics?url=s3:%2F%2Fghgc-data-store%2Feccodarwin-co2flux-monthgrid-v5%2FECCO-Darwin_CO2_flux_202211.tif&nodata=nan&dst_crs=%2Bproj%3Dcea

body:

```json { "type": "Feature", "properties": {}, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ -75.78859, 39.7222 ], [ -75.7669, 39.37765 ], [ -75.76689, 39.3775 ], [ -75.76044, 39.29679 ], [ -75.75643, 39.24669 ], [ -75.74815, 39.14313 ], [ -75.7231, 38.82983 ], [ -75.70755, 38.63539 ], [ -75.70755, 38.63534 ], [ -75.70177, 38.56077 ], [ -75.70038, 38.54274 ], [ -75.69372, 38.46013 ], [ -75.47928, 38.4537 ], [ -75.34128, 38.45244 ], [ -75.18545, 38.45102 ], [ -75.04893, 38.45127 ], [ -75.05397, 38.53628 ], [ -75.0718, 38.6965 ], [ -75.08947, 38.7972 ], [ -75.11333, 38.783 ], [ -75.15902, 38.7902 ], [ -75.23203, 38.84426 ], [ -75.30408, 38.91316 ], [ -75.30255, 38.939 ], [ -75.30665, 38.94766 ], [ -75.34089, 39.01996 ], [ -75.39628, 39.05789 ], [ -75.40747, 39.13371 ], [ -75.39479, 39.18836 ], [ -75.40837, 39.2647 ], [ -75.46932, 39.33082 ], [ -75.50564, 39.3704 ], [ -75.52168, 39.38787 ], [ -75.57183, 39.4389 ], [ -75.59307, 39.47919 ], [ -75.52809, 39.49812 ], [ -75.52767, 39.53528 ], [ -75.51273, 39.578 ], [ -75.54396, 39.596 ], [ -75.55944, 39.62981 ], [ -75.53514, 39.64721 ], [ -75.50974, 39.68611 ], [ -75.47764, 39.71501 ], [ -75.45944, 39.76581 ], [ -75.41506, 39.80192 ], [ -75.48121, 39.82919 ], [ -75.57043, 39.83919 ], [ -75.57965, 39.83742 ], [ -75.59256, 39.83493 ], [ -75.59432, 39.8346 ], [ -75.66284, 39.82143 ], [ -75.71706, 39.79233 ], [ -75.75323, 39.75799 ], [ -75.77378, 39.7222 ], [ -75.78859, 39.7222 ] ] ] ] } } ```

This is the file:

ECCO-Darwin_CO2_flux_202211.tif.zip

[j08lue]

Great report, @slesaad, thanks!

The /statistics request currently succeeds for me:

{
"b1": {
"min": -0.031021814920656237,
"max": 0.0307270460145791,
"mean": -0.000021450558869718596,
"count": 338606,
"sum": -7.263287936639935,
"std": 0.000443675030901694,
"median": -0.00000347941618740146,
"majority": -0.031021814920656237,
"minority": -0.031021814920656237,
"unique": 338606,
"histogram": [
[
9,
0,
0,
0,
19345,
319192,
0,
0,
0,
60
],
[
-0.031021814920656237,
-0.024846928827132703,
-0.01867204273360917,
-0.012497156640085636,
-0.0063222705465621025,
-0.00014738445303856887,
0.006027501640484965,
0.012202387734008498,
0.018377273827532032,
0.024552159921055566,
0.0307270460145791
]
],
"valid_percent": 64.46,
"masked_pixels": 186706,
"valid_pixels": 338606,
"percentile_2": -0.00018335218910390857,
"percentile_98": 0.00006901845370872633
}
}

But when looking at the /info, I see what GDAL seems to be complaining about in your traceback above:

{
"bounds": [
-180.125,
-90.124826629681,
179.875,
89.875173370319
],
"minzoom": 0,
"maxzoom": 2,
"band_metadata": [
[
"b1",
{}
]
],
"band_descriptions": [
[
"b1",
"Air-sea CO2 Flux"
]
],
"dtype": "float64",
"nodata_type": "None",
"colorinterp": [
"gray"
],
"driver": "GTiff",
"count": 1,
"width": 1440,
"height": 721,
"overviews": [
2,
4
]
}

i.e. latitudes ranging from -90.124826629681 to 89.875173370319. -90.12 is obviously out of bounds. We should check whether something went wrong during the creation of the ECCO-Darwin files.

[j08lue]

• https://github.com/US-GHG-Center/ghgc-data/issues/20

[SwordSaintLancelot]

This issue was caused by selecting the wrong area on the map. @j08lue already mentioned that we do get the statistics for the COGs. Can we close this issue as it seems to be resolved? @slesaad

[slesaad]

@SwordSaintLancelot since we have another ticket for wrong bounds, we can close this