search

developmentseed / titiler

Build your own Raster dynamic map tile services
https://developmentseed.org/titiler/
MIT License
755 stars 156 forks source link

Encoding issue using asset name with underscore `_` in expression #805

Closed emmanuelmathot closed 4 months ago

emmanuelmathot commented 4 months ago

Problem description

Doing a request similar to /stac/tiles/WebMercatorQuad/9/451/76.png?url=s3%3A%2F%2Fdc-development-catalog%2Fcalls%2Fcall-969%2FcalibratedDatasets%2FS2A_MSIL2A_20230928T042551_N0509_R133_T54XVM_20230928T053752-calibrated%2FS2A_MSIL2A_20230928T042551_N0509_R133_T54XVM_20230928T053752-calibrated.json&rescale=0%2C1&resampling_method=average&resampling=nearest&expression=where((s2cloudless_clp%3E0.5)%2C1%2C0)%3B&asset_as_band=true

Titiler returns the following error:

{
  "detail": "NumExpr 2 does not support Unicode as a dtype."
}

The error is raised when the expression parameter contains a reference to an asset with a name containing an underscore _. Here s2cloudless_clp. Using a asset name with no underscore works well (e.g. red)

Expected Output

Correctly processed tile

Environment Information

Titiler from 0.14.0 to 0.18.0 container ghcr.io/developmentseed/titiler:0.18.0 + boto3 and rio-stac

Debug log

"GET /stac/tiles/WebMercatorQuad/9/451/76.png?url=s3%3A%2F%2Fdc-development-catalog%2Fcalls%2Fcall-969%2FcalibratedDatasets%2FS2A_MSIL2A_20230928T042551_N0509_R133_T54XVM_20230928T053752-calibrated%2FS2A_MSIL2A_20230928T042551_N0509_R133_T54XVM_20230928T053752-calibrated.json&rescale=0%2C1&resampling_method=average&resampling=nearest&expression=where((s2cloudless_clp%3E0.5)%2C1%2C0)%3B&asset_as_band=true HTTP/1.1" 500
[2024-04-08 12:26:57 +0000] [50915] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/opt/bitnami/python/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/fastapi/applications.py", line 292, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/bitnami/python/lib/python3.11/site-packages/titiler/core/middleware.py", line 63, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette_cramjam/middleware.py", line 112, in __call__
    await responder(scope, receive, send)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette_cramjam/middleware.py", line 142, in __call__
    await self.app(scope, receive, self.send_with_compression)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/bitnami/python/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/opt/bitnami/python/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/fastapi/routing.py", line 273, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/fastapi/routing.py", line 192, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/titiler/core/factory.py", line 540, in tile
    image = src_dst.tile(
            ^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/io/base.py", line 525, in tile
    return img.apply_expression(expression)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/models.py", line 594, in apply_expression
    res.append(apply_expression(blocks, self.band_names, numpy.array(prod)))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/expression.py", line 81, in apply_expression
    [
  File "/opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/expression.py", line 83, in <listcomp>
    numexpr.evaluate(bloc.strip(), local_dict=dict(zip(bands, data)))
  File "/opt/bitnami/python/lib/python3.11/site-packages/numexpr/necompiler.py", line 975, in evaluate
    raise e
  File "/opt/bitnami/python/lib/python3.11/site-packages/numexpr/necompiler.py", line 877, in validate
    signature = [(name, getType(arg)) for (name, arg) in
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/numexpr/necompiler.py", line 877, in <listcomp>
    signature = [(name, getType(arg)) for (name, arg) in
                        ^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.11/site-packages/numexpr/necompiler.py", line 716, in getType
    raise ValueError('NumExpr 2 does not support Unicode as a dtype.')
ValueError: NumExpr 2 does not support Unicode as a dtype.

Analysis

Problem seems to come from rio-tiler. Checking further that way.

vincentsarago commented 4 months ago

thanks @emmanuelmathot

I believe, this is a rio-tiler issue.

Can you open an issue there? the problem is somewhere in https://github.com/cogeotiff/rio-tiler/blob/main/rio_tiler/expression.py#L12-L37

emmanuelmathot commented 4 months ago

Trying to reproduce the error in rio-tiler, I think the error comes from the data. I make more tests and open an issue if the problem comes from there

emmanuelmathot commented 4 months ago

Issue due to wrong statistics in STAC Item. Fixed in https://github.com/cogeotiff/rio-tiler/pull/695