search

stac-utils / stac-fastapi

STAC API implementation with FastAPI.
https://stac-utils.github.io/stac-fastapi/
MIT License
245 stars 99 forks source link

ciso8601 as an optional dependency #447

Closed keul closed 2 years ago

keul commented 2 years ago

We have currently an issue with stac_fastapi 2.4 in Python 3.10 conda environment due to the recent introduction on ciso8601 as a dependency: see https://github.com/stac-utils/stac-fastapi/pull/368

What about having this dependency optional? ciso8601 is a C dependency, so a bit trickier to be installed and it's only an optimisation.

vincentsarago commented 2 years ago

ref https://github.com/stac-utils/stac-fastapi/issues/204#issuecomment-1205276420

geospatial-jeff commented 2 years ago

What type of issue are you seeing? Including traceback would be helpful.

geospatial-jeff commented 2 years ago

I'm unable to replicate this on my machine (using miniconda):

conda create -n py310 python=3.10
conda activate py310
pip install stac-fastapi.sqlalchemy

Output:

Collecting stac-fastapi.sqlalchemy
  Downloading stac_fastapi.sqlalchemy-2.4.0-py3-none-any.whl (34 kB)
Collecting stac-fastapi.extensions
  Downloading stac_fastapi.extensions-2.4.0-py3-none-any.whl (16 kB)
Collecting attrs
  Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting geoalchemy2<0.8.0
  Downloading GeoAlchemy2-0.7.0-py2.py3-none-any.whl (21 kB)
Collecting stac-fastapi.api
  Downloading stac_fastapi.api-2.4.0-py3-none-any.whl (12 kB)
Collecting stac-pydantic>=2.0.3
  Using cached stac_pydantic-2.0.3-py3-none-any.whl (17 kB)
Collecting sqlalchemy==1.3.23
  Downloading SQLAlchemy-1.3.23.tar.gz (6.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 3.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting alembic
  Downloading alembic-1.8.1-py3-none-any.whl (209 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 kB 3.2 MB/s eta 0:00:00
Collecting shapely
  Using cached Shapely-1.8.2-cp310-cp310-win_amd64.whl (1.3 MB)
Collecting pydantic[dotenv]
  Using cached pydantic-1.9.1-cp310-cp310-win_amd64.whl (2.0 MB)
Collecting psycopg2-binary
  Downloading psycopg2_binary-2.9.3-cp310-cp310-win_amd64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 3.4 MB/s eta 0:00:00
Collecting stac-fastapi.types
  Downloading stac_fastapi.types-2.4.0-py3-none-any.whl (13 kB)
Collecting sqlakeyset
  Downloading sqlakeyset-1.0.1659142803-py3-none-any.whl (20 kB)
Collecting fastapi-utils
  Downloading fastapi_utils-0.2.1-py3-none-any.whl (16 kB)
Collecting click
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting geojson-pydantic
  Using cached geojson_pydantic-0.4.3-py3-none-any.whl (7.9 kB)
Collecting Mako
  Downloading Mako-1.2.1-py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.6/78.6 kB 4.3 MB/s eta 0:00:00
Collecting fastapi
  Downloading fastapi-0.79.0-py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.6/54.6 kB ? eta 0:00:00
Collecting typing-extensions>=3.7.4.3
  Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Collecting python-dotenv>=0.10.4
  Using cached python_dotenv-0.20.0-py3-none-any.whl (17 kB)
Collecting packaging>=20.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting python-dateutil
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting brotli-asgi
  Using cached brotli_asgi-1.1.0-py3-none-any.whl (4.3 kB)
Collecting ciso8601~=2.2.0
  Using cached ciso8601-2.2.0.tar.gz (18 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pystac==1.*
  Downloading pystac-1.6.1-py3-none-any.whl (146 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 146.0/146.0 kB 2.9 MB/s eta 0:00:00
Collecting starlette==0.19.1
  Downloading starlette-0.19.1-py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.3/63.3 kB 3.3 MB/s eta 0:00:00
Collecting anyio<5,>=3.4.0
  Using cached anyio-3.6.1-py3-none-any.whl (80 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting brotli>=1.0.7
  Downloading Brotli-1.0.9-cp310-cp310-win_amd64.whl (383 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 383.3/383.3 kB 3.4 MB/s eta 0:00:00
Collecting colorama
  Using cached colorama-0.4.5-py2.py3-none-any.whl (16 kB)
Collecting MarkupSafe>=0.9.2
  Using cached MarkupSafe-2.1.1-cp310-cp310-win_amd64.whl (17 kB)
Collecting idna>=2.8
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting sniffio>=1.1
  Using cached sniffio-1.2.0-py3-none-any.whl (10 kB)
Building wheels for collected packages: sqlalchemy, ciso8601
  Building wheel for sqlalchemy (setup.py) ... done
  Created wheel for sqlalchemy: filename=SQLAlchemy-1.3.23-cp310-cp310-win_amd64.whl size=1221724 sha256=6229034aba102b3725b9bb3cbf4cfd65e2aa74912a54f75661d6ac121cb18232
  Stored in directory: c:\users\jeffa\appdata\local\pip\cache\wheels\6b\bd\5f\b63dc9976fc54a34ac2019fdbef4eb6a15aef4958298ce4b55
  Building wheel for ciso8601 (pyproject.toml) ... done
  Created wheel for ciso8601: filename=ciso8601-2.2.0-cp310-cp310-win_amd64.whl size=14986 sha256=77476234792bda11e7514618bec4ca23824f131f41824fa6771489fcbbff5fba
  Stored in directory: c:\users\jeffa\appdata\local\pip\cache\wheels\02\1f\3b\a3fc12fc79c99e80e030beb98f07c92cbf1e43e00a7abccfcb
Successfully built sqlalchemy ciso8601
Installing collected packages: ciso8601, brotli, typing-extensions, sqlalchemy, sniffio, six, shapely, python-dotenv, pyparsing, psycopg2-binary, MarkupSafe, idna, colorama, attrs, python-dateutil, pydantic, packaging, Mako, geoalchemy2, click, anyio, starlette, sqlakeyset, pystac, geojson-pydantic, alembic, stac-pydantic, fastapi, brotli-asgi, stac-fastapi.types, fastapi-utils, stac-fastapi.api, stac-fastapi.extensions, stac-fastapi.sqlalchemy
Successfully installed Mako-1.2.1 MarkupSafe-2.1.1 alembic-1.8.1 anyio-3.6.1 attrs-22.1.0 brotli-1.0.9 brotli-asgi-1.1.0 ciso8601-2.2.0 click-8.1.3 colorama-0.4.5 fastapi-0.79.0 fastapi-utils-0.2.1 geoalchemy2-0.7.0 geojson-pydantic-0.4.3 idna-3.3 packaging-21.3 psycopg2-binary-2.9.3 pydantic-1.9.1 pyparsing-3.0.9 pystac-1.6.1 python-dateutil-2.8.2 python-dotenv-0.20.0 shapely-1.8.2 six-1.16.0 sniffio-1.2.0 sqlakeyset-1.0.1659142803 sqlalchemy-1.3.23 stac-fastapi.api-2.4.0 stac-fastapi.extensions-2.4.0 stac-fastapi.sqlalchemy-2.4.0 stac-fastapi.types-2.4.0 stac-pydantic-2.0.3 starlette-0.19.1 typing-extensions-4.3.0
keul commented 2 years ago

I missed another information: everything is inside a docker image based on continuumio/miniconda3

@geospatial-jeff 

#13 98.81 Installing pip dependencies: ...working... Pip subprocess error:
#13 108.2   error: subprocess-exited-with-error
#13 108.2
#13 108.2   × Building wheel for ciso8601 (pyproject.toml) did not run successfully.
#13 108.2   │ exit code: 1
#13 108.2   ╰─> [13 lines of output]
#13 108.2       running bdist_wheel
#13 108.2       running build
#13 108.2       running build_py
#13 108.2       creating build
#13 108.2       creating build/lib.linux-x86_64-cpython-310
#13 108.2       creating build/lib.linux-x86_64-cpython-310/ciso8601
#13 108.2       copying ciso8601/__init__.pyi -> build/lib.linux-x86_64-cpython-310/ciso8601
#13 108.2       copying ciso8601/py.typed -> build/lib.linux-x86_64-cpython-310/ciso8601
#13 108.2       running build_ext
#13 108.2       building 'ciso8601' extension
#13 108.2       creating build/temp.linux-x86_64-cpython-310
#13 108.2       gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -DCISO8601_VERSION=2.2.0 -DCISO8601_CACHING_ENABLED=1 -I/opt/conda/include/python3.10 -c module.c -o build/temp.linux-x86_64-cpython-310/module.o
#13 108.2       error: command 'gcc' failed: No such file or directory
#13 108.2       [end of output]
#13 108.2
#13 108.2   note: This error originates from a subprocess, and is likely not a problem with pip.
#13 108.2   ERROR: Failed building wheel for ciso8601
#13 108.2 ERROR: Could not build wheels for ciso8601, which is required to install pyproject.toml-based projects
#13 108.2
vincentsarago commented 2 years ago

error: command 'gcc' failed: No such file or directory

install gcc on your image 🤷‍♂️

geospatial-jeff commented 2 years ago

It looks like that image doesn't have gcc so its unable to build the wheel, can you try installing build-essential?

sudo apt-get install build-essential

https://github.com/stac-utils/stac-fastapi/blob/master/Dockerfile#L5-L9

duckontheweb commented 2 years ago

I still get an error after installing build-essential. This seems like a known issue with building ciso8601 in some environments (see closeio/ciso8601#123(comment).

I'll see if I can figure out a workaround for this.

geospatial-jeff commented 2 years ago

Looks like python 3.10 support is added in ciso8601==2.2.1 which was PRd but never merged https://github.com/closeio/ciso8601/pull/115. This release also adds wheels that @kylebarron made to make it easier to install (https://github.com/closeio/ciso8601/pull/109).

geospatial-jeff commented 2 years ago

So at this point I'd like to wait to see if the new version of ciso8601 is released. We have the option of switching to pyiso8601 but its not a complete RFC3339 parser (while ciso8601 is).

https://github.com/radiantearth/stac-api-spec/blob/main/implementation.md#datetime-parameter-handling

duckontheweb commented 2 years ago

I still get an error after installing build-essential. This seems like a known issue with building ciso8601 in some environments (see closeio/ciso8601#123(comment).

I'll see if I can figure out a workaround for this.

This particular error seems to be specific to Python 3.9 in the continuumio/miniconda3:latest image and it seems like it stems from the fact that that version of Python sets the -n1 compile flag when building C extensions, which gcc doesn't like:

(base) root@635c876df361:/opt/conda# python -c "from sysconfig import get_config_var; print(get_config_var(\"CFLAGS\"));"
-Wno-unused-result -Wsign-compare -DNDEBUG  -O2 -Wall  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include

Creating a Python 3.8 or 3.10 environment allows me to install the library without a problem:

(test) root@635c876df361:/opt/conda# conda create --name test-310 python=3.10
(test) root@635c876df361:/opt/conda# conda activate test-310
(test-310) root@635c876df361:/opt/conda# python -c "from sysconfig import get_config_var; print(get_config_var(\"CFLAGS\"));"
-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC   -O3  -isystem /opt/conda/envs/test-310/include -fPIC   -O3  -isystem /opt/conda/envs/test-310/include
(test-310) root@635c876df361:/opt/conda# pip install ciso8601
Collecting ciso8601
  Using cached ciso8601-2.2.0.tar.gz (18 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: ciso8601
  Building wheel for ciso8601 (pyproject.toml) ... done
  Created wheel for ciso8601: filename=ciso8601-2.2.0-cp310-cp310-linux_aarch64.whl size=29928 sha256=040a6185a7d8e1857e9edd3bef856225a647ee39f8c604ba75bca2867229837e
  Stored in directory: /root/.cache/pip/wheels/02/1f/3b/a3fc12fc79c99e80e030beb98f07c92cbf1e43e00a7abccfcb
Successfully built ciso8601
Installing collected packages: ciso8601
Successfully installed ciso8601-2.2.0
keul commented 2 years ago

There's also ciso8601 on conda. I'll try that before adding gcc

geospatial-jeff commented 2 years ago

@keul The ciso8601 dependency is removed in 2.4.1