search

developmentseed / titiler

Build your own Raster dynamic map tile services
https://developmentseed.org/titiler/
MIT License
755 stars 156 forks source link

Issue with pydantic-settings, Extra inputs are not permitted #799

Closed imanshafiei540 closed 5 months ago

imanshafiei540 commented 5 months ago

Problem description

I encountered an error related to the rio_tiler_max_threads environment variable when using TiTiler with the following environment configuration:

.env file:

RIO_TILER_MAX_THREADS=50

pyproject.toml file:

dependencies = [
    ...
    "pydantic-settings~=2.0",
]

pip freeze output:

pydantic-settings==2.2.1

The error occurred with the current implementation:

model_config = SettingsConfigDict(env_prefix="TITILER_API_", env_file=".env")

The error message is:

rio_tiler_max_threads
  Extra inputs are not permitted [type=extra_forbidden, input_value='50', input_type=str]
    For further information visit https://errors.pydantic.dev/2.4/v/extra_forbidden

Expected Output

No errors when loading the environment variables.

Proposed Fix

I fixed the issue by allowing extra settings in the SettingsConfigDict:

model_config = SettingsConfigDict(env_prefix="TITILER_API_", env_file=".env", extra="allow")

Environment Information

vincentsarago commented 5 months ago

@imanshafiei540 this is really weird because RIO_TILER_MAX_THREADS doesn't conflict with TITILER_API_ prefix 🤷

imanshafiei540 commented 5 months ago

Okay, I tried to dig more into the issue and ended up here in this line from Pydantic==2.6.4:

self.__pydantic_validator__.validate_python(data, self_instance=self)

Somehow, the api_settings = ApiSettings() coming from src/titiler/application/titiler/application/main.py cannot be validated by __pydantic_validator__ and I'm not sure what's the reason.

Could be some issues between Pydantic and Pydantic-Settings.

I hope someone else can also verify the issue; I'm constantly thinking maybe I'm doing something wrong on my end.

Anyway, thank you for the quick response, and I'll try to figure out what's wrong; it's probably time to raise an issue on the Pydantic-Setiings repo.

vincentsarago commented 5 months ago

can you share the full log? and/or share a reproducible example 🙏

imanshafiei540 commented 5 months ago

Sure, I'll prepare something ASAP, just as a note, everything is working with pydantic-settings==2.1.0 2.2.0 or above will break the TiTiler.

imanshafiei540 commented 5 months ago

Okay, here is how to reproduce (it's actually pretty simple):

  1. Fresh environment (I created one using venv)
  2. Installing the dependencies (I'm using zsh) 
    pip install pre-commit -e src/titiler/core\[test\] -e src/titiler/extensions\[test,cogeo,stac\] -e src/titiler/mosaic\[test\] -e src/titiler/application\[test\]
  3. Verify that everything is working as expected: 
    python -m pytest src/titiler/application --cov=titiler.application --cov-report=xml --cov-append --cov-report=term-missing
  4. Create a new .env file in the root of the project with the content below (it's not specified to this env variable, you can use other env variables if you want to): 
    RIO_TILER_MAX_THREADS=50
  5. Run the tests again: 
    python -m pytest src/titiler/application --cov=titiler.application --cov-report=xml --cov-append --cov-report=term-missing
  6. The tests will fail and you should see some errors like this one: pydantic_core._pydantic_core.ValidationError: 1 validation error for ApiSettings

More context (pip freeze):

Python=3.11.8

affine==2.4.0
annotated-types==0.6.0
anyio==4.3.0
attrs==23.2.0
boto3==1.34.66
botocore==1.34.66
brotlipy==0.7.0
cachetools==5.3.3
certifi==2024.2.2
cffi==1.16.0
cfgv==3.4.0
click==8.1.7
click-plugins==1.1.1
cligj==0.7.2
cogeo-mosaic==7.1.0
color-operations==0.1.3
coverage==7.4.4
cramjam==2.6.2
distlib==0.3.8
fastapi==0.110.0
filelock==3.13.1
geojson-pydantic==1.0.2
h11==0.14.0
httpcore==1.0.4
httpx==0.27.0
identify==2.5.35
idna==3.6
iniconfig==2.0.0
Jinja2==3.1.3
jmespath==1.0.1
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
MarkupSafe==2.1.5
morecantile==5.3.0
nodeenv==1.8.0
numexpr==2.9.0
numpy==1.26.4
packaging==24.0
platformdirs==4.2.0
pluggy==1.4.0
pre-commit==3.6.2
pycparser==2.21
pydantic==2.6.4
pydantic-settings==2.2.1
pydantic_core==2.16.3
pyparsing==3.1.2
pyproj==3.6.1
pystac==1.9.0
pytest==8.1.1
pytest-asyncio==0.23.6
pytest-cov==4.1.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
PyYAML==6.0.1
rasterio==1.3.9
referencing==0.34.0
rio-cogeo==5.2.0
rio-stac==0.8.1
rio-tiler==6.4.1
rpds-py==0.18.0
s3transfer==0.10.1
shapely==2.0.3
simplejson==3.19.2
six==1.16.0
sniffio==1.3.1
snuggs==1.4.7
starlette==0.36.3
starlette-cramjam==0.3.2
supermorecado==0.1.2
-e git+ssh://git@github.com/developmentseed/titiler.git@ffd67af34c2807a6e1447817f943446a58441ed8#egg=titiler.application&subdirectory=src/titiler/application
-e git+ssh://git@github.com/developmentseed/titiler.git@ffd67af34c2807a6e1447817f943446a58441ed8#egg=titiler.core&subdirectory=src/titiler/core
-e git+ssh://git@github.com/developmentseed/titiler.git@ffd67af34c2807a6e1447817f943446a58441ed8#egg=titiler.extensions&subdirectory=src/titiler/extensions
-e git+ssh://git@github.com/developmentseed/titiler.git@ffd67af34c2807a6e1447817f943446a58441ed8#egg=titiler.mosaic&subdirectory=src/titiler/mosaic
typing_extensions==4.10.0
urllib3==2.2.1
virtualenv==20.25.1

Using uvicorn to reproduce the issue

After creating the .env file:

  1. Install uvicorn by pip install uvicorn
  2. Run the application: uvicorn titiler.application.main:app --workers 3 --host localhost --port 8002

Now you should be able to see the errors, by removing the .env file, everything is working properly.

Let me know if you need more information. Thank you.

vincentsarago commented 5 months ago

ok I've been able to reproduce this 

uvicorn titiler.application.main:app --port 8000 --reload
INFO:     Will watch for changes in these directories: ['/Users/vincentsarago/Dev/DevSeed/titiler']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [79319] using WatchFiles
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/vincentsarago/Dev/venv/py39/lib/python3.9/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/Users/vincentsarago/Dev/venv/py39/lib/python3.9/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/Users/vincentsarago/Dev/venv/py39/lib/python3.9/site-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/Users/vincentsarago/Dev/venv/py39/lib/python3.9/site-packages/uvicorn/config.py", line 479, in load
    self.loaded_app = import_from_string(self.app)
  File "/Users/vincentsarago/Dev/venv/py39/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/vincentsarago/Dev/DevSeed/titiler/src/titiler/application/titiler/application/main.py", line 51, in <module>
    api_settings = ApiSettings()
  File "/Users/vincentsarago/Dev/venv/py39/lib/python3.9/site-packages/pydantic_settings/main.py", line 84, in __init__
    super().__init__(
  File "/Users/vincentsarago/Dev/venv/py39/lib/python3.9/site-packages/pydantic/main.py", line 171, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for ApiSettings
rio_tiler_max_threads
  Extra inputs are not permitted [type=extra_forbidden, input_value='50', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden

This is really weird and I wonder if this is a pydantic-setting bug instead of a titiler one.

pydantic ticket: https://github.com/pydantic/pydantic-settings/issues/245

I would be happy to review any PR to add extra='ignore' 👍

imanshafiei540 commented 5 months ago

Thanks for confirming the issue. I'll try to create a PR as soon as possible.