search

NOAA-OWP / hydrotools

Suite of tools for retrieving USGS NWIS observations and evaluating National Water Model (NWM) data.
Other
53 stars 12 forks source link

SVI Client get method failing due to Pydantic>2 issue #240

Closed aaraney closed 8 months ago

aaraney commented 8 months ago

Failure in gh action.

Related to #228

Reproduce Issue

Note, ive tested that this failed on all versions of pydantic>2.

pip install hydrotools.svi-client==0.0.1 pydantic==2.4.2
from hydrotools.svi_client import SVIClient

client = SVIClient()
df = client.get(
    location="AL",
    geographic_scale="county",
    year="2018",
    geographic_context="national",
)
Traceback (most recent call last):
  File "/home/hydrotools/fix_svi_client/python/svi_client/test.py", line 1, in <module>
    from hydrotools.svi_client import SVIClient
  File "/home/hydrotools/fix_svi_client/python/svi_client/venv/lib/python3.9/site-packages/hydrotools/svi_client/__init__.py", line 4, in <module>
    from .clients import SVIClient
  File "/home/hydrotools/fix_svi_client/python/svi_client/venv/lib/python3.9/site-packages/hydrotools/svi_client/clients.py", line 6, in <module>
    from . import url_builders
  File "/home/hydrotools/fix_svi_client/python/svi_client/venv/lib/python3.9/site-packages/hydrotools/svi_client/url_builders.py", line 4, in <module>
    from .types import utilities, field_name_map
  File "/home/hydrotools/fix_svi_client/python/svi_client/venv/lib/python3.9/site-packages/hydrotools/svi_client/types/field_name_map.py", line 59, in <module>
    CdcEsri2000CountiesFieldNameMap = FieldNameMap(
  File "/home/hydrotools/fix_svi_client/python/svi_client/venv/lib/python3.9/site-packages/pydantic/main.py", line 164, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 5 validation errors for FieldNameMap
socioeconomic_value
  Field required [type=missing, input_value={'state_name': 'STATE_NAM...TP', 'svi_rank': 'USTP'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/missing
household_comp_and_disability_value
  Field required [type=missing, input_value={'state_name': 'STATE_NAM...TP', 'svi_rank': 'USTP'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/missing
minority_status_and_lang_value
  Field required [type=missing, input_value={'state_name': 'STATE_NAM...TP', 'svi_rank': 'USTP'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/missing
housing_type_and_trans_value
  Field required [type=missing, input_value={'state_name': 'STATE_NAM...TP', 'svi_rank': 'USTP'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/missing
svi_value
  Field required [type=missing, input_value={'state_name': 'STATE_NAM...TP', 'svi_rank': 'USTP'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/missing
aaraney commented 8 months ago

The issue stems from how pydantic<2 and pydantic>2 treat Optional fields. For example:

from pydantic import BaseModel
from typing import Optional

class Foo(BaseModel):
  field: Optional[str]

# valid in pydantic<2
inst = Foo()
assert inst.field == None

# pydantic ValidationError in pydantic>2
# b.c. Optional fields are indicate either Some(value) or None, but either must be present
inst = Foo()

# Must either explicitly set field=None, or change Foo so that field has a default value of None

You can read more about the differences between pydantic versions here.

Below are the two places that are affected:

https://github.com/NOAA-OWP/hydrotools/blob/6c6c03367f3c4ccb19122f8efd1f487667df97d3/python/svi_client/src/hydrotools/svi_client/types/field_name_map.py#L59

https://github.com/NOAA-OWP/hydrotools/blob/6c6c03367f3c4ccb19122f8efd1f487667df97d3/python/svi_client/src/hydrotools/svi_client/types/field_name_map.py#L135