Metadata extraction fails with "got an unexpected keyword argument 'fail_fast'"

[bgruening]

We only have one test object_store enabled that is not used by default. And all of a sudden all our jobs failed with this error. I assume the S3 was not reachable for some time. Commenting out the S3 config and restart handlers do work for us.

Here is the stack-trace.

galaxy@sn06:~$ cat /data/jwd02f/main/071/343/71343694/metadata/outputs_populated/traceback.txt 
Traceback (most recent call last):
  File "/data/jwd02f/main/071/343/71343694/metadata/set.py", line 5, in <module>
    from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/server/lib/galaxy_ext/metadata/set_metadata.py", line 20, in <module>
    from galaxy.metadata.set_metadata import set_metadata
  File "/opt/galaxy/server/lib/galaxy/metadata/__init__.py", line 9, in <module>
    import galaxy.model
  File "/opt/galaxy/server/lib/galaxy/model/__init__.py", line 143, in <module>
    from galaxy.objectstore import ObjectStore
  File "/opt/galaxy/server/lib/galaxy/objectstore/__init__.py", line 1395, in <module>
    class ConcreteObjectStoreModel(BaseModel):
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 205, in __new__
    complete_model_class(
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 534, in complete_model_class
    schema = cls.__get_pydantic_core_schema__(cls, handler)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/main.py", line 642, in __get_pydantic_core_schema__
    return handler(source)
           ^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 512, in generate_schema
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 784, in _generate_schema_inner
    return self._model_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 591, in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 591, in <dictcomp>
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 947, in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1134, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1890, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1972, in new_handler
    schema = metadata_get_schema(source, get_inner_schema)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_std_types_schema.py", line 327, in __get_pydantic_core_schema__
    constrained_schema = core_schema.list_schema(items_schema, **metadata)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: list_schema() got an unexpected keyword argument 'fail_fast'

[bgruening]

xref: https://sentry.galaxyproject.org/organizations/galaxy/issues/63008/events/ee8b02163a0a4956b3485cd25bc118a5/

[mvdbeek]

What's the link with the sentry issue ? These seem to be different stack traces for different jobs. I also don't see the link with S3 from these issues.

I can't make sense of the traceback in the opening issue. This is the simplest possible pydantic model

class ConcreteObjectStoreModel(BaseModel):
    object_store_id: Optional[str] = None
    private: bool
    name: Optional[str] = None
    description: Optional[str] = None
    quota: QuotaModel
    badges: List[BadgeDict]
    device: Optional[str] = None

My best guess is that there was a dependency problem, maybe incompatible pydantic and pydantic-core packages ?

[mvdbeek]

FWIW fail_fast was added in pydantic 2.8.0 which was released yesterday (https://github.com/pydantic/pydantic/releases/tag/v2.8.0). Is there any chance you accidentally upgraded pydantic but not pydantic core ?

[bgruening]

That was wird. Our venv was updated and it shouldn't. Removing the object_store helped because this was the only code-path the new venv triggered the incompatibility.