[Bug]: llama-index-core 0.11.2 has incorrent dependency set on pydantic version

[AldebaranX]

Bug Description

llama-index 0.11.2 installed through conda (OS is Windows 10): conda install conda-forge::llama-index

In llama-index-core 0.11.2, the dependency on pydantic is set as: "pydantic >=2.0.0,<3.0.0", in

\llama-index-core-0.11.2-pyhd8ed1ab_0\info\index.json

However, when import the modules for reading documents: from llama_index.core import SimpleDirectoryReader from llama_index.core import Document

the error raise as: ImportError: cannot import name 'Secret' from 'pydantic' (.....\site-packages\pydantic\__init__.py) if using pydantic version 2.0 to 2.6.

Since 'Secret' class only exists for pydantic version >2.7, this is inconsistent with the dependency defined in index.json.

Update:

When using pydantic>=2.7, SchemaError raises when importing: from llama_index.core import SimpleDirectoryReader

SchemaError: Invalid Schema:
model.schema.model-fields.fields.node_type.schema.default.schema.nullable.schema
  Input tag 'enum' found using 'type' does not match any of the expected tags: 'any', 'none', 'bool', 'int', 'float', 'decimal', 'str', 'bytes', 'date', 'time', 'datetime', 'timedelta', 'literal', 'is-instance', 'is-subclass', 'callable', 'list', 'tuple-positional', 'tuple-variable', 'set', 'frozenset', 'generator', 'dict', 'function-after', 'function-before', 'function-wrap', 'function-plain', 'default', 'nullable', 'union', 'tagged-union', 'chain', 'lax-or-strict', 'json-or-python', 'typed-dict', 'model-fields', 'model', 'dataclass-args', 'dataclass', 'arguments', 'call', 'custom-error', 'json', 'url', 'multi-host-url', 'definitions', 'definition-ref', 'uuid' [type=union_tag_invalid, input_value={'type': 'enum', 'cls': <...t 0x00000194F3607CE0>]}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/union_tag_invalid

Version

0.11.2

Steps to Reproduce

conda install conda-forge::llama-index=0.11.2

from llama_index.core import SimpleDirectoryReader or from llama_index.core import Document

For pydantic version 2.0 to 2.6:

ImportError: cannot import name 'Secret' from 'pydantic' (.....\site-packages\pydantic\__init__.py)

For pydantic version 2.7/2.8:

SchemaError: Invalid Schema:
model.schema.model-fields.fields.node_type.schema.default.schema.nullable.schema
  Input tag 'enum' found using 'type' does not match any of the expected tags: 'any', 'none', 'bool', 'int', 'float', 'decimal', 'str', 'bytes', 'date', 'time', 'datetime', 'timedelta', 'literal', 'is-instance', 'is-subclass', 'callable', 'list', 'tuple-positional', 'tuple-variable', 'set', 'frozenset', 'generator', 'dict', 'function-after', 'function-before', 'function-wrap', 'function-plain', 'default', 'nullable', 'union', 'tagged-union', 'chain', 'lax-or-strict', 'json-or-python', 'typed-dict', 'model-fields', 'model', 'dataclass-args', 'dataclass', 'arguments', 'call', 'custom-error', 'json', 'url', 'multi-host-url', 'definitions', 'definition-ref', 'uuid' [type=union_tag_invalid, input_value={'type': 'enum', 'cls': <...t 0x00000194F3607CE0>]}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/union_tag_invalid

Relevant Logs/Tracbacks

No response

[dosubot[bot]]

Hello, @AldebaranX! I'm here to help you with any bugs, questions, or contributions you have regarding llama-index. Let's get this sorted out together!

There is indeed an incorrect dependency set on the pydantic version in llama-index-core 0.11.2 that causes an ImportError when using pydantic versions 2.0 to 2.6. The pyproject.toml file specifies the dependency as pydantic = ">=2.0.0,<3.0.0", but the code attempts to import pydantic.v1 and falls back to pydantic if the former is not available. This indicates that the code is not compatible with pydantic versions 2.0 to 2.6, which do not have a pydantic.v1 module ^[1][2].

A similar issue was discussed and resolved in a closed issue: [Feature Request]: Support Pydantic v2. The resolution involved a pull request that added compatibility for both Pydantic v1 and v2 by attempting to import pydantic.v1 and falling back to the default pydantic import if the former is not available ^[3].

To resolve this issue, you may need to update to a version of llama-index-core that includes this fix or manually adjust the imports in your local environment.

To continue talking to Dosu, mention @dosu.

[logan-markewich]

I'm using pydantic 2.8.2 and do not have this SchemaError: Invalid Schema: issue?

[logan-markewich]

pydantic 2.7 works fine as well

[logan-markewich]

In fact, a fresh install works fine (albeit without conda -- I don't use conda)