TypeError: typing.ClassVar[typing.List[typing.Type[great_expectations.experimental.datasources.interfaces.DataAsset]]] is not valid as type argument

[RamanDamayeu]

Type checking error while loading great_expectations.

python -c "import great_expectations;"

Gives:

  File "/home/user/.local/share/virtualenvs/project_airflow-ZgRTdk43/lib/python3.10/site-packages/pydantic/typing.py", line 287, in resolve_annotations
    value = _eval_type(value, base_globals, None)
  File "/usr/lib/python3.10/typing.py", line 327, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "/usr/lib/python3.10/typing.py", line 693, in _evaluate
    type_ = _type_check(
  File "/usr/lib/python3.10/typing.py", line 167, in _type_check
    raise TypeError(f"{arg} is not valid as type argument")
TypeError: typing.ClassVar[typing.List[typing.Type[great_expectations.experimental.datasources.interfaces.DataAsset]]] is not valid as type argument

Expected behavior Type checking passes successfully.

Environment:

• Operating System: Ubuntu 22.04
• Great Expectations Version: 0.15.44
• typing_extensions Version: 4.4.0
• pydantic Version: 1.8.2
• Python Version: 3.10.6

[austiezr]

Hey @RamanDamayeu ! Thanks for raising this; we'll investigate and be in touch. 🙇

[austiezr]

Hey @RamanDamayeu! It looks like this is specifically an issue between pydantic & Python 3.10; we support pydantic>=1.0,<2.0 & Python 3.10, but support for Python 3.10 didn't come to pydantic until v1.9.0. If you're able to upgrade pydantic to at least v1.9.0, or come down to Python 3.9, this issue should resolve for you.

Closing this out for now, but feel free to re-open if this doesn't resolve for you. 🙇

[RamanDamayeu]

Hi @austiezr, thanks for such a quick response. I've tried 3.9 the issue seems to be in place:

airflow@2503f136fe8b:/opt/airflow$ python --version
Python 3.9.16
airflow@2503f136fe8b:/opt/airflow$ python -m pip freeze
adal==1.2.7
aiofiles==22.1.0
aiohttp==3.8.3
aiosignal==1.3.1
aiosqlite==0.18.0
airflow-provider-great-expectations==0.2.4
alembic==1.9.2
altair==4.2.0
amqp==5.1.1
anyio==3.6.2
apache-airflow==2.5.1
apache-airflow-providers-amazon==7.1.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==5.1.1
apache-airflow-providers-common-sql==1.3.3
apache-airflow-providers-databricks==4.0.0
apache-airflow-providers-docker==3.4.0
apache-airflow-providers-elasticsearch==4.3.3
apache-airflow-providers-ftp==3.3.0
apache-airflow-providers-google==8.8.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.2.0
apache-airflow-providers-http==4.1.1
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-microsoft-azure==5.1.0
apache-airflow-providers-mysql==4.0.0
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-sendgrid==3.1.0
apache-airflow-providers-sftp==4.2.1
apache-airflow-providers-slack==7.2.0
apache-airflow-providers-snowflake==4.0.2
apache-airflow-providers-sqlite==3.3.1
apache-airflow-providers-ssh==3.4.0
apispec==3.3.2
argcomplete==2.0.0
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
asgiref==3.6.0
asn1crypto==1.5.1
asttokens==2.2.1
async-generator==1.10
async-lru==1.0.3
async-timeout==4.0.2
attrs==22.2.0
Authlib==1.2.0
azure-batch==13.0.0
azure-common==1.1.28
azure-core==1.26.2
azure-cosmos==4.3.0
azure-datalake-store==0.0.52
azure-identity==1.12.0
azure-keyvault-secrets==4.6.0
azure-kusto-data==0.0.45
azure-mgmt-containerinstance==1.5.0
azure-mgmt-core==1.3.2
azure-mgmt-datafactory==1.1.0
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==22.0.0
azure-nspkg==3.0.2
azure-servicebus==7.8.2
azure-storage-blob==12.14.1
azure-storage-common==2.1.0
azure-storage-file==2.1.0
azure-storage-file-datalake==12.9.1
azure-synapse-spark==0.7.0
Babel==2.11.0
backcall==0.2.0
backoff==1.10.0
bcrypt==4.0.1
beautifulsoup4==4.11.2
billiard==3.6.4.0
bleach==6.0.0
blinker==1.5
boto3==1.26.65
botocore==1.29.65
cachelib==0.9.0
cachetools==4.2.2
cattrs==22.2.0
celery==5.2.7
certifi==2022.12.7
cffi==1.15.1
chardet==4.0.0
charset-normalizer==2.0.12
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
clickclick==20.10.2
cloudpickle==2.2.0
colorama==0.4.6
colorlog==4.8.0
comm==0.1.2
commonmark==0.9.1
ConfigUpdater==3.1.1
connexion==2.14.2
cron-descriptor==1.2.35
croniter==1.3.8
cryptography==38.0.4
dask==2023.1.0
databricks-sql-connector==2.3.1.dev1
db-dtypes==1.0.5
debugpy==1.6.6
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.13
dill==0.3.6
distlib==0.3.6
distributed==2023.1.0
dnspython==2.3.0
docker==6.0.1
docutils==0.19
elasticsearch==7.13.4
elasticsearch-dbapi==0.2.9
elasticsearch-dsl==7.4.0
email-validator==1.3.1
entrypoints==0.4
eventlet==0.33.3
exceptiongroup==1.1.0
executing==1.2.0
fastjsonschema==2.16.2
filelock==3.9.0
Flask==2.2.2
Flask-AppBuilder==4.1.4
Flask-Babel==2.0.0
Flask-Caching==2.0.2
Flask-JWT-Extended==4.4.4
Flask-Login==0.6.2
Flask-Session==0.4.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==1.1.1
flower==1.2.0
fqdn==1.5.1
frozenlist==1.3.3
fsspec==2022.11.0
funcy==1.18
future==0.18.3
gcloud-aio-auth==4.1.5
gcloud-aio-bigquery==6.2.0
gcloud-aio-storage==8.0.0
gevent==22.10.2
google-ads==18.0.0
google-api-core==2.8.2
google-api-python-client==1.12.11
google-auth==2.16.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.8.0
google-cloud-aiplatform==1.16.1
google-cloud-appengine-logging==1.1.3
google-cloud-audit-log==0.2.4
google-cloud-automl==2.8.0
google-cloud-bigquery==2.34.4
google-cloud-bigquery-datatransfer==3.7.0
google-cloud-bigquery-storage==2.14.1
google-cloud-bigtable==1.7.3
google-cloud-build==3.9.0
google-cloud-compute==0.7.0
google-cloud-container==2.11.1
google-cloud-core==2.3.2
google-cloud-datacatalog==3.9.0
google-cloud-dataform==0.2.0
google-cloud-dataplex==1.1.0
google-cloud-dataproc==5.0.0
google-cloud-dataproc-metastore==1.6.0
google-cloud-dlp==1.0.2
google-cloud-kms==2.12.0
google-cloud-language==1.3.2
google-cloud-logging==3.2.1
google-cloud-memcache==1.4.1
google-cloud-monitoring==2.11.0
google-cloud-orchestration-airflow==1.4.1
google-cloud-os-login==2.7.1
google-cloud-pubsub==2.13.5
google-cloud-redis==2.9.0
google-cloud-resource-manager==1.6.0
google-cloud-secret-manager==1.0.2
google-cloud-spanner==1.19.3
google-cloud-speech==1.3.4
google-cloud-storage==2.7.0
google-cloud-tasks==2.10.1
google-cloud-texttospeech==1.0.3
google-cloud-translate==1.7.2
google-cloud-videointelligence==1.16.3
google-cloud-vision==1.0.2
google-cloud-workflows==1.7.1
google-crc32c==1.5.0
google-resumable-media==2.4.0
googleapis-common-protos==1.56.4
graphviz==0.20.1
great-expectations==0.15.47
greenlet==2.0.2
grpc-google-iam-v1==0.12.4
grpcio==1.51.1
grpcio-gcp==0.2.2
grpcio-status==1.48.2
gunicorn==20.1.0
h11==0.14.0
HeapDict==1.0.1
httpcore==0.16.3
httplib2==0.20.4
httpx==0.23.3
humanize==4.4.0
hvac==1.0.2
idna==3.4
importlib-metadata==6.0.0
inflection==0.5.1
ipykernel==6.21.1
ipython==8.9.0
ipywidgets==8.0.4
isodate==0.6.1
isoduration==20.11.0
itsdangerous==2.1.2
jedi==0.18.2
Jinja2==3.1.2
jmespath==1.0.1
json-merge-patch==0.2
json5==0.9.11
jsonpatch==1.32
jsonpath-ng==1.5.3
jsonpointer==2.3
jsonschema==4.17.3
jupyter-client==8.0.2
jupyter-core==5.2.0
jupyter-events==0.5.0
jupyter-lsp==1.5.1
jupyter-server==2.2.1
jupyter-server-fileid==0.6.0
jupyter-server-terminals==0.4.4
jupyter-server-ydoc==0.6.1
jupyter-ydoc==0.3.1
jupyterlab==4.0.0a32
jupyterlab-pygments==0.2.2
jupyterlab-server==2.16.3
jupyterlab-widgets==3.0.5
kombu==5.2.4
kubernetes==23.6.0
lazy-object-proxy==1.9.0
ldap3==2.9.1
linkify-it-py==2.0.0
locket==1.0.0
lockfile==0.12.2
loguru==0.6.0
looker-sdk==22.20.0
lxml==4.9.2
lz4==4.3.2
makefun==1.15.0
Mako==1.2.4
Markdown==3.4.1
markdown-it-py==2.1.0
MarkupSafe==2.1.2
marshmallow==3.19.0
marshmallow-enum==1.5.1
marshmallow-oneofschema==3.0.1
marshmallow-sqlalchemy==0.26.1
matplotlib-inline==0.1.6
mdit-py-plugins==0.3.3
mdurl==0.1.2
mistune==2.0.5
msal==1.20.0
msal-extensions==1.0.0
msgpack==1.0.4
msrest==0.7.1
msrestazure==0.6.4
multidict==6.0.4
mypy-boto3-appflow==1.26.53
mypy-boto3-rds==1.26.47
mypy-boto3-redshift-data==1.26.30
mysql-connector-python==8.0.32
mysqlclient==2.1.1
nbclient==0.7.2
nbconvert==7.2.9
nbformat==5.7.3
nest-asyncio==1.5.6
notebook==7.0.0a11
notebook-shim==0.2.2
numpy==1.23.4
oauthlib==3.2.2
odd-great-expectations==0.1.7
odd-models==2.0.10
oddrn-generator==0.1.62
oscrypto==1.3.0
outcome==1.2.0
packaging==23.0
pandas==1.5.3
pandas-gbq==0.17.9
pandocfilters==1.5.0
paramiko==3.0.0
parso==0.8.3
partd==1.3.0
pathspec==0.9.0
pendulum==3.0.0a1
pexpect==4.8.0
pickleshare==0.7.5
pipenv==2023.2.4
platformdirs==3.0.0
pluggy==1.0.0
ply==3.11
portalocker==2.6.0
prison==0.2.1
prometheus-client==0.16.0
prompt-toolkit==3.0.36
proto-plus==1.19.6
protobuf==3.20.0
psutil==5.9.4
psycopg2-binary==2.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pycryptodomex==3.17
pydantic==1.8.2
pydata-google-auth==1.5.0
Pygments==2.14.0
pyhcl==0.4.4
PyJWT==2.6.0
PyNaCl==1.5.0
pyodbc==4.0.35
pyOpenSSL==22.1.0
pyparsing==3.0.9
pyrsistent==0.19.3
PySocks==1.7.1
python-daemon==2.3.2
python-dateutil==2.8.2
python-dotenv==0.21.0
python-http-client==3.3.7
python-json-logger==2.0.4
python-ldap==3.4.3
python-nvd3==0.15.0
python-slugify==8.0.0
pytz==2022.7.1
pytz-deprecation-shim==0.1.0.post0
pytzdata==2020.1
PyYAML==6.0
pyzmq==25.0.0
redis==3.5.3
redshift-connector==2.0.910
requests==2.27.1
requests-oauthlib==1.3.1
requests-toolbelt==0.10.1
rfc3339-validator==0.1.4
rfc3986==1.5.0
rfc3986-validator==0.1.1
rich==13.3.1
rsa==4.9
ruamel.yaml==0.17.17
ruamel.yaml.clib==0.2.7
s3transfer==0.6.0
scipy==1.10.0
scramp==1.4.4
selenium==4.8.0
Send2Trash==1.8.1b0
sendgrid==6.9.7
setproctitle==1.3.2
six==1.16.0
slack-sdk==3.19.5
sniffio==1.3.0
snowflake-connector-python==2.9.0
snowflake-sqlalchemy==1.4.4
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
sql-metadata==2.6.0
SQLAlchemy==1.4.46
sqlalchemy-bigquery==1.5.0
SQLAlchemy-JSONField==1.0.1.post0
sqlalchemy-redshift==0.8.12
SQLAlchemy-Utils==0.39.0
sqlparse==0.4.2
sshtunnel==0.4.0
stack-data==0.6.2
starkbank-ecdsa==2.2.0
statsd==4.0.1
tabulate==0.9.0
tblib==1.7.0
tenacity==8.2.0
termcolor==2.2.0
terminado==0.17.1
text-unidecode==1.3
thrift==0.16.0
time-machine==2.9.0
tinycss2==1.2.1
tomli==2.0.1
toolz==0.12.0
tornado==6.2
tqdm==4.64.1
traitlets==5.9.0
trio==0.22.0
trio-websocket==0.9.2
typing-extensions==4.4.0
tzdata==2022.7
tzlocal==4.2
uamqp==1.6.3
uc-micro-py==1.0.1
unicodecsv==0.14.1
uri-template==1.2.0
uritemplate==3.0.1
urllib3==1.26.14
vine==5.0.0
virtualenv==20.17.1
virtualenv-clone==0.5.7
watchtower==2.0.1
wcwidth==0.2.6
webcolors==1.12
webencodings==0.5.1
websocket-client==1.5.1
Werkzeug==2.2.2
widgetsnbextension==4.0.5
wrapt==1.15.0rc1
wsproto==1.2.0
WTForms==3.0.1
y-py==0.5.5
yarl==1.8.2
ypy-websocket==0.8.2
zict==2.2.0
zipp==3.12.1
zope.event==4.6
zope.interface==5.5.2

[austiezr]

Hey @RamanDamayeu ! To be clear, are you experiencing an identical error? Are you able to share that stack trace?

[RamanDamayeu]

Yeah, the error will be the same, @austiezr :

irflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 339, in parse
airflow-webserver_1  |     loader.exec_module(new_module)
airflow-webserver_1  |   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
airflow-webserver_1  |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
airflow-webserver_1  |   File "/opt/airflow/dags/dummy_gx_with_odd.py", line 9, in <module>
airflow-webserver_1  |     from great_expectations_provider.operators.great_expectations import GreatExpectationsOperator  # type: ignore
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations_provider/operators/great_expectations.py", line 24, in <module>
airflow-webserver_1  |     import great_expectations as ge
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/__init__.py", line 6, in <module>
airflow-webserver_1  |     from great_expectations.data_context.migrator.cloud_migrator import CloudMigrator
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/data_context/__init__.py", line 1, in <module>
airflow-webserver_1  |     from great_expectations.data_context.data_context import (
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/data_context/data_context/__init__.py", line 1, in <module>
airflow-webserver_1  |     from great_expectations.data_context.data_context.abstract_data_context import (
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/data_context/data_context/abstract_data_context.py", line 37, in <module>
airflow-webserver_1  |     from great_expectations.core import ExpectationSuite
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/core/__init__.py", line 11, in <module>
airflow-webserver_1  |     from .expectation_validation_result import (
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/core/expectation_validation_result.py", line 17, in <module>
airflow-webserver_1  |     from great_expectations.core.batch import BatchDefinition, BatchMarkers
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/core/batch.py", line 13, in <module>
airflow-webserver_1  |     from great_expectations.experimental.datasources.interfaces import (
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/experimental/__init__.py", line 1, in <module>
airflow-webserver_1  |     from great_expectations.experimental import datasources
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/experimental/datasources/__init__.py", line 1, in <module>
airflow-webserver_1  |     from great_expectations.experimental.datasources.pandas_datasource import (
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/experimental/datasources/pandas_datasource.py", line 15, in <module>
airflow-webserver_1  |     from great_expectations.experimental.datasources.dynamic_pandas import (
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/experimental/datasources/dynamic_pandas.py", line 35, in <module>
airflow-webserver_1  |     from great_expectations.experimental.datasources.interfaces import DataAsset
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/experimental/datasources/interfaces.py", line 314, in <module>
airflow-webserver_1  |     class Datasource(
airflow-webserver_1  |   File "/home/airflow/.local/lib/python3.9/site-packages/great_expectations/experimental/datasources/metadatasource.py", line 37, in __new__
airflow-webserver_1  |     cls = super().__new__(meta_cls, cls_name, bases, cls_dict)
airflow-webserver_1  |   File "pydantic/main.py", line 282, in pydantic.main.ModelMetaclass.__new__
airflow-webserver_1  |   File "pydantic/typing.py", line 287, in pydantic.typing.resolve_annotations
airflow-webserver_1  |   File "/usr/local/lib/python3.9/typing.py", line 292, in _eval_type
airflow-webserver_1  |     return t._evaluate(globalns, localns, recursive_guard)
airflow-webserver_1  |   File "/usr/local/lib/python3.9/typing.py", line 553, in _evaluate
airflow-webserver_1  |     type_ = _type_check(
airflow-webserver_1  |   File "/usr/local/lib/python3.9/typing.py", line 158, in _type_check
airflow-webserver_1  |     raise TypeError(f"{arg} is not valid as type argument")
airflow-webserver_1  | TypeError: typing.ClassVar[typing.List[typing.Type[great_expectations.experimental.datasources.interfaces.DataAsset]]] is not valid as type argument

[RamanDamayeu]

Upgrading to pydantic>=1.9.1 fixed the issues for me! (Python 3.9)