search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
3.92k stars 427 forks source link

[BUG]: Table pipeline_deployment.pipeline_config column TEXT datatype is not long enough #1420

Closed francoisserra closed 1 year ago

francoisserra commented 1 year ago

Contact Details [Optional]

francois.serra@adeo.com

System Information

ZENML_LOCAL_VERSION: 0.35.1 ZENML_SERVER_VERSION: 0.35.1 ZENML_SERVER_DATABASE: mysql ZENML_SERVER_DEPLOYMENT_TYPE: other ZENML_CONFIG_DIR: /home/REDACTED/.config/zenml ZENML_LOCAL_STORE_DIR: /home/REDACTED/.config/zenml/local_stores ZENML_SERVER_URL: REDACTED ZENML_ACTIVE_REPOSITORY_ROOT: None PYTHON_VERSION: 3.9.15 ENVIRONMENT: native SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'} ACTIVE_WORKSPACE: default ACTIVE_STACK: default ACTIVE_USER: fserra TELEMETRY_STATUS: enabled ANALYTICS_CLIENT_ID: 9e0d5260-2ef5-4828-be32-d9f57c0d4f1f ANALYTICS_USER_ID: eb05dc4d-7152-4ba9-881a-81ceeed47534 ANALYTICS_SERVER_ID: 85bd5e5b-67a5-4f15-a699-ba90d709b722 INTEGRATIONS: ['kaniko', 'kubeflow', 'kubernetes', 'mlflow', 'pillow', 'scipy', 'sklearn'] PACKAGES: {'zope.interface': '5.5.2', 'zope.event': '4.5.0', 'zipp': '3.11.0', 'zenml': '0.35.1', 'yarl': '1.8.2', 'wrapt': '1.14.1', 'widgetsnbextension': '3.6.1', 'wheel': '0.38.4', 'werkzeug': '2.2.2', 'websocket-client': '1.4.2', 'webencodings': '0.5.1', 'webcolors': '1.12', 'wcwidth': '0.2.5', 'validators': '0.18.2', 'uvloop': '0.17.0', 'uvicorn': '0.20.0', 'urllib3': '1.26.13', 'uritemplate': '3.0.1', 'uri-template': '1.2.0', 'typing-inspect': '0.8.0', 'typing-extensions': '4.4.0', 'typer': '0.7.0', 'typeguard': '2.13.3', 'tritonclient': '2.28.0', 'traitlets': '5.6.0', 'tqdm': '4.65.0', 'tornado': '6.2', 'tomli': '2.0.1', 'threadpoolctl': '3.1.0', 'testslide': '2.7.0', 'testpath': '0.6.0', 'terminado': '0.17.1', 'termcolor': '2.2.0', 'tabulate': '0.9.0', 'strip-hints': '0.1.10', 'starlette': '0.22.0', 'starlette-exporter': '0.14.0', 'stack-data': '0.6.2', 'sqlparse': '0.4.3', 'sqlmodel': '0.0.8', 'sqlalchemy2-stubs': '0.0.2a29', 'sqlalchemy': '1.4.41', 'sqlalchemy-utils': '0.38.3', 'soupsieve': '2.3.2.post1', 'sortedcontainers': '2.4.0', 'soda-core': '3.0.14', 'soda-core-bigquery': '3.0.14', 'sniffio': '1.3.0', 'smmap': '5.0.0', 'slicer': '0.0.7', 'six': '1.16.0', 'shap': '0.41.0', 'setuptools': '58.1.0', 'send2trash': '1.8.0', 'scipy': '1.9.3', 'scikit-learn': '1.2.1', 'ruamel.yaml': '0.17.21', 'ruamel.yaml.clib': '0.2.7', 'rsa': '4.9', 'rich': '12.6.0', 'rfc3986-validator': '0.1.1', 'rfc3339-validator': '0.1.4', 'requests': '2.27.1', 'requests-toolbelt': '0.10.1', 'requests-oauthlib': '1.3.1', 'querystring-parser': '1.2.4', 'pyzmq': '24.0.1', 'pyyaml': '5.4.1', 'pytz': '2022.6', 'python-terraform': '0.10.1', 'python-rapidjson': '1.9', 'python-json-logger': '2.0.4', 'python-dotenv': '0.21.0', 'python-dateutil': '2.8.2', 'pytest': '7.2.1', 'pyrsistent': '0.19.2', 'pyre-extensions': '0.0.30', 'pyre-check': '0.9.18', 'pyparsing': '2.4.7', 'pymysql': '1.0.2', 'pyjwt': '2.6.0', 'pygments': '2.13.0', 'pyflakes': '3.0.1', 'pydantic': '1.9.2', 'pycparser': '2.21', 'pycodestyle': '2.10.0', 'pyasn1': '0.4.8', 'pyasn1-modules': '0.2.8', 'pyarrow': '9.0.0', 'py-grpc-prometheus': '0.7.0', 'pure-eval': '0.2.2', 'ptyprocess': '0.7.0', 'psutil': '5.9.4', 'protobuf': '3.19.6', 'proto-plus': '1.22.1', 'prompt-toolkit': '3.0.36', 'prometheus-flask-exporter': '0.21.0', 'prometheus-client': '0.15.0', 'portpicker': '1.5.2', 'pluggy': '1.0.0', 'platformdirs': '2.6.0', 'pip': '23.0.1', 'pillow': '9.4.0', 'pickleshare': '0.7.5', 'pexpect': '4.8.0', 'pathspec': '0.10.2', 'passlib': '1.7.4', 'parso': '0.8.3', 'pandocfilters': '1.5.0', 'pandas': '1.5.2', 'packaging': '21.3', 'orjson': '3.8.3', 'opentelemetry-semantic-conventions': '0.30b1', 'opentelemetry-sdk': '1.11.1', 'opentelemetry-proto': '1.11.1', 'opentelemetry-exporter-otlp-proto-http': '1.11.1', 'opentelemetry-api': '1.11.1', 'oauthlib': '3.2.2', 'oauth2client': '4.1.3', 'numpy': '1.23.5', 'numba': '0.56.4', 'notebook': '6.5.2', 'notebook-shim': '0.2.2', 'nest-asyncio': '1.5.6', 'nbformat': '5.7.0', 'nbconvert': '6.4.5', 'nbclient': '0.5.13', 'nbclassic': '0.4.8', 'mypy-extensions': '0.4.3', 'multidict': '6.0.3', 'monotonic': '1.6', 'mlserver': '1.2.0', 'mlserver-mlflow': '1.2.0', 'mlflow': '2.0.1', 'mlflow-skinny': '2.0.1', 'ml-pipelines-sdk': '1.8.0', 'ml-metadata': '1.8.0', 'mistune': '0.8.4', 'mccabe': '0.7.0', 'matplotlib': '3.7.1', 'matplotlib-inline': '0.1.6', 'marshmallow': '3.19.0', 'marshmallow-enum': '1.5.1', 'markupsafe': '2.1.2', 'markdown': '3.4.1', 'mako': '1.2.4', 'llvmlite': '0.39.1', 'libcst': '0.4.9', 'kubernetes': '18.20.0', 'kiwisolver': '1.4.4', 'kfp': '1.8.16', 'kfp-server-api': '1.8.5', 'kfp-pipeline-spec': '0.1.17', 'kafka-python': '2.0.2', 'jupyterlab-widgets': '1.1.1', 'jupyterlab-pygments': '0.2.2', 'jupyter-server': '2.0.1', 'jupyter-server-terminals': '0.4.2', 'jupyter-events': '0.6.3', 'jupyter-core': '5.1.0', 'jupyter-client': '7.4.8', 'jsonschema': '3.2.0', 'jsonpointer': '2.3', 'joblib': '1.2.0', 'jinja2': '3.1.2', 'jedi': '0.18.2', 'itsdangerous': '2.1.2', 'isort': '5.12.0', 'isoduration': '20.11.0', 'ipywidgets': '7.7.2', 'ipython': '8.7.0', 'ipython-genutils': '0.2.0', 'ipykernel': '6.19.1', 'intervaltree': '3.1.0', 'iniconfig': '2.0.0', 'importlib-resources': '5.12.0', 'importlib-metadata': '5.1.0', 'idna': '3.4', 'hypothesis': '6.68.2', 'httplib2': '0.19.1', 'h11': '0.14.0', 'gunicorn': '20.1.0', 'grpcio': '1.51.1', 'grpcio-status': '1.48.2', 'grpc-google-iam-v1': '0.12.6', 'greenlet': '2.0.1', 'googleapis-common-protos': '1.57.0', 'google-resumable-media': '2.4.0', 'google-crc32c': '1.5.0', 'google-cloud-storage': '2.7.0', 'google-cloud-secret-manager': '2.12.6', 'google-cloud-scheduler': '2.9.0', 'google-cloud-resource-manager': '1.8.0', 'google-cloud-functions': '1.10.0', 'google-cloud-core': '2.3.2', 'google-cloud-bigquery': '2.34.4', 'google-cloud-aiplatform': '1.19.0', 'google-auth': '2.15.0', 'google-auth-oauthlib': '0.8.0', 'google-auth-httplib2': '0.1.0', 'google-apitools': '0.5.32', 'google-api-python-client': '1.12.11', 'google-api-core': '2.11.0', 'gitpython': '3.1.29', 'gitdb': '4.0.10', 'geventhttpclient': '2.0.2', 'gevent': '22.10.2', 'gcsfs': '2022.11.0', 'fsspec': '2022.11.0', 'frozenlist': '1.3.3', 'fqdn': '1.5.1', 'fonttools': '4.39.0', 'flask': '2.2.2', 'flake8': '6.0.0', 'fire': '0.5.0', 'fastjsonschema': '2.16.2', 'fasteners': '0.18', 'fastapi': '0.88.0', 'executing': '1.2.0', 'exceptiongroup': '1.1.0', 'entrypoints': '0.4', 'docstring-parser': '0.15', 'docker': '6.0.1', 'distro': '1.8.0', 'deprecated': '1.2.13', 'defusedxml': '0.7.1', 'decorator': '5.1.1', 'debugpy': '1.6.4', 'db-dtypes': '1.0.4', 'datadog': '0.44.0', 'dataclasses-json': '0.5.7', 'databricks-cli': '0.17.4', 'cycler': '0.11.0', 'contourpy': '1.0.7', 'commonmark': '0.9.1', 'comm': '0.1.2', 'cloudpickle': '2.2.0', 'click': '8.1.3', 'click-params': '0.3.0', 'charset-normalizer': '2.0.12', 'cffi': '1.15.1', 'certifi': '2022.12.7', 'cachetools': '5.2.0', 'brotli': '1.0.9', 'bleach': '5.0.1', 'black': '22.12.0', 'beautifulsoup4': '4.11.1', 'bcrypt': '4.0.1', 'backoff': '1.10.0', 'backcall': '0.2.0', 'autopep8': '2.0.1', 'attrs': '20.3.0', 'async-timeout': '4.0.2', 'asttokens': '2.2.1', 'arrow': '1.2.3', 'argon2-cffi': '21.3.0', 'argon2-cffi-bindings': '21.2.0', 'anyio': '3.6.2', 'antlr4-python3-runtime': '4.11.1', 'analytics-python': '1.4.0', 'alembic': '1.8.1', 'aiosignal': '1.3.1', 'aiokafka': '0.8.0', 'aiohttp': '3.8.3', 'aiofiles': '22.1.0', 'absl-py': '1.3.0'}

What happened?

SQLAlchemy exception when trying to launch a pipeline, due to the fact that the datatype of the column step_configurations of the table pipeline_deployment is TEXT which is limited to 65,535 chars.

the usage of a wider text datatype (MEDIUMTEXT or String(2**24) for sql alchemy) should do the trick

Reproduction steps

use wide step configuration parameter object that will produce a wide step_configurations json representation is > 65,535 chars and/or use lot of steps

Relevant log output

2023-03-16T14:13:23.869739647Z   File "/usr/local/lib/python3.8/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
2023-03-16T14:13:23.869743157Z     raise errorclass(errno, errval)
2023-03-16T14:13:23.869746657Z sqlalchemy.exc.DataError: (pymysql.err.DataError) (1406, "Data too long for column 'step_configurations' at row 1")
2023-03-16T14:13:23.869750897Z [SQL: INSERT INTO pipeline_deployment (stack_id, pipeline_id, build_id, schedule_id, user_id, workspace_id, pipeline_configuration, step_configurations, client_environment, id, created, updated, run_name_template) VALUES (%(stack_id)s, %(pipeline_id)s, %(build_id)s, %(schedule_id)s, %(user_id)s, %(workspace_id)s, %(pipeline_configuration)s, %(step_configurations)s, %(client_environment)s, %(id)s, %(created)s, %(updated)s, %(run_name_template)s)]
2023-03-16T14:13:23.869763785Z [parameters: {'stack_id': 'faf848638e5845f2b132bfcdf1f5b787', 'pipeline_id': '0a751ece32a64573a83a002255d0532a', 'build_id': None, 'schedule_id': None, 'user_id': 'eb05dc4d71524ba9881a81ceeed47534', 'workspace_id': '8864453db23346538e17a1d79c8a6711', 'pipeline_configuration': '{"enable_cache": false, "enable_artifact_metadata": null, "settings": {"docker": {"install_stack_requirements": false, "environment": {... (63 characters truncated) ... : null}, "requirements": "./requirements/vertex_requirements.txt"}}, "extra": {}, "name": "ranking_training_pipeline"}', 'step_configurations': '{"retrieve_data_points": {"spec": {"source": "steps.retrieve_data_points.bq_retrieve_data_points.retrieve_data_points", "upstream_steps": [], "inputs ... (82326 characters truncated) ... iltInContainerMaterializer"}, "test_table_ref": {"materializer_source": "zenml.materializers.built_in_materializer.BuiltInContainerMaterializer"}}}}}', 'client_environment': '{"environment": "native", "os": "linux", "linux_distro": "ubuntu", "linux_distro_like": "debian", "linux_distro_version": "22.04", "python_version": "3.9.15"}', 'id': '2b3e826cdbae40bbb7e018bba5998a9d', 'created': datetime.datetime(2023, 3, 16, 14, 13, 23, 858137), 'updated': datetime.datetime(2023, 3, 16, 14, 13, 23, 858141), 'run_name_template': 'ranking_training_pipeline-{date}-{time}'}]
2023-03-16T14:13:23.869771982Z (Background on this error at: https://sqlalche.me/e/14/9h9h)

Code of Conduct

strickvl commented 1 year ago

Thanks for filing this issue. I've made a PR that should open things up for you again.

strickvl commented 1 year ago

This has now been merged to develop via #1422 and it'll be in the next release.