search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
3.92k stars 427 forks source link

[BUG]: `enable_cache` at the Step Configuration does not work #1446

Closed northanapon closed 1 year ago

northanapon commented 1 year ago

Contact Details [Optional]

No response

System Information

ZENML_SERVER_VERSION: 0.36.1
ZENML_SERVER_DATABASE: sqlite
ZENML_SERVER_DEPLOYMENT_TYPE: local
ZENML_CONFIG_DIR: /root/.config/zenml
ZENML_LOCAL_STORE_DIR: /root/.config/zenml/local_stores
ZENML_SERVER_URL: http://127.0.0.1:8237
ZENML_ACTIVE_REPOSITORY_ROOT: /caa-debug
PYTHON_VERSION: 3.8.13
ENVIRONMENT: docker
SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '20.04'}
ACTIVE_WORKSPACE: default
ACTIVE_STACK: caa_debug_stack
ACTIVE_USER: default
TELEMETRY_STATUS: enabled
ANALYTICS_CLIENT_ID: 86e09550-975e-4064-8a42-6837dc070629
ANALYTICS_USER_ID: d51b2809-915b-4a22-8071-48cfeaca6fb6
ANALYTICS_SERVER_ID: 86e09550-975e-4064-8a42-6837dc070629
INTEGRATIONS: ['github', 'graphviz', 'kaniko', 'lightgbm', 'mlflow', 'plotly', 'pytorch', 'pytorch_lightning', 'scipy', 'sklearn', 'xgboost']
PACKAGES: {'xarray': '2022.12.0', 'regex': '2022.3.15', 'fsspec': '2022.3.0', 'pytz': '2022.1', 'dask': '2021.11.2', 'distributed': '2021.11.2', 'certifi': '2021.10.8', 
'setuptools': '59.5.0', 'cryptography': '36.0.2', 'gevent': '22.10.2', 'pyzmq': '22.3.0', 'dask-cuda': '22.2.0', 'dask-cudf': '22.2.0a0+309.gdad51a548e', 'cudf': 
'22.2.0a0+309.gdad51a548e', 'cuml': '22.2.0a0+109.gc41beba23', 'cugraph': '22.2.0a0+95.g1a25c914', 'rmm': '22.2.0a0+37.gf2b1364', 'aiofiles': '22.1.0', 'pyopenssl': 
'22.0.0', 'attrs': '21.4.0', 'packaging': '21.3', 'argon2-cffi': '21.3.0', 'pip': '21.2.4', 'argon2-cffi-bindings': '21.2.0', 'gunicorn': '20.1.0', 'rich': '12.6.0', 
'cuda-python': '11.6.1', 'pynvml': '11.4.1', 'websockets': '10.4', 'cupy-cuda115': '9.6.0', 'pillow': '9.0.1', 'tensorrt': '8.2.4.2', 'ipython': '8.2.0', 'tenacity': 
'8.1.0', 'thinc': '8.0.15', 'click': '8.0.4', 'ipywidgets': '7.7.2', 'jupyter-client': '7.2.2', 'ipykernel': '6.13.0', 'colorlog': '6.7.0', 'nbconvert': '6.5.0', 'notebook':
'6.4.1', 'coverage': '6.3.2', 'pytest': '6.2.5', 'python-slugify': '6.1.1', 'tornado': '6.1', 'multidict': '6.0.4', 'docker': '6.0.1', 'pbr': '5.11.0', 'plotly': '5.11.0', 
'psutil': '5.9.0', 'importlib-resources': '5.7.0', 'zope.interface': '5.5.2', 'pyyaml': '5.4.1', 'nbformat': '5.3.0', 'smart-open': '5.2.1', 'decorator': '5.1.1', 
'traitlets': '5.1.1', 'smmap': '5.0.0', 'pyarrow': '5.0.0', 'cachetools': '5.0.0', 'bleach': '5.0.0', 'tqdm': '4.64.0', 'hypothesis': '4.50.8', 'fonttools': '4.32.0', 
'jsonargparse': '4.19.0', 'conda': '4.12.0', 'importlib-metadata': '4.11.3', 'beautifulsoup4': '4.11.1', 'antlr4-python3-runtime': '4.9.3', 'jupyter-core': '4.9.2', 
'pexpect': '4.8.0', 'rsa': '4.8', 'zope.event': '4.6', 'sphinx': '4.5.0', 'jsonschema': '4.4.0', 'typing-extensions': '4.1.1', 'stevedore': '4.1.1', 'cliff': '4.1.0', 
'gitdb': '4.0.10', 'mock': '4.0.3', 'async-timeout': '4.0.2', 'bcrypt': '4.0.1', 'libarchive-c': '4.0', 'chardet': '4.0.0', 'conda-build': '3.21.8', 'protobuf': '3.20.0', 
'orjson': '3.8.4', 'aiohttp': '3.8.3', 'zipp': '3.8.0', 'nltk': '3.7', 'anyio': '3.6.2', 'widgetsnbextension': '3.6.1', 'filelock': '3.6.0', 'asgiref': '3.6.0', 
'matplotlib': '3.5.1', 'markdown': '3.3.6', 'lightgbm': '3.3.4', 'langcodes': '3.3.0', 'idna': '3.3', 'python-jose': '3.3.0', 'spacy': '3.2.4', 'prettytable': '3.2.0', 
'oauthlib': '3.2.0', 'gitpython': '3.1.30', 'jinja2': '3.1.1', 'openpyxl': '3.1.1', 'threadpoolctl': '3.1.0', 'prompt-toolkit': '3.0.29', 'spacy-legacy': '3.0.9', 'preshed':
'3.0.6', 'optuna': '3.0.5', 'pytest-cov': '3.0.0', 'tritonclient': '2.29.0', 'requests': '2.27.1', 'pycparser': '2.21', 'fastjsonschema': '2.15.3', 'tensorboard': '2.12.0', 
'paramiko': '2.12.0', 'pygments': '2.11.2', 'pybind11': '2.9.2', 'babel': '2.9.1', 'python-dateutil': '2.8.2', 'google-auth': '2.6.5', 'networkx': '2.6.3', 'pyjwt': '2.6.0',
'tensorboardx': '2.5.1', 'pyparsing': '2.4.7', 'srsly': '2.4.3', 'cmd2': '2.4.2', 'sortedcontainers': '2.4.0', 'jupyterlab': '2.3.2', 'soupsieve': '2.3.1', 'omegaconf': 
'2.3.0', 'treelite-runtime': '2.2.1', 'treelite': '2.2.1', 'snowballstemmer': '2.2.0', 'codecov': '2.1.12', 'audioread': '2.1.9', 'pytorch-quantization': '2.1.2', 
'itsdangerous': '2.1.2', 'markupsafe': '2.1.1', 'mlflow': '2.1.1', 'werkzeug': '2.1.1', 'flask': '2.1.1', 'typeshed-client': '2.1.0', 'zict': '2.1.0', 'markdown-it-py': 
'2.1.0', 'charset-normalizer': '2.0.12', 'catalogue': '2.0.6', 'cymem': '2.0.6', 'asttokens': '2.0.5', 'kafka-python': '2.0.2', 'geventhttpclient': '2.0.2', 'greenlet': 
'2.0.1', 'tomli': '2.0.1', 'pycocotools': '2.0+nv0.6.0', 'cloudpickle': '2.0.0', 'sphinxcontrib-htmlhelp': '2.0.0', 'grpcio': '1.51.1', 'urllib3': '1.26.9', 'numpy': 
'1.22.3', 'python-hostlist': '1.21', 'six': '1.16.0', 'cffi': '1.15.0', 'jupytext': '1.13.8', 'nvidia-dali-cuda110': '1.12.0', 'torch': '1.12.0a0+bd13bc6', 'onnx': '1.11.0',
'py': '1.11.0', 'backoff': '1.10.0', 'pydantic': '1.9.2', 'python-rapidjson': '1.9', 'pytorch-lightning': '1.8.6', 'pkginfo': '1.8.2', 'pyperclip': '1.8.2', 'yarl': '1.8.2',
'conda-package-handling': '1.8.1', 'alembic': '1.8.1', 'scipy': '1.8.1', 'tensorboard-plugin-wit': '1.8.1', 'distro': '1.8.0', 'send2trash': '1.8.0', 'passlib': '1.7.4', 
'pysocks': '1.7.1', 'tblib': '1.7.0', 'backports.functools-lru-cache': '1.6.4', 'monotonic': '1.6', 'pooch': '1.6.0', 'debugpy': '1.6.0', 'nest-asyncio': '1.5.5', 
'feature-engine': '1.5.2', 'xgboost': '1.5.2', 'pynacl': '1.5.0', 'pandocfilters': '1.5.0', 'sqlalchemy': '1.4.41', 'appdirs': '1.4.4', 'websocket-client': '1.4.2', 'pydot':
'1.4.2', 'kiwisolver': '1.4.2', 'shellingham': '1.4.0', 'analytics-python': '1.4.0', 'pandas': '1.3.5', 'frozenlist': '1.3.3', 'aiosignal': '1.3.1', 'hydra-core': '1.3.1', 
'requests-oauthlib': '1.3.1', 'sniffio': '1.3.0', 'imagesize': '1.3.0', 'lmdb': '1.3.0', 'text-unidecode': '1.3', 'mako': '1.2.4', 'querystring-parser': '1.2.4', 
'mlserver-mlflow': '1.2.1', 'mlserver': '1.2.1', 'joblib': '1.2.0', 'scikit-learn': '1.2.0', 'partd': '1.2.0', 'jupyterlab-server': '1.2.0', 'sphinxcontrib-serializinghtml':
'1.1.5', 'munkres': '1.1.4', 'jupyterlab-widgets': '1.1.1', 'catboost': '1.1.1', 'iniconfig': '1.1.1', 'tinycss2': '1.1.1', 'et-xmlfile': '1.1.0', 'torch-tensorrt': 
'1.1.0a0', 'brotli': '1.0.9', 'nvidia-pyindex': '1.0.9', 'murmurhash': '1.0.6', 'msgpack': '1.0.3', 'sphinxcontrib-qthelp': '1.0.3', 'spacy-loggers': '1.0.2', 'pymysql': 
'1.0.2', 'sphinxcontrib-applehelp': '1.0.2', 'sphinxcontrib-devhelp': '1.0.2', 'heapdict': '1.0.1', 'sphinxcontrib-jsmath': '1.0.1', 'absl-py': '1.0.0', 'sphinx-rtd-theme': 
'1.0.0', 'pluggy': '1.0.0', 'fastapi': '0.75.2', 'numba': '0.53.1', 'shap': '0.41.0', 'sqlalchemy-utils': '0.38.3', 'wheel': '0.37.1', 'zenml': '0.36.1', 'llvmlite': 
'0.36.0', 'polygraphy': '0.33.0', 'cython': '0.29.28', 'prometheus-flask-exporter': '0.21.0', 'python-dotenv': '0.21.0', 'ucx-py': '0.21.0a0+37.gbfa0450', 'graphviz': 
'0.20.1', 'httplib2': '0.19.1', 'validators': '0.18.2', 'future': '0.18.2', 'jedi': '0.18.1', 'pyrsistent': '0.18.1', 'ecdsa': '0.18.0', 'uvicorn': '0.17.6', 
'databricks-cli': '0.17.4', 'starlette': '0.17.1', 'docutils': '0.17.1', 'uvloop': '0.17.0', 'ruamel-yaml-conda': '0.15.80', 'docstring-parser': '0.15', 'python-nvd3': 
'0.15.0', 'prometheus-client': '0.14.1', 'h11': '0.14.0', 'starlette-exporter': '0.14.0', 'statsmodels': '0.13.5', 'terminado': '0.13.3', 'torchvision': '0.13.0a0', 
'seaborn': '0.12.2', 'huggingface-hub': '0.12.1', 'toolz': '0.11.2', 'torchmetrics': '0.11.0', 'cycler': '0.11.0', 'soundfile': '0.10.3.post1', 'toml': '0.10.2', 
'python-terraform': '0.10.1', 'snorkel': '0.9.9', 'json5': '0.9.6', 'wasabi': '0.9.1', 'commonmark': '0.9.1', 'cmaes': '0.9.1', 'tabulate': '0.8.9', 'mistune': '0.8.4', 
'parso': '0.8.3', 'executing': '0.8.3', 'watchgod': '0.8.2', 'librosa': '0.8.1', 'dataclasses': '0.8', 'aiokafka': '0.8.0', 'fastrlock': '0.8', 'alabaster': '0.7.12', 
'blis': '0.7.7', 'pickleshare': '0.7.5', 'pytest-pythonpath': '0.7.4', 'defusedxml': '0.7.1', 'glob2': '0.7', 'brotlipy': '0.7.0', 'ptyprocess': '0.7.0', 
'tensorboard-data-server': '0.7.0', 'py-grpc-prometheus': '0.7.0', 'uff': '0.6.9', 'pycosat': '0.6.3', 'pathy': '0.6.1', 'nbclient': '0.6.0', 'patsy': '0.5.3', 'autopage': 
'0.5.1', 'webencodings': '0.5.1', 'skops': '0.5.0', 'httptools': '0.5.0', 'lightning-utilities': '0.5.0', 'pyasn1': '0.4.8', 'google-auth-oauthlib': '0.4.6', 'graphsurgeon':
'0.4.5', 'colorama': '0.4.4', 'sqlparse': '0.4.3', 'typer': '0.4.1', 'entrypoints': '0.4', 'click-params': '0.3.0', 'sphinx-glpi-theme': '0.3', 'mdit-py-plugins': '0.3.0', 
'pysftp': '0.2.9', 'pyasn1-modules': '0.2.8', 'wcwidth': '0.2.5', 'nvtx': '0.2.4', 'pure-eval': '0.2.2', 'resampy': '0.2.2', 'jupyterlab-pygments': '0.2.2', 'fastapi-utils':
'0.2.1', 'locket': '0.2.1', 'stack-data': '0.2.0', 'backcall': '0.2.0', 'jupyter-tensorboard': '0.2.0', 'ipython-genutils': '0.2.0', 'matplotlib-inline': '0.1.3', 
'expecttest': '0.1.3', 'mdurl': '0.1.1', 'apex': '0.1', 'sacremoses': '0.0.49', 'sqlmodel': '0.0.8', 'slicer': '0.0.7', 'python-multipart': '0.0.5', 'revtok': '0.0.3', 
'sqlalchemy2-stubs': '0.0.2a31', 'torchtext': '0.13.0a0'}

CURRENT STACK

Name: caa_debug_stack
ID: 3e988b7a-10e4-4439-975f-ced95f672107
Shared: No
User: default / d51b2809-915b-4a22-8071-48cfeaca6fb6
Workspace: default / a426b027-5f09-4503-bbb6-09aa00b7b3bc

ORCHESTRATOR: default

Name: default
ID: c6281fad-5876-4456-ad4c-093cfc9b1b5d
Type: orchestrator
Flavor: local
Configuration: {}
Shared: No
User: default / d51b2809-915b-4a22-8071-48cfeaca6fb6
Workspace: default / a426b027-5f09-4503-bbb6-09aa00b7b3bc

ARTIFACT_STORE: default

Name: default
ID: 5d50ea2a-97b6-4801-9180-2c43eebc1d08
Type: artifact_store
Flavor: local
Configuration: {'path': ''}
Shared: No
User: default / d51b2809-915b-4a22-8071-48cfeaca6fb6
Workspace: default / a426b027-5f09-4503-bbb6-09aa00b7b3bc

EXPERIMENT_TRACKER: mlflow_tracker

Name: mlflow_tracker
ID: 3d0d1bb1-eb3b-46e8-852e-fd24d629115f
Type: experiment_tracker
Flavor: mlflow
Configuration: {'experiment_name': None, 'nested': False, 'tags': {}, 'tracking_uri': 'http://10.34.33.203:5011/', 'tracking_username': '********', 'tracking_password': 
'********', 'tracking_token': '********', 'tracking_insecure_tls': True, 'databricks_host': None}
Shared: No
User: default / d51b2809-915b-4a22-8071-48cfeaca6fb6
Workspace: default / a426b027-5f09-4503-bbb6-09aa00b7b3bc

What happened?

I set the enable_cache for a step to be false, but zenml run still uses the cached version. Below is the example of the debug message.

Reproduction steps

  1. At the pipeline configuration, enable_cache: true
  2. At the step configuration, enable_cache: false

Relevant log output

2023-03-30 02:58:56,023 - zenml.steps.base_step - DEBUG - Step 'log_model': Caching disabled. (base_step.py:293)
2023-03-30 02:58:56,023 - zenml.steps.base_step - DEBUG - Step 'log_model': Artifact metadata enabled. (base_step.py:303)
2023-03-30 02:58:56,024 - zenml.steps.base_step - DEBUG - Updated step configuration: (base_step.py:861)
2023-03-30 02:58:56,024 - zenml.steps.base_step - DEBUG - name='log_model' enable_cache=False enable_artifact_metadata=None step_operator=None experiment_tracker='mlflow_tracker' parameters={} settings={} extra={} failure_hook_source=None success_hook_source=None outputs={} caching_parameters={} inputs={} (base_step.py:862)
2023-03-30 02:58:56,025 - zenml.zen_stores.rest_zen_store - DEBUG - Sending GET request to /steps... (rest_zen_store.py:2036)
2023-03-30 02:58:56,961 - zenml.zen_stores.rest_zen_store - DEBUG - Sending GET request to /steps... (rest_zen_store.py:2036)
2023-03-30 02:58:57,002 - zenml.orchestrators.step_launcher - INFO - Using cached version of log_model. (step_launcher.py:331)


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct
schustmi commented 1 year ago

Hey @northanapon, I'm not able to reproduce this issue. Can you share some minimal code example that runs into this issue?

One thing that might cause your step to be cached: If you pass cache_enabled=True to pipeline_instance.run(...) either directly or as part of a configuration file, that overrides the values defined for any step in the pipeline. Could that be the case for you?

northanapon commented 1 year ago

@schustmi I set the cache_enabled=True at the top of my configuration file, thinking each step setting would override it.

After removing the top-level setting, it works as expected now. Thank you for your suggestion.