zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
3.95k stars 430 forks source link

[BUG]: Cannot run example with Label Studio #1533

Closed adamwawrzynski closed 1 year ago

adamwawrzynski commented 1 year ago

Contact Details [Optional]

adamwawrzynski@wp.pl

System Information

ZENML_LOCAL_VERSION: 0.39.1 ZENML_SERVER_VERSION: 0.39.1 ZENML_SERVER_DATABASE: sqlite ZENML_SERVER_DEPLOYMENT_TYPE: other ZENML_CONFIG_DIR: /home/user/.config/zenml ZENML_LOCAL_STORE_DIR: /home/user/.config/zenml/local_stores ZENML_SERVER_URL: sqlite:////home/user/.config/zenml/local_stores/default_zen_store/zenml.db ZENML_ACTIVE_REPOSITORY_ROOT: /home/user/praca/zenml_playground/label_studio/zenml_examples/label_studio_annotation PYTHON_VERSION: 3.8.16 ENVIRONMENT: native SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'} ACTIVE_WORKSPACE: default ACTIVE_STACK: annotation_gcp ACTIVE_USER: default TELEMETRY_STATUS: enabled ANALYTICS_CLIENT_ID: 57f7fb12-7d7f-45e7-8425-422a67baee43 ANALYTICS_USER_ID: f76122fb-e645-423c-b5e0-459bc7c11e1f ANALYTICS_SERVER_ID: 57f7fb12-7d7f-45e7-8425-422a67baee43 INTEGRATIONS: ['gcp', 'kaniko', 'kubeflow', 'kubernetes', 'label_studio', 'pillow', 'pytorch', 'seldon'] PACKAGES: {'certifi': '2023.5.7', 'regex': '2023.5.5', 'gcsfs': '2023.5.0', 'fsspec': '2023.5.0', 'tzdata': '2023.3', 'pytz': '2019.3', 'setuptools': '65.6.3', 'cryptography': '40.0.2', 'pyzmq': '25.0.2', 'pip': '23.1.2', 'packaging': '23.1', 'attrs': '23.1.0', 'argon2-cffi': '21.3.0', 'argon2-cffi-bindings': '21.2.0', 'isoduration': '20.11.0', 'kubernetes': '18.20.0', 'lit': '16.0.3', 'azure-storage-blob': '12.16.0', 'rich': '12.6.0', 'nvidia-cublas-cu11': '11.10.3.66', 'nvidia-cuda-cupti-cu11': '11.7.101', 'nvidia-cuda-runtime-cu11': '11.7.99', 'nvidia-cuda-nvrtc-cu11': '11.7.99', 'nvidia-nvtx-cu11': '11.7.91', 'nvidia-cusparse-cu11': '11.7.4.91', 'nvidia-cusolver-cu11': '11.4.0.1', 'websockets': '11.0.3', 'nvidia-cufft-cu11': '10.9.0.58', 'nvidia-curand-cu11': '10.2.10.91', 'pillow': '9.5.0', 'ipython': '8.12.2', 'nvidia-cudnn-cu11': '8.5.0.96', 'jupyter-client': '8.2.0', 'click': '8.1.3', 'ipywidgets': '7.7.5', 'nbconvert': '7.4.0', 'launchdarkly-server-sdk': '7.3.0', 'ipykernel': '6.23.0', 'importlib-metadata': '6.6.0', 'notebook': '6.5.4', 'tornado': '6.3.1', 'multidict': '6.0.4', 'docker': '6.0.1', 'bleach': '6.0.0', 'importlib-resources': '5.12.0', 'psutil': '5.9.5', 'traitlets': '5.9.0', 'nbformat': '5.8.0', 'ujson': '5.7.0', 'pyyaml': '5.4.1', 'jupyter-core': '5.3.0', 'cachetools': '5.3.0', 'decorator': '5.1.1', 'smmap': '5.0.0', 'tqdm': '4.65.0', 'beautifulsoup4': '4.12.2', 'lxml': '4.9.2', 'rsa': '4.9', 'pexpect': '4.8.0', 'redis': '4.5.5', 'typing-extensions': '4.5.0', 'django-model-utils': '4.1.1', 'gitdb': '4.0.10', 'async-timeout': '4.0.2', 'ordered-set': '4.0.2', 'bcrypt': '4.0.1', 'cmake': '3.26.3', 'protobuf': '3.20.3', 'simplejson': '3.19.1', 'google-cloud-build': '3.16.0', 'zipp': '3.15.0', 'djangorestframework': '3.13.1', 'filelock': '3.12.0', 'google-cloud-bigquery': '3.10.0', 'orjson': '3.8.12', 'aiohttp': '3.8.4', 'nltk': '3.6.7', 'widgetsnbextension': '3.6.4', 'anyio': '3.6.2', 'django-cors-headers': '3.6.0', 'asgiref': '3.6.0', 'platformdirs': '3.5.0', 'idna': '3.4', 'python-jose': '3.3.0', 'django': '3.2.14', 'oauthlib': '3.2.2', 'django-debug-toolbar': '3.2.1', 'jsonschema': '3.2.0', 'gitpython': '3.1.31', 'jinja2': '3.1.2', 'django-extensions': '3.1.0', 'networkx': '3.1', 'prompt-toolkit': '3.0.38', 'uritemplate': '3.0.1', 'boto': '2.49.0', 'requests': '2.27.1', 'pycparser': '2.21', 'google-auth': '2.18.0', 'fastjsonschema': '2.16.3', 'google-cloud-secret-manager': '2.16.1', 'pygments': '2.15.1', 'nvidia-nccl-cu11': '2.14.3', 'semver': '2.13.0', 'google-api-core': '2.11.0', 'google-cloud-scheduler': '2.11.0', 'psycopg2-binary': '2.9.1', 'python-dateutil': '2.8.2', 'google-cloud-logging': '2.7.2', 'django-rq': '2.5.1', 'jupyter-server': '2.5.0', 'google-cloud-storage': '2.5.0', 'pyparsing': '2.4.7', 'soupsieve': '2.4.1', 'django-filter': '2.4.0', 'google-resumable-media': '2.3.3', 'coreapi': '2.3.3', 'google-cloud-core': '2.3.2', 'jsonpointer': '2.3', 'termcolor': '2.3.0', 'asttokens': '2.2.1', 'cloudpickle': '2.2.1', 'django-rest-swagger': '2.2.0', 'rules': '2.2', 'user-agents': '2.2.0', 'markupsafe': '2.1.2', 'charset-normalizer': '2.0.12', 'mistune': '2.0.5', 'python-json-logger': '2.0.4', 'greenlet': '2.0.2', 'torch': '2.0.1', 'triton': '2.0.0', 'googleapis-common-protos': '1.56.4', 'grpcio': '1.54.0', 'grpcio-status': '1.48.2', 'urllib3': '1.26.15', 'azure-core': '1.26.4', 'google-cloud-aiplatform': '1.25.0', 'numpy': '1.24.3', 'proto-plus': '1.22.2', 'sentry-sdk': '1.22.2', 'drf-yasg': '1.20.0', 'botocore': '1.19.63', 'boto3': '1.16.63', 'six': '1.16.0', 'cffi': '1.15.1', 'wrapt': '1.15.0', 'webcolors': '1.13', 'google-api-python-client': '1.12.11', 'sympy': '1.12', 'google-cloud-functions': '1.12.0', 'pydantic': '1.10.7', 'rq': '1.10.1', 'backoff': '1.10.0', 'google-cloud-resource-manager': '1.10.0', 'yarl': '1.9.2', 'kfp': '1.8.16', 'shapely': '1.8.5.post1', 'kfp-server-api': '1.8.5', 'send2trash': '1.8.2', 'alembic': '1.8.1', 'distro': '1.8.0', 'passlib': '1.7.4', 'debugpy': '1.6.7', 'monotonic': '1.6', 'label-studio': '1.6.0', 'nest-asyncio': '1.5.6', 'websocket-client': '1.5.1', 'fqdn': '1.5.1', 'pandocfilters': '1.5.0', 'google-crc32c': '1.5.0', 'sqlalchemy': '1.4.41', 'appdirs': '1.4.4', 'analytics-python': '1.4.post1', 'absl-py': '1.4.0', 'pkgutil-resolve-name': '1.3.10', 'pandas': '1.3.5', 'frozenlist': '1.3.3', 'openapi-codec': '1.3.2', 'requests-oauthlib': '1.3.1', 'aiosignal': '1.3.1', 'sniffio': '1.3.0', 'mpmath': '1.3.0', 'deprecated': '1.2.13', 'mako': '1.2.4', 'arrow': '1.2.3', 'tinycss2': '1.2.1', 'itypes': '1.2.0', 'executing': '1.2.0', 'uri-template': '1.2.0', 'joblib': '1.2.0', 'expiringdict': '1.1.4', 'jupyterlab-widgets': '1.1.4', 'google-cloud-appengine-logging': '1.1.0', 'pyrfc3339': '1.1', 'pymysql': '1.0.3', 'google-auth-oauthlib': '1.0.0', 'python-dotenv': '1.0.0', 'nbclassic': '1.0.0', 'fastapi': '0.75.2', 'zenml': '0.39.1', 'wheel': '0.38.4', 'sqlalchemy-utils': '0.38.3', 'pyrsistent': '0.19.3', 'httplib2': '0.19.1', 'validators': '0.18.2', 'jedi': '0.18.2', 'ecdsa': '0.18.0', 'ruamel.yaml': '0.17.26', 'uvicorn': '0.17.6', 'starlette': '0.17.1', 'terminado': '0.17.1', 'uvloop': '0.17.0', 'ua-parser': '0.16.1', 'prometheus-client': '0.16.0', 'torchvision': '0.15.2', 'docstring-parser': '0.15', 'h11': '0.14.0', 'grpc-google-iam-v1': '0.12.6', 'lockfile': '0.12.2', 'django-annoying': '0.10.6', 'requests-toolbelt': '0.10.1', 'python-terraform': '0.10.1', 'jmespath': '0.10.0', 'drf-flex-fields': '0.9.5', 'commonmark': '0.9.1', 'tabulate': '0.9.0', 'typer': '0.9.0', 'parso': '0.8.3', 'watchgod': '0.8.2', 'pickleshare': '0.7.5', 'nbclient': '0.7.4', 'defusedxml': '0.7.1', 'ptyprocess': '0.7.0', 'jupyter-events': '0.6.3', 'stack-data': '0.6.2', 'isodate': '0.6.1', 'webencodings': '0.5.1', 'inflection': '0.5.1', 'httptools': '0.5.0', 'pyasn1': '0.5.0', 'fire': '0.5.0', 'colorama': '0.4.6', 'sqlparse': '0.4.4', 'jupyter-server-terminals': '0.4.4', 'django-user-agents': '0.4.0', 's3transfer': '0.3.7', 'attr': '0.3.1', 'pyasn1-modules': '0.3.0', 'click-params': '0.3.0', 'drf-dynamic-fields': '0.3.0', 'drf-generators': '0.3.0', 'ruamel.yaml.clib': '0.2.7', 'wcwidth': '0.2.6', 'notebook-shim': '0.2.3', 'jupyterlab-pygments': '0.2.2', 'pure-eval': '0.2.2', 'fastapi-utils': '0.2.1', 'ipython-genutils': '0.2.0', 'google-cloud-audit-log': '0.2.0', 'xmljson': '0.2.0', 'backcall': '0.2.0', 'kfp-pipeline-spec': '0.1.16', 'htmlmin': '0.1.12', 'strip-hints': '0.1.10', 'matplotlib-inline': '0.1.6', 'rfc3339-validator': '0.1.4', 'boxing': '0.1.4', 'comm': '0.1.3', 'django-ranged-fileresponse': '0.1.2', 'rfc3986-validator': '0.1.1', 'google-auth-httplib2': '0.1.0', 'label-studio-converter': '0.0.44', 'label-studio-sdk': '0.0.23', 'sqlmodel': '0.0.8', 'python-multipart': '0.0.6', 'coreschema': '0.0.4', 'sqlalchemy2-stubs': '0.0.2a34', 'label-studio-tools': '0.0.1'}

CURRENT STACK

Name: annotation_gcp ID: 74dca47f-cd22-4f0c-8867-d0ddbea65dd4 Shared: No User: default / f76122fb-e645-423c-b5e0-459bc7c11e1f Workspace: default / ef66861d-f662-47f7-8fb2-7f60a7f8e8d3

ORCHESTRATOR: default

Name: default ID: d5a4642a-6849-4f15-a361-0380bdec52a4 Type: orchestrator Flavor: local Configuration: {} Shared: No User: default / f76122fb-e645-423c-b5e0-459bc7c11e1f Workspace: default / ef66861d-f662-47f7-8fb2-7f60a7f8e8d3

ARTIFACT_STORE: gcp_artifact_store

Name: gcp_artifact_store ID: 49628835-50e5-4618-8d51-7f2e57809153 Type: artifact_store Flavor: gcp Configuration: {'authentication_secret': 'GCP_AUTH_SECRET', 'path': 'gs://label_studio_example'} Shared: No User: default / f76122fb-e645-423c-b5e0-459bc7c11e1f Workspace: default / ef66861d-f662-47f7-8fb2-7f60a7f8e8d3

ANNOTATOR: label_studio

Name: label_studio ID: 84ecd3bb-85f3-474b-9f9c-db2ce4ff9856 Type: annotator Flavor: label_studio Configuration: {'authentication_secret': 'LABEL_STUDIO_SECRET', 'instance_url': 'http://localhost', 'port': 8093} Shared: No User: default / f76122fb-e645-423c-b5e0-459bc7c11e1f Workspace: default / ef66861d-f662-47f7-8fb2-7f60a7f8e8d3

What happened?

I've tried to run the example with Label Studio from zenml in version 0.39.1 and I've encountered errors due to missing packages: zenml.artifacts removed in release 0.39.0.

Reproduction steps

  1. Following steps from tutorial v0.39.1 (with GCP)
  2. Run command python run.py --train

Relevant log output

$ python run.py --train
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/user/praca/zenml_playground/label_studio/zenml_examples/label_studio_annotation/run.py:22  │
│ in <module>                                                                                      │
│                                                                                                  │
│   19 from steps.get_labeled_data import get_labeled_data_step                                    │
│   20 from steps.get_or_create_dataset import get_or_create_the_dataset                           │
│   21 from steps.load_image_data_step import LoadImageDataParameters, load_image_data             │
│ ❱ 22 from steps.model_deployers import model_deployer                                            │
│   23 from steps.prediction_steps import (                                                        │
│   24 │   PredictionServiceLoaderParameters,                                                      │
│   25 │   prediction_service_loader,                                                              │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/zenml_examples/label_studio_annotation/steps/mode │
│ l_deployers.py:15 in <module>                                                                    │
│                                                                                                  │
│   12 #  or implied. See the License for the specific language governing                          │
│   13 #  permissions and limitations under the License.                                           │
│   14                                                                                             │
│ ❱ 15 from zenml.artifacts.model_artifact import ModelArtifact                                    │
│   16 from zenml.steps import step                                                                │
│   17                                                                                             │
│   18                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'zenml.artifacts'

Code of Conduct

adamwawrzynski commented 1 year ago

When I've removed model_deployer from label_studio_annotation/pipelines/training_pipeline.py, and the whole file label_studio_annotation/steps/model_deployers.py, and removed import from steps.model_deployers import model_deployer and passing parameter model_deployer=model_deployer, to training_pipeline function call in run.py script I was able to run `python run.py --train without an issue.

Now command python run.py --inference gave me the same error as older versions I've tested, like 0.35.0 and 0.38.0. It looks like there is a problem with matching filenames in data uploaded to Label Studio and model predictions.

$ python run.py --inference
Registered pipeline inference_pipeline (version 1).
Running pipeline inference_pipeline on stack annotation_gcp (caching enabled)
Step get_or_create_dataset has started.
Step get_or_create_dataset has finished in 0.651s.
Step inference_data_loader has started.
Step inference_data_loader has finished in 4.683s.
Step prediction_service_loader has started.
Step prediction_service_loader has finished in 5.926s.
Step predictor has started.
/home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
  warnings.warn(
Step predictor has finished in 1.982s.
Step data_syncer has started.
Failed to run step data_syncer.
Pipeline run inference_pipeline-2023_05_11-13_26_16_462715 failed.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/user/praca/zenml_playground/label_studio/zenml_examples/label_studio_annotation/run.py:82  │
│ in <module>                                                                                      │
│                                                                                                  │
│   79                                                                                             │
│   80                                                                                             │
│   81 if __name__ == "__main__":                                                                  │
│ ❱ 82 │   main()                                                                                  │
│   83                                                                                             │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/click/core.py:1 │
│ 130 in __call__                                                                                  │
│                                                                                                  │
│   1127 │                                                                                         │
│   1128 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1129 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1130 │   │   return self.main(*args, **kwargs)                                                 │
│   1131                                                                                           │
│   1132                                                                                           │
│   1133 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/click/core.py:1 │
│ 055 in main                                                                                      │
│                                                                                                  │
│   1052 │   │   try:                                                                              │
│   1053 │   │   │   try:                                                                          │
│   1054 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                  │
│ ❱ 1055 │   │   │   │   │   rv = self.invoke(ctx)                                                 │
│   1056 │   │   │   │   │   if not standalone_mode:                                               │
│   1057 │   │   │   │   │   │   return rv                                                         │
│   1058 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                               │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/click/core.py:1 │
│ 404 in invoke                                                                                    │
│                                                                                                  │
│   1401 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1402 │   │                                                                                     │
│   1403 │   │   if self.callback is not None:                                                     │
│ ❱ 1404 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1405 │                                                                                         │
│   1406 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1407 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/click/core.py:7 │
│ 60 in invoke                                                                                     │
│                                                                                                  │
│    757 │   │                                                                                     │
│    758 │   │   with augment_usage_errors(__self):                                                │
│    759 │   │   │   with ctx:                                                                     │
│ ❱  760 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    761 │                                                                                         │
│    762 │   def forward(                                                                          │
│    763 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902             │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/zenml_examples/label_studio_annotation/run.py:66  │
│ in main                                                                                          │
│                                                                                                  │
│   63 │   │   │   deployment_trigger=deployment_trigger(),                                        │
│   64 │   │   ).run()                                                                             │
│   65 │   elif pipeline == "inference":                                                           │
│ ❱ 66 │   │   inference_pipeline(                                                                 │
│   67 │   │   │   get_or_create_dataset=get_or_create_the_dataset,                                │
│   68 │   │   │   inference_data_loader=load_image_data(                                          │
│   69 │   │   │   │   params=LoadImageDataParameters(                                             │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/pipelines │
│ /base_pipeline.py:659 in run                                                                     │
│                                                                                                  │
│    656 │   │   │   # unexpected behavior                                                         │
│    657 │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True                            │
│    658 │   │   │   try:                                                                          │
│ ❱  659 │   │   │   │   stack.deploy_pipeline(deployment=deployment_model)                        │
│    660 │   │   │   finally:                                                                      │
│    661 │   │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = False                       │
│    662                                                                                           │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/stack/sta │
│ ck.py:864 in deploy_pipeline                                                                     │
│                                                                                                  │
│    861 │   │   Returns:                                                                          │
│    862 │   │   │   The return value of the call to `orchestrator.run_pipeline(...)`.             │
│    863 │   │   """                                                                               │
│ ❱  864 │   │   return self.orchestrator.run(deployment=deployment, stack=self)                   │
│    865 │                                                                                         │
│    866 │   def _get_active_components_for_step(                                                  │
│    867 │   │   self, step_config: "StepConfiguration"                                            │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/orchestra │
│ tors/base_orchestrator.py:166 in run                                                             │
│                                                                                                  │
│   163 │   │                                                                                      │
│   164 │   │   environment = get_config_environment_vars()                                        │
│   165 │   │   try:                                                                               │
│ ❱ 166 │   │   │   result = self.prepare_or_run_pipeline(                                         │
│   167 │   │   │   │   deployment=deployment, stack=stack, environment=environment                │
│   168 │   │   │   )                                                                              │
│   169 │   │   finally:                                                                           │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/orchestra │
│ tors/local/local_orchestrator.py:82 in prepare_or_run_pipeline                                   │
│                                                                                                  │
│    79 │   │   │   │   │   step.config.name,                                                      │
│    80 │   │   │   │   )                                                                          │
│    81 │   │   │                                                                                  │
│ ❱  82 │   │   │   self.run_step(                                                                 │
│    83 │   │   │   │   step=step,                                                                 │
│    84 │   │   │   )                                                                              │
│    85                                                                                            │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/orchestra │
│ tors/base_orchestrator.py:186 in run_step                                                        │
│                                                                                                  │
│   183 │   │   │   step=step,                                                                     │
│   184 │   │   │   orchestrator_run_id=self.get_orchestrator_run_id(),                            │
│   185 │   │   )                                                                                  │
│ ❱ 186 │   │   launcher.launch()                                                                  │
│   187 │                                                                                          │
│   188 │   @staticmethod                                                                          │
│   189 │   def requires_resources_in_orchestration_environment(                                   │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/orchestra │
│ tors/step_launcher.py:197 in launch                                                              │
│                                                                                                  │
│   194 │   │   │                                                                                  │
│   195 │   │   │   if execution_needed:                                                           │
│   196 │   │   │   │   try:                                                                       │
│ ❱ 197 │   │   │   │   │   self._run_step(                                                        │
│   198 │   │   │   │   │   │   pipeline_run=pipeline_run,                                         │
│   199 │   │   │   │   │   │   step_run=step_run_response,                                        │
│   200 │   │   │   │   │   )                                                                      │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/orchestra │
│ tors/step_launcher.py:375 in _run_step                                                           │
│                                                                                                  │
│   372 │   │   │   │   │   step_run_info=step_run_info,                                           │
│   373 │   │   │   │   )                                                                          │
│   374 │   │   │   else:                                                                          │
│ ❱ 375 │   │   │   │   self._run_step_without_step_operator(                                      │
│   376 │   │   │   │   │   step_run_info=step_run_info,                                           │
│   377 │   │   │   │   │   input_artifacts=step_run.input_artifacts,                              │
│   378 │   │   │   │   │   output_artifact_uris=output_artifact_uris,                             │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/orchestra │
│ tors/step_launcher.py:445 in _run_step_without_step_operator                                     │
│                                                                                                  │
│   442 │   │   │   output_artifact_uris: The output artifact URIs of the current step.            │
│   443 │   │   """                                                                                │
│   444 │   │   runner = StepRunner(step=self._step, stack=self._stack)                            │
│ ❱ 445 │   │   runner.run(                                                                        │
│   446 │   │   │   input_artifacts=input_artifacts,                                               │
│   447 │   │   │   output_artifact_uris=output_artifact_uris,                                     │
│   448 │   │   │   step_run_info=step_run_info,                                                   │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/orchestra │
│ tors/step_runner.py:133 in run                                                                   │
│                                                                                                  │
│   130 │   │   │   self._stack.prepare_step_run(info=step_run_info)                               │
│   131 │   │   │   step_failed = False                                                            │
│   132 │   │   │   try:                                                                           │
│ ❱ 133 │   │   │   │   return_values = step_entrypoint(**function_params)                         │
│   134 │   │   │   except BaseException as step_exception:  # noqa: E722                          │
│   135 │   │   │   │   step_failed = True                                                         │
│   136 │   │   │   │   failure_hook_source = self.configuration.failure_hook_source               │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/integrati │
│ ons/label_studio/steps/label_studio_standard_steps.py:240 in sync_new_data_to_label_studio       │
│                                                                                                  │
│   237 │   │   │   filename_reference = TASK_TO_FILENAME_REFERENCE_MAPPING[                       │
│   238 │   │   │   │   params.label_config_type                                                   │
│   239 │   │   │   ]                                                                              │
│ ❱ 240 │   │   │   preds_with_task_ids = convert_pred_filenames_to_task_ids(                      │
│   241 │   │   │   │   predictions,                                                               │
│   242 │   │   │   │   dataset.tasks,                                                             │
│   243 │   │   │   │   filename_reference,                                                        │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/integrati │
│ ons/label_studio/label_studio_utils.py:51 in convert_pred_filenames_to_task_ids                  │
│                                                                                                  │
│    48 │   │   │   {"filename": quote(pred["filename"]), "result": pred["result"]}                │
│    49 │   │   │   for pred in preds                                                              │
│    50 │   │   ]                                                                                  │
│ ❱  51 │   return [                                                                               │
│    52 │   │   {                                                                                  │
│    53 │   │   │   "task": int(                                                                   │
│    54 │   │   │   │   filename_id_mapping[os.path.basename(pred["filename"])]                    │
│                                                                                                  │
│ /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/integrati │
│ ons/label_studio/label_studio_utils.py:54 in <listcomp>                                          │
│                                                                                                  │
│    51 │   return [                                                                               │
│    52 │   │   {                                                                                  │
│    53 │   │   │   "task": int(                                                                   │
│ ❱  54 │   │   │   │   filename_id_mapping[os.path.basename(pred["filename"])]                    │
│    55 │   │   │   ),                                                                             │
│    56 │   │   │   "result": pred["result"],                                                      │
│    57 │   │   }                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'ARIA_IMG_1481.jpeg'
adamwawrzynski commented 1 year ago

And here are printed values from /home/user/praca/zenml_playground/label_studio/.venv/lib/python3.8/site-packages/zenml/integrations/label_studio/label_studio_utils.py file (I've printed only 2 objects of list tasks to make output shorter):

filename_id_mapping: {'image_file.JPEG': 160}
tasks: [{'id': 71, 'predictions': [], 'annotations': [{'id': 1, 'created_username': ' user@gmail.com, 1', 'created_ago': '35\xa0minutes', 'completed_by': 1, 'result': [{'value': {'choices': ['aria']}, 'id': '94W5CCD9HP', 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}], 'was_cancelled': False, 'ground_truth': False, 'created_at': '2023-05-11T13:00:11.592787Z', 'updated_at': '2023-05-11T13:00:15.675992Z', 'lead_time': 6.492, 'task': 71, 'parent_prediction': None, 'parent_annotation': None}], 'drafts': [], 'annotators': [1], 'inner_id': 1, 'cancelled_annotations': 0, 'total_annotations': 1, 'total_predictions': 0, 'completed_at': '2023-05-11T13:00:11.592787Z', 'annotations_results': '[{value: {choices: [aria]}, id: 94W5CCD9HP, from_name: choice, to_name: image, type: choices, origin: manual}]', 'predictions_results': '', 'predictions_score': None, 'file_upload': None, 'storage_filename': 'load_image_data/images/c43524a8-361f-4fd9-8b33-09b0a52e950c/1/0/image_file.JPEG', 'annotations_ids': '1', 'predictions_model_versions': '', 'avg_lead_time': 6.492, 'updated_by': [{'user_id': 1}], 'data': {'image': 'https://storage.googleapis.com/label_studio_example/load_image_data/images/c43524a8-361f-4fd9-8b33-09b0a52e950c/1/0/image_file.JPEG?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=label-studio-service%40adam-wawrzynski-sandbox.iam.gserviceaccount.com%2F20230511%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230511T133604Z&X-Goog-Expires=60&X-Goog-SignedHeaders=host&X-Goog-Signature=6c5a8ba89fd8e12a078f0a98e5f4aca18f4e33f498f2225f6cda3e6db95bbc8e4c898f54abb1e6a618b15de82dd894192fb16d05a65cbe6a4e059d06e3af70a4c957cf7b2f47d23c28eb6239700a75c4d7e22474ad9a1797b4bd3d6723c38042f9aff3a8a4fe48b49bb18c533160dd144bbf62ecfe3c46ae3dab0c608645a24837a8688cabc3ea393bdca409c9f42f92a9931682e1d2f5b932aeb6195a8c169ee71f02bdee22f73a74d18729b35cf49f1ca22ba0733e27330a7f2c44b2d88418cdafd5f98d0f23be50348ca4b99a26940c1b5a662243e161cc2f883c72b1a11cd31ca3f6789627e62414d604c6f36e4a1745dc2507ab857562f8361c3c1296e2'}, 'meta': {}, 'created_at': '2023-05-11T12:50:26.153471Z', 'updated_at': '2023-05-11T13:00:15.704163Z', 'is_labeled': True, 'overlap': 1, 'comment_count': 0, 'unresolved_comment_count': 0, 'last_comment_updated_at': None, 'project': 2, 'comment_authors': []}, {'id': 72, 'predictions': [], 'annotations': [{'id': 2, 'created_username': ' user@gmail.com, 1', 'created_ago': '35\xa0minutes', 'completed_by': 1, 'result': [{'value': {'choices': ['not_aria']}, 'id': 'TH2vHJ37HH', 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}], 'was_cancelled': False, 'ground_truth': False, 'created_at': '2023-05-11T13:00:19.696569Z', 'updated_at': '2023-05-11T13:00:19.696596Z', 'lead_time': 2.349, 'task': 72, 'parent_prediction': None, 'parent_annotation': None}], 'drafts': [], 'annotators': [1], 'inner_id': 2, 'cancelled_annotations': 0, 'total_annotations': 1, 'total_predictions': 0, 'completed_at': '2023-05-11T13:00:19.696569Z', 'annotations_results': '[{value: {choices: [not_aria]}, id: TH2vHJ37HH, from_name: choice, to_name: image, type: choices, origin: manual}]', 'predictions_results': '', 'predictions_score': None, 'file_upload': None, 'storage_filename': 'load_image_data/images/c43524a8-361f-4fd9-8b33-09b0a52e950c/1/1/image_file.JPEG', 'annotations_ids': '2', 'predictions_model_versions': '', 'avg_lead_time': 2.349, 'updated_by': [{'user_id': 1}], 'data': {'image': 'https://storage.googleapis.com/label_studio_example/load_image_data/images/c43524a8-361f-4fd9-8b33-09b0a52e950c/1/1/image_file.JPEG?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=label-studio-service%40adam-wawrzynski-sandbox.iam.gserviceaccount.com%2F20230511%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230511T133604Z&X-Goog-Expires=60&X-Goog-SignedHeaders=host&X-Goog-Signature=484cab67f5b8cf89db47a2147353f7cd34e9b7060fe360aa19c782552eb40484b686624531897df24391bc94fc5adcb831beafdc9870acba66ec180e14a5aa92c42deda43014aa0cf6b2558e98fd121d71a9be986e34f3e27b226846b9788997f76eaef1f1b5925fb0aa527e35eb813d14f6c855f1a7600023c82f2837ba579ee8028cabb8402d9d2b982c090ccf77e57d69f25c3a9ec52f80ecf40d96a6dab928614b070dbe0b608242e0b7b201fe4636118e9cc9a2a0769e15c8d49d0d7cc0e869c0be5b41f3cf1500b293081779556b696769de2d5dfd427d37f1d54f6c889953a634e28829fbcf3717869a32b87beee00e665510bf03e516bde036ee6bcc'}, 'meta': {}, 'created_at': '2023-05-11T12:50:26.159904Z', 'updated_at': '2023-05-11T13:00:19.721768Z', 'is_labeled': True, 'overlap': 1, 'comment_count': 0, 'unresolved_comment_count': 0, 'last_comment_updated_at': None, 'project': 2, 'comment_authors': []}]
preds: [{'filename': 'ARIA_IMG_1481.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': 'IMG_2640.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': 'IMG_1892.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': '2dc5fe08-152d-47d0-bdd4-f70ddbfb6486.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': '7f5d8d61-c507-434f-9d4b-de28f93dad1e.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': 'ARIA_95ffaa51-3894-4773-a8c6-a4e7d09de050.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': 'IMG_2262.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': 'ARIA_IMG_0584.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': 'ARIA_IMG_0728.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}, {'filename': 'ARIA_IMG_0688.jpeg', 'result': [{'value': {'choices': ['not_aria']}, 'from_name': 'choice', 'to_name': 'image', 'type': 'choices', 'origin': 'manual'}]}]
storage_type: gcs
adamwawrzynski commented 1 year ago

It works in version 0.23.0. I don't have much time to test other versions and pinpoint which changes broke the example but I will analyze that in the next week.

fa9r commented 1 year ago

@adamwawrzynski Thanks a lot for reporting this, I just fixed the artifacts import issue in #1534.

Background: zenml.artifacts was deprecated long ago and finally deleted in the last release. Apparently, the example you were running was outdated. Sorry for the inconvenience.

strickvl commented 1 year ago

I'll take a look at the parts that were unrelated to the zenml.artifacts issue to address what remains from your reports.

strickvl commented 1 year ago

@adamwawrzynski I've successfully replicated the error you're facing when you run the inference pipeline the first time. I haven't quite identified the cause of the bug (a change in the ZenML materialization, or maybe in the Label Studio Python SDK?) but I will work on a fix and let you know when I have a way for you to move forward with this.

adamwawrzynski commented 1 year ago

I don't know if this problem is related to the one above but I've tried to run this fixed example and encountered different problems. I've installed develop branch using this command pip install git+https://github.com/zenml-io/zenml.git@develop --upgrade, then followed Label Studio example. zenml up starts server but at given port in the browser I've got only {"detail":"Not Found"}.

strickvl commented 1 year ago

Hi @adamwawrzynski. I went through and gave the integration / example a full review (testing on all three major cloud providers) and updated / fixed what needs fixing. https://github.com/zenml-io/zenml/pull/1542 still needs reviews on our side, but anticipating this being part of the next release, with everything working again. Thanks for being patient.

adamwawrzynski commented 1 year ago

@strickvl I really appreciate your quick response and your work to fix this problem and looking forward to have it working at the main branch.

strickvl commented 1 year ago

@adamwawrzynski it'll be out in the next release, so next week.

strickvl commented 1 year ago

Merged this into develop just now, so it'll be in the coming release. Thanks for taking the time to run it and sorry it wasn't working!