kubeflow / metadata

Repository for assets related to Metadata.
Apache License 2.0
121 stars 67 forks source link

Metadata Workspace names can clash based on case #226

Closed nlarusstone closed 3 years ago

nlarusstone commented 4 years ago

/kind bug

What steps did you take and what happened: I created a workspace called "test":

metadata.Workspace(name='test', store=store)

I then tried to create a workspace called "TEST":

metadata.Workspace(name='TEST', store=store, reuse_workspace_if_exists=True)

I get the following error:

tensorflow.python.framework.errors_impl.AlreadyExistsError: Given node already exists: type_id: 5
Internal: mysql_query failed: errno: 1062, error: Duplicate entry '5-MLINFRA' for key 'type_id'

What did you expect to happen: I expected to be returned either a new workspace or (preferably) the original workspace I created

Anything else you would like to add: I think the issue comes from this line: https://github.com/kubeflow/metadata/blob/master/sdk/python/kubeflow/metadata/metadata.py#L222

Kubeflow metadata checks for literal string equality (is case-sensitive), while it seems like the backing mysql DB is case-insensitive

I think this is likely to occur for other objects, but I have not confirmed.

Environment:

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
bug 0.98

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

konradr commented 4 years ago

Hi, I think I'm hitting the same issue trying to run the tfx example: https://github.com/tensorflow/tfx/blob/master/tfx/examples/chicago_taxi_pipeline/taxi_pipeline_kubeflow_local.py

INFO:absl:Running driver for CsvExampleGen
INFO:absl:MetadataStore with gRPC connection initialized
INFO:absl:select span = None
Traceback (most recent call last):
  File "/opt/venv/lib/python3.6/site-packages/ml_metadata/metadata_store/metadata_store.py", line 163, in _call_method
    response.CopyFrom(grpc_method(request))
  File "/opt/venv/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/venv/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.ALREADY_EXISTS
    details = "Given node already exists: type_id: 5
name: "chicago_taxi_pipeline_kubeflow_local"
properties {
  key: "pipeline_name"
  value {
    string_value: "chicago_taxi_pipeline_kubeflow_local"
  }
}
Internal: mysql_query failed: errno: 1062, error: Duplicate entry '5-chicago_taxi_pipeline_kubeflow_local' for key 'type_id'"
    debug_error_string = "{"created":"@1598001336.385728324","description":"Error received from peer ipv4:10.152.183.26:8080","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Given node already exists: type_id: 5\nname: "chicago_taxi_pipeline_kubeflow_local"\nproperties {\n  key: "pipeline_name"\n  value {\n    string_value: "chicago_taxi_pipeline_kubeflow_local"\n  }\n}\nInternal: mysql_query failed: errno: 1062, error: Duplicate entry '5-chicago_taxi_pipeline_kubeflow_local' for key 'type_id'","grpc_status":6}"

Im running latest microk8s kubeflow deployment:

microk8s juju show-model kubeflow
kubeflow:
  name: admin/kubeflow
  short-name: kubeflow
  model-uuid: d5ee14c7-9766-4813-81d9-41d7ad1c3ce7
  model-type: caas
  controller-uuid: 93880e47-bf14-4557-86f0-d3c41030ae96
  controller-name: uk8s
  is-controller: false
  owner: admin
  cloud: microk8s
  region: localhost
  type: kubernetes
  life: alive
  status:
    current: available
    since: "2020-08-04"
  users:
    admin:
      display-name: admin
      access: admin
      last-connection: "2020-08-18"
  sla: unsupported
  agent-version: 2.7.6
nlarusstone commented 4 years ago

@konradr this project seems to be unsupported, so we've had to monkeypatch it to get the functionality we want