Metastore package should not require installation of ML frameworks such tensorflow

[apryiomka]

/kind feature

Describe the solution you'd like Currently a lot of packages are being installed whether requested or not. For instance, we do not use tensorflow and work with sckit-learn instead, but the image gets bloated with tensorflow packages and also tensorflow is loaded automatically when trying to log metrics

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/zillow/rent_zestimate/train.py", line 12, in <module>
    from kubeflow.metadata import metadata
  File "/usr/local/lib/python3.6/site-packages/kubeflow/metadata/metadata.py", line 21, in <module>
    from ml_metadata.metadata_store import metadata_store
  File "/usr/local/lib/python3.6/site-packages/ml_metadata/metadata_store/__init__.py", line 15, in <module>
    from ml_metadata.metadata_store.metadata_store import MetadataStore
  File "/usr/local/lib/python3.6/site-packages/ml_metadata/metadata_store/metadata_store.py", line 36, in <module>
    from tensorflow.python.framework import errors
  File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 101, in <module>
    from tensorflow_core import *
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/__init__.py", line 46, in <module>
    from . _api.v2 import compat
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/_api/v2/compat/__init__.py", line 39, in <module>
    from . import v1
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/_api/v2/compat/v1/__init__.py", line 32, in <module>
    from . import compat
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/_api/v2/compat/v1/compat/__init__.py", line 39, in <module>
    from . import v1
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/_api/v2/compat/v1/compat/v1/__init__.py", line 29, in <module>
    from tensorflow._api.v2.compat.v1 import app
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/_api/v2/compat/__init__.py", line 39, in <module>
    from . import v1
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/_api/v2/compat/v1/__init__.py", line 32, in <module>
    from . import compat
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/_api/v2/compat/v1/compat/__init__.py", line 39, in <module>
    from . import v1
  File "/usr/local/lib/python3.6/site-packages/tensorflow_core/_api/v2/compat/v1/compat/v1/__init__.py", line 667, in <module>
    from tensorflow_estimator.python.estimator.api._v1 import estimator
  File "/usr/local/lib/python3.6/site-packages/tensorflow_estimator/__init__.py", line 10, in <module>
    from tensorflow_estimator._api.v1 import estimator
  File "/usr/local/lib/python3.6/site-packages/tensorflow_estimator/_api/v1/estimator/__init__.py", line 10, in <module>
    from tensorflow_estimator._api.v1.estimator import experimental
  File "/usr/local/lib/python3.6/site-packages/tensorflow_estimator/_api/v1/estimator/experimental/__init__.py", line 10, in <module>
    from tensorflow_estimator.python.estimator.canned.dnn import dnn_logit_fn_builder
  File "/usr/local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 33, in <module>
    from tensorflow_estimator.python.estimator import estimator
  File "/usr/local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 53, in <module>
    from tensorflow_estimator.python.estimator import util as estimator_util
  File "/usr/local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/util.py", line 75, in <module>
    class _DatasetInitializerHook(tf.compat.v1.train.SessionRunHook):
AttributeError: module 'tensorflow' has no attribute 'compat'

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.93

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[akirillov]

It seems that parts of TensorFlow are used in google/ml-metadata which is a dependency of kubeflow/metadata. The use-case is not very clear to me but it looks like various TensorFlow-specific domain objects are used in the RPC calls between SDK and Metadata Service.

I'm wondering if there's a temporary workaround for this? We're using PyTorch and MXNet and having TensorFlow installed as Metadata SDK dependency impacts our image sizes significantly.

[akirillov]

Here's a relevant issue in google/ml-metadata repo with a PR open: https://github.com/google/ml-metadata/issues/25. Once landed, the problem with TensorFlow dependency in Metadata SDK should be resolved after upgrading this dependency.

[sbakiu]

Kind reminder, there is a new release in google/ml-metadata/releases that solves this issue

[ca-scribner]

Is there a plan to move to ml-metadata>=0.23.0? Current version of 0.21.0 forces an install of tensorflow anywhere someone wants to use kubeflow-metadata, which is undesirable and unintuitive