search

google / ml-metadata

For recording and retrieving metadata associated with ML developer and data scientist workflows.
https://www.tensorflow.org/tfx/guide/mlmd
Apache License 2.0
616 stars 145 forks source link

mysql setup for ml-metadata #149

Closed CandiedCode closed 2 years ago

CandiedCode commented 2 years ago

Hello,

I'm hoping someone can provide some guidance. I've tried to setup a quick example of connecting ml-metadata to a mysql database running locally.

I've tried on my mac as well as via docker containers incase it was a mac/mysql c connector issue. Both cases I get the same error message.

Traceback (most recent call last):
  File "/code/ml_metadata_playground/metadata.py", line 44, in <module>
    store = connect_mysql()
  File "/code/ml_metadata_playground/metadata.py", line 17, in connect_mysql
    return metadata_store.MetadataStore(connection_config)
  File "/usr/local/lib/python3.9/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 103, in __init__
    self._metadata_store = metadata_store_serialized.CreateMetadataStore(
RuntimeError: mysql_real_connect failed: errno: , error:

For my mac: I'm using python3.9 nightly build

The full error via this pytest

self = <ml_metadata.metadata_store.metadata_store.MetadataStore object at 0x7f980ad70be0>
config = mysql {
  host: "0.0.0.0"
  port: 3306
  database: "metadata"
  user: "test"
  password: "secret"
}
, enable_upgrade_migration = True

    def __init__(self,
                 config: Union[proto.ConnectionConfig,
                               proto.MetadataStoreClientConfig],
                 enable_upgrade_migration: bool = False):
      """Initialize the MetadataStore.

      MetadataStore can directly connect to either the metadata database or
      the MLMD MetadataStore gRPC server.

      Args:
        config: `proto.ConnectionConfig` or `proto.MetadataStoreClientConfig`.
          Configuration to connect to the database or the metadata store server.
        enable_upgrade_migration: if set to True, the library upgrades the db
          schema and migrates all data if it connects to an old version backend.
          It is ignored when using gRPC `proto.MetadataStoreClientConfig`.
      """
      self._max_num_retries = 5
      if isinstance(config, proto.ConnectionConfig):
        self._using_db_connection = True
        migration_options = metadata_store_pb2.MigrationOptions()
        migration_options.enable_upgrade_migration = enable_upgrade_migration
>       self._metadata_store = metadata_store_serialized.CreateMetadataStore(
            config.SerializeToString(), migration_options.SerializeToString())
E       RuntimeError: mysql_real_connect failed: errno: , error:

I've tried to quickly create a dummy repo to share what my images look like via docker-compose via https://github.com/CandiedCode/ml-metadata-playground/blob/mysql_test/ml_metadata_playground/metadata.py#L42 https://github.com/CandiedCode/ml-metadata-playground/blob/mysql_test/docker-compose.yaml

docker compose run meta
[+] Running 1/0
 ⠿ Container ml-metadata-playground-db-1  Run...                                         0.0s
Traceback (most recent call last):
  File "/code/ml_metadata_playground/metadata.py", line 44, in <module>
    store = connect_mysql()
  File "/code/ml_metadata_playground/metadata.py", line 17, in connect_mysql
    return metadata_store.MetadataStore(connection_config)
  File "/usr/local/lib/python3.9/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 103, in __init__
    self._metadata_store = metadata_store_serialized.CreateMetadataStore(
RuntimeError: mysql_real_connect failed: errno: , error:

Not sure if this is an issue with the nightly 3.9 build or my local MySQL c driver setup.

CandiedCode commented 2 years ago

created https://github.com/google/ml-metadata/pull/150 to test local wheel package for python 3.9

CandiedCode commented 2 years ago

Full python 3.9 support wasn't part of pybindings until v2.6.0rc1

Changing bazel to use 2.6.0 for pybind11 and running docker compose run manylinux-python39


INFO: Elapsed time: 1387.665s, Critical Path: 732.63s
INFO: 1777 processes: 58 internal, 1719 processwrapper-sandbox.
INFO: Build completed successfully, 1777 total actions
INFO: Build completed successfully, 1777 total actions
+ local bazel_genfiles
+ [[ -d /build/bazel-genfiles ]]
+ bazel_genfiles=bazel-bin
+ cp -f /build/bazel-bin/ml_metadata/proto/metadata_store_pb2.py /build/ml_metadata/proto
+ cp -f /build/bazel-bin/ml_metadata/proto/metadata_store_service_pb2.py /build/ml_metadata/proto
+ cp -f /build/bazel-bin/ml_metadata/proto/metadata_store_service_pb2_grpc.py /build/ml_metadata/proto
+ cp -f /build/bazel-bin/ml_metadata/simple_types/proto/simple_types_pb2.py /build/ml_metadata/simple_types/proto
+ _is_windows
++ uname -s
++ tr A-Z a-z
+ [[ linux =~ (cygwin|mingw32|mingw64|msys)_nt* ]]
+ MLMD_EXTENSION=ml_metadata/metadata_store/pywrap/metadata_store_extension.so
+ cp -f /build/bazel-bin/ml_metadata/metadata_store/pywrap/metadata_store_extension.so /build/ml_metadata/metadata_store/pywrap/metadata_store_extension.so
+ chmod +w /build/ml_metadata/metadata_store/pywrap/metadata_store_extension.so
running build_py
copying ml_metadata/proto/metadata_store_pb2.py -> build/lib/ml_metadata/proto
copying ml_metadata/proto/metadata_store_service_pb2.py -> build/lib/ml_metadata/proto
copying ml_metadata/proto/metadata_store_service_pb2_grpc.py -> build/lib/ml_metadata/proto
copying ml_metadata/simple_types/proto/simple_types_pb2.py -> build/lib/ml_metadata/simple_types/proto
running egg_info
writing ml_metadata.egg-info/PKG-INFO
writing dependency_links to ml_metadata.egg-info/dependency_links.txt
writing namespace_packages to ml_metadata.egg-info/namespace_packages.txt
writing requirements to ml_metadata.egg-info/requires.txt
writing top-level names to ml_metadata.egg-info/top_level.txt
reading manifest file 'ml_metadata.egg-info/SOURCES.txt'
writing manifest file 'ml_metadata.egg-info/SOURCES.txt'
copying ml_metadata/metadata_store/pywrap/metadata_store_extension.so -> build/lib/ml_metadata/metadata_store/pywrap
running build_ext
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/ml_metadata
copying build/lib/ml_metadata/__init__.py -> build/bdist.linux-x86_64/wheel/ml_metadata
copying build/lib/ml_metadata/errors.py -> build/bdist.linux-x86_64/wheel/ml_metadata
creating build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store
copying build/lib/ml_metadata/metadata_store/__init__.py -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store
copying build/lib/ml_metadata/metadata_store/metadata_store.py -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store
copying build/lib/ml_metadata/metadata_store/metadata_store_test.py -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store
copying build/lib/ml_metadata/metadata_store/mlmd_types.py -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store
copying build/lib/ml_metadata/metadata_store/mlmd_types_test.py -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store
creating build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store/pywrap
copying build/lib/ml_metadata/metadata_store/pywrap/__init__.py -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store/pywrap
copying build/lib/ml_metadata/metadata_store/pywrap/metadata_store_extension.so -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store/pywrap
copying build/lib/ml_metadata/metadata_store/types.py -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store
copying build/lib/ml_metadata/metadata_store/types_test.py -> build/bdist.linux-x86_64/wheel/ml_metadata/metadata_store
creating build/bdist.linux-x86_64/wheel/ml_metadata/proto
copying build/lib/ml_metadata/proto/__init__.py -> build/bdist.linux-x86_64/wheel/ml_metadata/proto
copying build/lib/ml_metadata/proto/metadata_store_pb2.py -> build/bdist.linux-x86_64/wheel/ml_metadata/proto
copying build/lib/ml_metadata/proto/metadata_store_service_pb2.py -> build/bdist.linux-x86_64/wheel/ml_metadata/proto
copying build/lib/ml_metadata/proto/metadata_store_service_pb2_grpc.py -> build/bdist.linux-x86_64/wheel/ml_metadata/proto
creating build/bdist.linux-x86_64/wheel/ml_metadata/simple_types
copying build/lib/ml_metadata/simple_types/__init__.py -> build/bdist.linux-x86_64/wheel/ml_metadata/simple_types
creating build/bdist.linux-x86_64/wheel/ml_metadata/simple_types/proto
copying build/lib/ml_metadata/simple_types/proto/__init__.py -> build/bdist.linux-x86_64/wheel/ml_metadata/simple_types/proto
copying build/lib/ml_metadata/simple_types/proto/simple_types_pb2.py -> build/bdist.linux-x86_64/wheel/ml_metadata/simple_types/proto
copying build/lib/ml_metadata/version.py -> build/bdist.linux-x86_64/wheel/ml_metadata
running install_egg_info
Copying ml_metadata.egg-info to build/bdist.linux-x86_64/wheel/ml_metadata-1.8.0.dev0-py3.9.egg-info
running install_scripts
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
creating build/bdist.linux-x86_64/wheel/ml_metadata-1.8.0.dev0.dist-info/WHEEL
creating 'dist/ml_metadata-1.8.0.dev0-cp39-cp39-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'ml_metadata/__init__.py'
adding 'ml_metadata/errors.py'
adding 'ml_metadata/version.py'
adding 'ml_metadata/metadata_store/__init__.py'
adding 'ml_metadata/metadata_store/metadata_store.py'
adding 'ml_metadata/metadata_store/metadata_store_test.py'
adding 'ml_metadata/metadata_store/mlmd_types.py'
adding 'ml_metadata/metadata_store/mlmd_types_test.py'
adding 'ml_metadata/metadata_store/types.py'
adding 'ml_metadata/metadata_store/types_test.py'
adding 'ml_metadata/metadata_store/pywrap/__init__.py'
adding 'ml_metadata/metadata_store/pywrap/metadata_store_extension.so'
adding 'ml_metadata/proto/__init__.py'
adding 'ml_metadata/proto/metadata_store_pb2.py'
adding 'ml_metadata/proto/metadata_store_service_pb2.py'
adding 'ml_metadata/proto/metadata_store_service_pb2_grpc.py'
adding 'ml_metadata/simple_types/__init__.py'
adding 'ml_metadata/simple_types/proto/__init__.py'
adding 'ml_metadata/simple_types/proto/simple_types_pb2.py'
adding 'ml_metadata-1.8.0.dev0.dist-info/LICENSE'
adding 'ml_metadata-1.8.0.dev0.dist-info/METADATA'
adding 'ml_metadata-1.8.0.dev0.dist-info/WHEEL'
adding 'ml_metadata-1.8.0.dev0.dist-info/namespace_packages.txt'
adding 'ml_metadata-1.8.0.dev0.dist-info/top_level.txt'
adding 'ml_metadata-1.8.0.dev0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
+ stamp_wheel
++ ls /build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-linux_x86_64.whl /build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
+ WHEEL_PATH='/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-linux_x86_64.whl
/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl'
++ dirname '/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-linux_x86_64.whl
/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl'
+ WHEEL_DIR='/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-linux_x86_64.whl
/build/dist'
+ auditwheel repair --plat manylinux2010_x86_64 -w '/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-linux_x86_64.whl
/build/dist' '/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-linux_x86_64.whl
/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl'
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: cannot access /build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-linux_x86_64.whl
/build/dist/ml_metadata-1.8.0.dev0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl. No such file
CandiedCode commented 2 years ago

I downgraded to python 3.8 via https://github.com/CandiedCode/ml-metadata-playground/tree/mysql_test and running docker compose build meta && docker compose run meta

still produces the same error

Traceback (most recent call last): File "ml_metadata_playground/metadata.py", line 44, in store = connect_mysql() File "ml_metadata_playground/metadata.py", line 17, in connect_mysql return metadata_store.MetadataStore(connection_config) File "/usr/local/lib/python3.8/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 103, in init self._metadata_store = metadata_store_serialized.CreateMetadataStore( RuntimeError: mysql_real_connect failed: errno: , error:

BrianSong commented 2 years ago

Are you using Azure MySQL?

Also, could you try "127.0.0.1" instead of "0.0.0.0" for your local host?

CandiedCode commented 2 years ago

I'm just running a mysql docker container in my same network via docker-compose.

I've tried both 0.0.0.0:3306 when running python locally and then also 'db:3306' when running inside docker compose.

def connect_mysql() -> metadata_store.MetadataStore:
    connection_config = metadata_store_pb2.ConnectionConfig()
    connection_config.mysql.host = '0.0.0.0' # use 0.0.0.0 when running python file locally, or 'db' when running via docker-compose
    connection_config.mysql.port = 3306
    connection_config.mysql.database = 'metadata'
    connection_config.mysql.user = 'test'
    connection_config.mysql.password = 'secret'
    return metadata_store.MetadataStore(connection_config)
version: "3.9"
services:
  meta:
    build: .
    depends_on:
      - db
    links:
      - db
  db:
    image: mysql:8.0.28
    restart: always
    environment:
      MYSQL_DATABASE: 'metadata'
      MYSQL_USER: 'test'
      MYSQL_PASSWORD: 'secret'
      MYSQL_ROOT_PASSWORD: 'password'
    ports:
      - '3306:3306'
    expose:
      - '3306'
    volumes:
      - my-db:/var/lib/mysql
volumes:
  my-db:

libraries install in dockerfile

RUN apt-get update && apt-get install -y --no-install-recommends \
    python3.8 python3.8-dev python3-pip libssl-dev libmysqlclient-dev mysql-client

via local shell

mysql --host=0.0.0.0 --port=3306 --user=test --password=secret  metadata
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.28 MySQL Community Server - GPL

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

inside the docker container, i use the service name since '0.0.0.0' doesn't work

# inside the meta docker container
mysql --host=db --port=3306 --user=test --password=secret  metadata
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 13
Server version: 8.0.28 MySQL Community Server - GPL

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

Since the error message isn't descriptive, I feel like I'm missing something in my dockerfile for the python script for running the mysql connect c library. 🤷🏽‍♀️ maybe.

GerardoGR commented 2 years ago

Hey @CandiedCode, I had a similar issue in a local deployment of kubeflow that uses mysql:8.0.29, I think the error you are seeing:

RuntimeError: mysql_real_connect failed: errno: , error:

is due to https://github.com/google/ml-metadata/issues/128: by default mysql 8 uses caching_sha2_password . To work around that in your example, you can set the the default-authentication-plugin as a CLI parameter in your docker-compose, e.g:

  db:                                                                                                                                                                        
    image: mysql:8.0.28                                                                                                                                                      
    restart: always                                                                                                                                                          
    # Taken from docker hub: https://hub.docker.com/_/mysql/                                                                                                                 
    command: --default-authentication-plugin=mysql_native_password                                                                                                           
    environment:                                                                                                                                                             
      MYSQL_DATABASE: 'metadata'                                                                                                                                             
      MYSQL_USER: 'test'                                                                                                                                                     
      MYSQL_PASSWORD: 'secret'                                                                                                                                               
      MYSQL_ROOT_PASSWORD: 'password'                                                                                                                                        
    ports:                                                                                                                                                                   
      - '3306:3306'                                                                                                                                                          
    expose:                                                                                                                                                                  
      - '3306'                                                                                                                                                               
    volumes:                                                                                                                                                                 
      - my-db:/var/lib/mysql
CandiedCode commented 2 years ago

Thanks @GerardoGR

That did solve my issue.