Containerized registry server fails on startup due to a ModuleNotFoundError

tchughesiv commented 3 weeks ago

Expected Behavior

Registry server should run without issue when using the feastdev/feature-server:0.41.3 container image.

Current Behavior

In the latest feast release image, registry server errors on startup and the container stops. The following error is thrown - ModuleNotFoundError: No module named 'grpc_health'

Steps to reproduce

$ podman run -ti -e FEATURE_STORE_YAML_BASE64="cHJvamVjdDogdGVzdApwcm92aWRlcjogbG9jYWwKcmVnaXN0cnk6CiAgICBwYXRoOiAvdG1wL3JlZ2lzdHJ5LmRiCiAgICByZWdpc3RyeV90eXBlOiBmaWxlCmVudGl0eV9rZXlfc2VyaWFsaXphdGlvbl92ZXJzaW9uOiAzCg==" feastdev/feature-server:0.41.3 feast serve_registry

Received base64 encoded feature_store.yaml
Traceback (most recent call last):
  File "/usr/local/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/feast/cli.py", line 1043, in serve_registry_command
    store.serve_registry(port)
  File "/usr/local/lib/python3.9/dist-packages/feast/feature_store.py", line 1954, in serve_registry
    from feast import registry_server
  File "/usr/local/lib/python3.9/dist-packages/feast/registry_server.py", line 7, in <module>
    from grpc_health.v1 import health, health_pb2, health_pb2_grpc
ModuleNotFoundError: No module named 'grpc_health'

This appears to only impact the registry server... whereas online and offline startup fine. Its also worth noting that this bug doesn't present itself when running in the cli directly (no container).

Specifications

Version: 0.41.3
Platform: Any container runtime

Possible Solution

Possibly related to / caused by https://github.com/feast-dev/feast/pull/4421

ckavili commented 2 weeks ago

came here to raise the same issue for the latest version of the image but for the missing psycopg2 module.

Received base64 encoded feature_store.yaml
Traceback (most recent call last):
  File "/usr/local/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/feast/cli.py", line 182, in ui
    store = create_feature_store(ctx)
  File "/usr/local/lib/python3.9/dist-packages/feast/repo_operations.py", line 383, in create_feature_store
    return FeatureStore(repo_path=str(repo_path.resolve()))
  File "/usr/local/lib/python3.9/dist-packages/feast/feature_store.py", line 151, in __init__
    self._registry = SqlRegistry(registry_config, self.config.project, None)
  File "/usr/local/lib/python3.9/dist-packages/feast/infra/registry/sql.py", line 256, in __init__
    self.write_engine: Engine = create_engine(
  File "<string>", line 2, in create_engine
  File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
  File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/create.py", line 599, in create_engine
    dbapi = dbapi_meth(**dbapi_args)
  File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi
    import psycopg2
ModuleNotFoundError: No module named 'psycopg2'

tokoko commented 2 weeks ago

@tchughesiv I don't think this is specific to grpc_health, looks like we are not installing grpc extra in the default image. I guess we can add it since we are relying on grpc more now with remote registry, although hard to decide which extras need to be included and which can be left out. I think we need to do something like #4379 to solve this properly.

@ckavili What uri are you using with the sql registry? with sql registry the user is responsible for bringing in the dependencies needed for the sqlalchemy backend. Looks like you're using psycopg2 for registry backend, you need to bring that in yourself, feast[postgres] won't help here because feast postgres components (offline, online) depend on psycopg (psycopg3) instead. Alternatively you can switch to psycopg for the registry as well.

feast-dev / feast