search

CZ-NIC / setuptools-grpc

Setuptools module for building protobuf and grpc service python modules.
GNU General Public License v3.0
5 stars 1 forks source link

Is there a working example of how to use this project anywhere? #16

Open jfly opened 6 months ago

jfly commented 6 months ago

I've been trying to follow the README, and have run into a couple of issues:

  1. The globs in the README example didn't work for me. I think that's just a simple typo, which I've attempted to fix here: https://github.com/CZ-NIC/setuptools-grpc/pull/15
  2. It's not immediately obvious to me what the correct way to configure the proto_path and output_path is to ensure that the generated files are included in a built wheel.

The one example I've been able to find that uses setuptools-grpc is https://github.com/mlcommons/chakra, which chooses to set proto_path and output_path to the same thing, which is not something I'd have ever expected to do.

I think a simple, working example of one correct way of using this tool would go a long way towards clearing up any confusion.

stinovlas commented 6 months ago

We do use it in a multiple projects, although with the same input and output path. Take a look here:

https://gitlab.nic.cz/fred/api/messenger (or any other project in the same group)

As for setuptools picking up the generated modules, this may require setting options.packages = find_namespace: if there are no __init__.py modules present. I did think about adding generating of __init__.py to the setuptools-grpc as well, so you wouldn't have to use namespace packages, but so far no one requested that and we actually benefit from having namespace packages, since we have a top level fred_api namespace for all APIs in the group.

I agree that it makes sense to add examples directory with some sample configurations.

stinovlas commented 6 months ago

Just to be clear, if you would benefit from generating empty __init__.py files in the output dirs, so that you don't need to use namespace packages, I'm fine with adding that as a feature (probably opt-in, so we don't break people depending on namespace packaging). So far no one requested that and I didn't want to implement it just for the sake of it. But if there is a demand, it should be fairly easy to add.

jfly commented 6 months ago

Thanks for the info. I'm trying to put together an example with different input and output paths, and I can't get it to work. See https://github.com/jfly/setuptools-grpc-demo/tree/broken-different-proto-and-output-paths (in particular this commit message). Also here's a quick demo:

Trying to install this in a fresh venv (`pip install 'jfly @ git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths'`) ``` $ pip install 'jfly @ git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths' -vvv Using pip 23.2.1 from /home/jeremy/tmp/2024-01-12-pyhack-2/.direnv/python-3.11/lib/python3.11/site-packages/pip (python 3.11) Non-user install because user site-packages disabled Created temporary directory: /run/user/1000/pip-build-tracker-1esenusq Initialized build tracking at /run/user/1000/pip-build-tracker-1esenusq Created build tracker: /run/user/1000/pip-build-tracker-1esenusq Entered build tracker: /run/user/1000/pip-build-tracker-1esenusq Created temporary directory: /run/user/1000/pip-install-qoucadyw Created temporary directory: /run/user/1000/pip-ephem-wheel-cache-0koqn4fr Collecting jfly@ git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths Cloning https://github.com/jfly/setuptools-grpc-demo.git (to revision broken-different-proto-and-output-paths) to /run/user/1000/pip-install-qoucadyw/jfly_8df21ecae779444e991200dc438863ae Running command git version git version 2.42.0 Running command git clone --filter=blob:none --verbose --progress https://github.com/jfly/setuptools-grpc-demo.git /run/user/1000/pip-install-qoucadyw/jfly_8df21ecae779444e991200dc438863ae Cloning into '/run/user/1000/pip-install-qoucadyw/jfly_8df21ecae779444e991200dc438863ae'... POST git-upload-pack (175 bytes) POST git-upload-pack (322 bytes) remote: Enumerating objects: 7, done. remote: Counting objects: 14% (1/7) remote: Counting objects: 28% (2/7) remote: Counting objects: 42% (3/7) remote: Counting objects: 57% (4/7) remote: Counting objects: 71% (5/7) remote: Counting objects: 85% (6/7) remote: Counting objects: 100% (7/7) remote: Counting objects: 100% (7/7), done. remote: Compressing objects: 25% (1/4) remote: Compressing objects: 50% (2/4) remote: Compressing objects: 75% (3/4) remote: Compressing objects: 100% (4/4) remote: Compressing objects: 100% (4/4), done. remote: Total 7 (delta 1), reused 6 (delta 0), pack-reused 0 Receiving objects: 14% (1/7) Receiving objects: 28% (2/7) Receiving objects: 42% (3/7) Receiving objects: 57% (4/7) Receiving objects: 71% (5/7) Receiving objects: 85% (6/7) Receiving objects: 100% (7/7) Receiving objects: 100% (7/7), done. Resolving deltas: 0% (0/1) Resolving deltas: 100% (1/1) Resolving deltas: 100% (1/1), done. Running command git show-ref broken-different-proto-and-output-paths 650848228143e7ef54f3bcab53f8945a3fc0f471 refs/remotes/origin/broken-different-proto-and-output-paths Rev options , branch_name broken-different-proto-and-output-paths Running command git symbolic-ref -q HEAD refs/heads/working-same-proto-and-output-paths Running command git checkout -b broken-different-proto-and-output-paths --track origin/broken-different-proto-and-output-paths Switched to a new branch 'broken-different-proto-and-output-paths' branch 'broken-different-proto-and-output-paths' set up to track 'origin/broken-different-proto-and-output-paths'. Resolved https://github.com/jfly/setuptools-grpc-demo.git to commit 650848228143e7ef54f3bcab53f8945a3fc0f471 Running command git rev-parse HEAD 650848228143e7ef54f3bcab53f8945a3fc0f471 Added jfly@ git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths from git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths to build tracker '/run/user/1000/pip-build-tracker-1esenusq' Created temporary directory: /run/user/1000/pip-build-env-tst4wk_e Running command pip subprocess to install build dependencies Using pip 23.2.1 from /home/jeremy/tmp/2024-01-12-pyhack-2/.direnv/python-3.11/lib/python3.11/site-packages/pip (python 3.11) Collecting setuptools Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/55/3a/5121b58b578a598b269537e09a316ad2a94fdd561a2c6eb75cd68578cc6b/setuptools-69.0.3-py3-none-any.whl.metadata Using cached setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB) Collecting setuptools-grpc Obtaining dependency information for setuptools-grpc from https://files.pythonhosted.org/packages/cf/2f/d332a2265239ddb2a0a3fd4af2b5fd9eb862eba2d870c397562995605fa8/setuptools_grpc-0.5-py3-none-any.whl.metadata Using cached setuptools_grpc-0.5-py3-none-any.whl.metadata (5.4 kB) Collecting grpcio-tools~=1.49 (from setuptools-grpc) Obtaining dependency information for grpcio-tools~=1.49 from https://files.pythonhosted.org/packages/29/0f/fdfa88aff42abc0caa29f74cfa47e77ea1d6385c073c082fef582ac0ec9f/grpcio_tools-1.60.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Using cached grpcio_tools-1.60.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB) Collecting protobuf<5.0dev,>=4.21.6 (from grpcio-tools~=1.49->setuptools-grpc) Obtaining dependency information for protobuf<5.0dev,>=4.21.6 from https://files.pythonhosted.org/packages/81/9e/63501b8d5b4e40c7260049836bd15ec3270c936e83bc57b85e4603cc212c/protobuf-4.25.2-cp37-abi3-manylinux2014_x86_64.whl.metadata Using cached protobuf-4.25.2-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes) Collecting grpcio>=1.60.0 (from grpcio-tools~=1.49->setuptools-grpc) Obtaining dependency information for grpcio>=1.60.0 from https://files.pythonhosted.org/packages/de/01/a8d9bcc59526f22b8fef29c234cc63434f05dae1154d979222c02b31a557/grpcio-1.60.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Using cached grpcio-1.60.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB) Using cached setuptools-69.0.3-py3-none-any.whl (819 kB) Using cached setuptools_grpc-0.5-py3-none-any.whl (17 kB) Using cached grpcio_tools-1.60.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB) Using cached grpcio-1.60.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB) Using cached protobuf-4.25.2-cp37-abi3-manylinux2014_x86_64.whl (294 kB) Installing collected packages: setuptools, protobuf, grpcio, grpcio-tools, setuptools-grpc Successfully installed grpcio-1.60.0 grpcio-tools-1.60.0 protobuf-4.25.2 setuptools-69.0.3 setuptools-grpc-0.5 [notice] A new release of pip is available: 23.2.1 -> 23.3.2 [notice] To update, run: pip install --upgrade pip Installing build dependencies ... done Running command Getting requirements to build wheel running egg_info creating jfly.egg-info writing jfly.egg-info/PKG-INFO writing dependency_links to jfly.egg-info/dependency_links.txt writing top-level names to jfly.egg-info/top_level.txt writing manifest file 'jfly.egg-info/SOURCES.txt' reading manifest file 'jfly.egg-info/SOURCES.txt' writing manifest file 'jfly.egg-info/SOURCES.txt' Getting requirements to build wheel ... done Running command pip subprocess to install backend dependencies Using pip 23.2.1 from /home/jeremy/tmp/2024-01-12-pyhack-2/.direnv/python-3.11/lib/python3.11/site-packages/pip (python 3.11) Collecting wheel Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/c7/c3/55076fc728723ef927521abaa1955213d094933dc36d4a2008d5101e1af5/wheel-0.42.0-py3-none-any.whl.metadata Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB) Using cached wheel-0.42.0-py3-none-any.whl (65 kB) Installing collected packages: wheel Creating /run/user/1000/pip-build-env-tst4wk_e/normal/bin changing mode of /run/user/1000/pip-build-env-tst4wk_e/normal/bin/wheel to 755 Successfully installed wheel-0.42.0 [notice] A new release of pip is available: 23.2.1 -> 23.3.2 [notice] To update, run: pip install --upgrade pip Installing backend dependencies ... done Created temporary directory: /run/user/1000/pip-modern-metadata-welukk06 Running command Preparing metadata (pyproject.toml) running dist_info creating /run/user/1000/pip-modern-metadata-welukk06/jfly.egg-info writing /run/user/1000/pip-modern-metadata-welukk06/jfly.egg-info/PKG-INFO writing dependency_links to /run/user/1000/pip-modern-metadata-welukk06/jfly.egg-info/dependency_links.txt writing top-level names to /run/user/1000/pip-modern-metadata-welukk06/jfly.egg-info/top_level.txt writing manifest file '/run/user/1000/pip-modern-metadata-welukk06/jfly.egg-info/SOURCES.txt' reading manifest file '/run/user/1000/pip-modern-metadata-welukk06/jfly.egg-info/SOURCES.txt' writing manifest file '/run/user/1000/pip-modern-metadata-welukk06/jfly.egg-info/SOURCES.txt' creating '/run/user/1000/pip-modern-metadata-welukk06/jfly-0.0.1.dist-info' Preparing metadata (pyproject.toml) ... done Source in /run/user/1000/pip-install-qoucadyw/jfly_8df21ecae779444e991200dc438863ae has version 0.0.1, which satisfies requirement jfly@ git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths from git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths Removed jfly@ git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths from git+https://github.com/jfly/setuptools-grpc-demo.git@broken-different-proto-and-output-paths from build tracker '/run/user/1000/pip-build-tracker-1esenusq' Created temporary directory: /run/user/1000/pip-unpack-kfess7h3 Building wheels for collected packages: jfly Running command git rev-parse HEAD 650848228143e7ef54f3bcab53f8945a3fc0f471 Created temporary directory: /run/user/1000/pip-wheel-3_863fyq Destination directory: /run/user/1000/pip-wheel-3_863fyq Running command Building wheel for jfly (pyproject.toml) running bdist_wheel running build running build_grpc building protos generating foo/module1.proto → foo/module1_pb2.py building grpc generating foo/module1.proto → foo/module1_pb2_grpc.py running build_py running egg_info writing jfly.egg-info/PKG-INFO writing dependency_links to jfly.egg-info/dependency_links.txt writing top-level names to jfly.egg-info/top_level.txt reading manifest file 'jfly.egg-info/SOURCES.txt' writing manifest file 'jfly.egg-info/SOURCES.txt' installing to build/bdist.linux-x86_64/wheel running install running install_lib warning: install_lib: 'build/lib' does not exist -- no Python modules to install running install_egg_info Copying jfly.egg-info to build/bdist.linux-x86_64/wheel/jfly-0.0.1-py3.11.egg-info running install_scripts creating build/bdist.linux-x86_64/wheel/jfly-0.0.1.dist-info/WHEEL creating '/run/user/1000/pip-wheel-3_863fyq/.tmp-o6ahiqsk/jfly-0.0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it adding 'jfly-0.0.1.dist-info/METADATA' adding 'jfly-0.0.1.dist-info/WHEEL' adding 'jfly-0.0.1.dist-info/top_level.txt' adding 'jfly-0.0.1.dist-info/RECORD' removing build/bdist.linux-x86_64/wheel Building wheel for jfly (pyproject.toml) ... done Created wheel for jfly: filename=jfly-0.0.1-py3-none-any.whl size=904 sha256=c7eed92a4897aca10015c53a8a81b552a36bb2b5629561a37643829907516819 Stored in directory: /run/user/1000/pip-ephem-wheel-cache-0koqn4fr/wheels/4c/25/e6/5c477311458918af6e1b244dd9c80b43a75e862fd4787c43cc Successfully built jfly Installing collected packages: jfly Successfully installed jfly-0.0.1 Remote version of pip: 23.3.2 Local version of pip: 23.2.1 Was pip installed by pip? True [notice] A new release of pip is available: 23.2.1 -> 23.3.2 [notice] To update, run: pip install --upgrade pip Removed build tracker: '/run/user/1000/pip-build-tracker-1esenusq' ```

Note how there are no module1_pb2.* files:

$ pip show -f jfly
Name: jfly
Version: 0.0.1
Summary:
Home-page:
Author:
Author-email:
License:
Location: /home/jeremy/tmp/2024-01-12-pyhack-2/.direnv/python-3.11/lib/python3.11/site-packages
Requires:
Required-by:
Files:
  jfly-0.0.1.dist-info/INSTALLER
  jfly-0.0.1.dist-info/METADATA
  jfly-0.0.1.dist-info/RECORD
  jfly-0.0.1.dist-info/REQUESTED
  jfly-0.0.1.dist-info/WHEEL
  jfly-0.0.1.dist-info/direct_url.json
  jfly-0.0.1.dist-info/top_level.txt

Is this possible? From the experimentation I've been doing, it feels to me like the directory structure must be in place before build_grpc even runs.

stinovlas commented 6 months ago

So, first thing I see is that you need to add include_package_data = true to your your setup.cfg (section options) and also indicate in MANIFEST.in that you want to include all proto files (for example with graft src).

stinovlas commented 6 months ago

I usually use pyhon -m build to test the build itself. In this case, the setuptools complain about this:

error: error in 'output_path' option: './out' does not exist or is not a directory

However, this directory actually does exist. I'm not sure why setuptools don't recognize it, but we use Command.ensure_dirname("output_path") for this check. I'll try to look into it and find out what it is that setuptools ignore this directory.

stinovlas commented 6 months ago

OK, I get it now. The problem is that out directory does not contain any python modules and is therefore not included in the sdist befor building the wheel from it. This should fix it:

setup.cfg

[options]
packages = find_namespace:
include_package_data = true

[build_grpc]
proto_files = **/*.proto
grpc_files = **/*.proto
proto_path = ./src
output_path = ./out

MANIFEST.in

graft src
graft out

Now, as you said, the directory structure indeed has to be in place at the moment. grpc_tools.protoc does not create the output dir if not present, but I wonder whether setuptools_grpc should. It's definitely something to think about.