Closed elagil closed 1 year ago
This pull request introduces 1 alert when merging a8f18ca961bae971fac5c61a6e0bbac1ac3fbc70 into e93b0452cc5c1d1fe47e4adf5799deaf7198a671 - view on LGTM.com
new alerts:
I'm still interested in this. Will need to get the CI issues fixed.
@haata I am still working on it, but I decided to rewrite it. Testing the original implementation against test.capnp
showed irregularities (incorrect nesting, lists of lists, etc.) and missing type hints (e.g. pycapnp readers and builders, the builder method write()
, and something like as_builder()
or as_reader()
), so lots of fixes were required.
Since my rewrite causes increased complexity and modularization, this can also become a standalone package with its own CLI.
The CLI takes:
Then, it creates type hints for all matching schemas, as well as a *.py
file that handles loading the schemas and provides a functional Python module. I will show an example tomorrow.
@haata Here is an example. Please let me know, if you would add some important methods that I might have missed. The generator can now also handle type imports (also outside of the current directory) from other schemas without issues.
This was generated using the following command line call:
capnp-stub-generator -r -p "ex*.capnp"
Consider this nested schema ex.capnp
:
@0x9420d0efc6c9fee8;
using import "ex_imp.capnp".TestImport;
struct TestNestedTypes {
enum NestedEnum1 {
foo @0;
bar @1;
}
struct NestedStruct {
enum NestedEnum2 {
baz @0;
qux @1;
quux @2;
}
outerNestedEnum @0 :TestNestedTypes.NestedEnum1 = bar;
innerNestedEnum @1 :NestedEnum2 = quux;
listOuterNestedEnum @2 :List(NestedEnum1) = [foo, bar];
listInnerNestedEnum @3 :List(NestedEnum2) = [quux, qux];
}
nestedStruct @0 :NestedStruct;
outerNestedEnum @1 :NestedEnum1 = bar;
innerNestedEnum @2 :NestedStruct.NestedEnum2 = quux;
someListofList @3: List(List(List(NestedEnum1)));
importedVariable @4: TestImport;
}
alongside this ex_imp.capnp
file, from which a type is imported:
@0x9420d0efc6c9fed8;
struct TestImport {
aVariable @0: Float32;
}
The output of the stub generator is currently this ex_capnp.pyi
file:
"""This is an automatically generated stub for `ex.capnp`."""
from __future__ import annotations
from contextlib import contextmanager
from io import BufferedWriter
from typing import Iterator, List, Literal, Union, overload
from .ex_imp_capnp import TestImport
class TestNestedTypes:
class NestedStruct:
NestedEnum1 = Literal["foo", "bar"]
NestedEnum2 = Literal["baz", "qux", "quux"]
outerNestedEnum: TestNestedTypes.NestedStruct.NestedEnum1
innerNestedEnum: TestNestedTypes.NestedStruct.NestedEnum2
listOuterNestedEnum: List[TestNestedTypes.NestedStruct.NestedEnum1]
listInnerNestedEnum: List[TestNestedTypes.NestedStruct.NestedEnum2]
@staticmethod
@contextmanager
def from_bytes(
data: bytes, traversal_limit_in_words: Union[int, None] = ..., nesting_limit: Union[int, None] = ...
) -> Iterator[TestNestedTypes.NestedStructReader]: ...
def to_bytes(self) -> bytes: ...
@staticmethod
def new_message() -> TestNestedTypes.NestedStructBuilder: ...
class NestedStructReader(TestNestedTypes.NestedStruct):
def as_builder(self) -> TestNestedTypes.NestedStructBuilder: ...
class NestedStructBuilder(TestNestedTypes.NestedStruct):
def as_reader(self) -> TestNestedTypes.NestedStructReader: ...
@staticmethod
def write(file: BufferedWriter) -> None: ...
nestedStruct: TestNestedTypes.NestedStruct
outerNestedEnum: TestNestedTypes.NestedStruct.NestedEnum1
innerNestedEnum: TestNestedTypes.NestedStruct.NestedEnum2
someListofList: List[List[List[TestNestedTypes.NestedStruct.NestedEnum1]]]
importedVariable: TestImport
@overload
def init(self, name: Literal["nestedStruct"]) -> TestNestedTypes.NestedStruct: ...
@overload
def init(self, name: Literal["importedVariable"]) -> TestImport: ...
@staticmethod
@contextmanager
def from_bytes(
data: bytes, traversal_limit_in_words: Union[int, None] = ..., nesting_limit: Union[int, None] = ...
) -> Iterator[TestNestedTypesReader]: ...
def to_bytes(self) -> bytes: ...
@staticmethod
def new_message() -> TestNestedTypesBuilder: ...
class TestNestedTypesReader(TestNestedTypes):
def as_builder(self) -> TestNestedTypesBuilder: ...
class TestNestedTypesBuilder(TestNestedTypes):
def as_reader(self) -> TestNestedTypesReader: ...
@staticmethod
def write(file: BufferedWriter) -> None: ...
And this ex_capnp.py
file, handling the load
of the schema by means of pycapnp
:
"""This is an automatically generated stub for `ex.capnp`."""
import os
import capnp # type: ignore
capnp.remove_import_hook()
here = os.path.dirname(os.path.abspath(__file__))
module_file = os.path.abspath(os.path.join(here, "ex.capnp"))
TestNestedTypes = capnp.load(module_file).TestNestedTypes
TestNestedTypesBuilder = TestNestedTypes
TestNestedTypesReader = TestNestedTypes
This is the generated stub ex_imp_capnp.pyi
"""This is an automatically generated stub for `ex_imp.capnp`."""
from __future__ import annotations
from contextlib import contextmanager
from io import BufferedWriter
from typing import Iterator, Union
class TestImport:
aVariable: float
@staticmethod
@contextmanager
def from_bytes(
data: bytes, traversal_limit_in_words: Union[int, None] = ..., nesting_limit: Union[int, None] = ...
) -> Iterator[TestImportReader]: ...
def to_bytes(self) -> bytes: ...
@staticmethod
def new_message() -> TestImportBuilder: ...
class TestImportReader(TestImport):
def as_builder(self) -> TestImportBuilder: ...
class TestImportBuilder(TestImport):
def as_reader(self) -> TestImportReader: ...
@staticmethod
def write(file: BufferedWriter) -> None: ...
And the load
handler ex_imp_capnp.py
"""This is an automatically generated stub for `ex_imp.capnp`."""
import os
import capnp # type: ignore
capnp.remove_import_hook()
here = os.path.dirname(os.path.abspath(__file__))
module_file = os.path.abspath(os.path.join(here, "ex_imp.capnp"))
TestImport = capnp.load(module_file).TestImport
TestImportBuilder = TestImport
TestImportReader = TestImport
Is this PR still maintained? What are the current remaining issues?
@brainslush Yes, it is! I will submit a new version of it soon, as I have been optimizing it in a production environment. A lot of bugs have popped up over time, so the module wasn't really ready for many kinds of schemas.
This would be wonderful to have! It's very tedious working with pycapnp with no editor assistance.
Sorry to keep you waiting, I will try to bring something up as fast as possible
Please check https://gitlab.com/mic_public/tools/python-helpers/capnp-stub-generator
This will probably not be the final place/shape for release, but we have been using it successfully like this.
For now I will close this merge request, because this is an entirely separate tool.
Cool! I've added a link in the top-level README. https://github.com/capnproto/pycapnp/tree/ed894304a34ce28254779ea5215afe942a0d5b31#stub-file-generation
Longer term it may make more sense to integrate this directly under https://github.com/orgs/capnproto. But there is no rush for this.
Great, thanks for adding it!
I agree, it would be better to integrate it directly. However, it might need to stay as a standalone package and command-line tool, since it is too complex to be put into just a single script.
This is awesome, thank you for sharing @elagil!
@loloxwg Yes, it's actually not published on the public pypi server, only on our company internal server. I will fix that soon.
In the mean time, just clone the repository and install it with pip from the local source.
You can also let pip clone it:
pip install git+https://gitlab.com/mic_public/tools/python-helpers/capnp-stub-generator.git@main
Since there is no movement in https://github.com/capnproto/pycapnp/pull/260, I decided to pick up that pull request and fix remaining CI issues.
Update: This is now a separate application, see https://github.com/capnproto/pycapnp/pull/289#issuecomment-1519595221