search

HPInc / HP-Digital-Microfluidics

HP Digital Microfluidics Software Platform and Libraries
MIT License
3 stars 1 forks source link

Support for automatic pybind11 stub generation #265

Closed EvanKirshenbaum closed 8 months ago

EvanKirshenbaum commented 8 months ago

The Glider implementation is written in C++ in a different repo. To be able to call it from Python code, a set of adaptor classes are written using the pybind11 library.

In order for Python code to use the generated library safely, it needs access to a .pyi stub file. Up to this point, I've been attempting to maintain this file by hand, in both repositories. It hasn't always gone well. As I was making significant changes to the Glider code to support sensors (DMFSoftware/DynamicHAL#5), I realized that somebody had probably come up with a way to do this automatically, and, lo and behold, there is.

If you install the pybind11-stubgen module, you can, at least in Eclipse, run the command

pybind11-stubgen -o ${project_loc:MPAM/stubs} pyglider

and it will generate them for you. (At least, it looks as though it does. They warn that it might not be quite right, and I haven't tested it yet, but it looks okay.)

To run this, you need to make sure that when that command is run, the directory containing pyglider.pyd is in $PYTHONPATH. In Eclipse, this can be done in the External Tools Configuration for the comand on the Environment tab, setting the value of PYTHONPATH to D:\dmf-git\DynamicHAL\x64\Debug;${env_var:PYTHONPATH} (subtituting in your own directory, of course).

Note that this puts the .pyi file under /stubs, whereas it had been under /src, so you will have to make sure that /stubs is on your $PYTHONPATH when you run anything involving Bilby.

If this works, I am planning on removing the stub file from the Glider repo, and thus getting rid of something else that can become inconsistent. I will check in the automatically generated stub on the Python side, so others don't have to worry about regenerating it, but if they ever do, follow the instructions above (which should be reflected in the currently-nonexistent documentation). And I will delete the hand-written version of pyglider.pyi from /src.

While I'm doing this, I'm planning on splitting the current requirements.txt file into two: requirements.txt and dev-requirements.text. I'm going to move the sphinx requirements to the new file and put pybind-stubgen there, as normal users won't need these. I haven't thought much about whether the new file should (or can) be configured to pull in the user-needed packages.

Migrated from internal repository. Originally created by @EvanKirshenbaum on Apr 14, 2023 at 4:36 PM PDT. Closed on Apr 17, 2023 at 3:16 PM PDT.
EvanKirshenbaum commented 8 months ago

This issue was referenced by the following commits before migration:

EvanKirshenbaum commented 8 months ago

If you're wondering about that long string of commits with identical messages (or, as I now see, identical other than not being able to spell "issue" on the last two--sigh), for some unknown reason, MyPy on the server started complaining about three things in glider_client. MyPy on my machine had no problem. I enlisted GPT-4 to try to help me figure out why it was unable to do simple type inference, and it suggested a lot of things to try (there are actually more rounds; I eventually realized I should take the issue reference out of the message), but in the end it wound up just as mystified as I was, and I decided I had spent enough time and just suppressed the error. The last one is the only real one left in the branch.

Migrated from internal repository. Originally created by @EvanKirshenbaum on Apr 17, 2023 at 2:49 PM PDT.
EvanKirshenbaum commented 8 months ago

To do the stub generation, I added dev-tools/pyglider_stubs.py:

usage: pyglider_stubs.py [-h] [--module MODULE] [--stub-dir STUB_DIR]
                         [--prog PROG] [--search SEARCH]
                         [--logging {DEBUG,INFO,WARNING,ERROR,CRITICAL}]

options:
  -h, --help            show this help message and exit
  --module MODULE       Module to create stubs for. Default is 'pyglider'.
  --stub-dir STUB_DIR   Stub directory. Default is 'D:\dmf-
                        git\thylacine\mpam\stubs
  --prog PROG           Stub generator. Default is pybind11-stubgen
  --search SEARCH       Directories added to the search for the .pyd file
  --logging {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        The logging level. Default is INFO

I run it with --search d:\dmf-git\DynamicHAL\x64\Debug to point it to where the .pyd file is on my machine. The default for --stub-dir is actually the stubs directory parallel to the folder the module is in. Unless you move the module, it should be fine.

To run this, you need to install pybind11-stubgen. This can be done via pip.

To simplify things, I split a dev-requirements.txt out of requirements.txt. This is also suitable to passing to pip -r, but it holds the packages (currently pybind11-stubgen and the various sphinx pacakages) that are needed by developers but not by users.

Finally, an important note:

EvanKirshenbaum commented 8 months ago

I'm going to close this, but we still need to change the user docs to note that /stubs needs to be added to $PYTHONPATH. I think that this may be a doc that @cumbiem holds himself, so I've contacted him about it.

Migrated from internal repository. Originally created by @EvanKirshenbaum on Apr 17, 2023 at 3:16 PM PDT.