Pebaz / nimporter

Compile Nim Extensions for Python On Import!
MIT License
822 stars 33 forks source link

What's the expected "workflow"? #58

Closed macintacos closed 2 years ago

macintacos commented 2 years ago

Hello! New to Nim, not-so-new to Python, but trying to figure out how to use this tool to get the best of both.

I setup the following repo to mess around with some stuff based on what I'm reading in the README: https://github.com/macintacos/nimport-playground

The project is pretty "rough", I would say. I'm not really understanding what the expected "workflow" is supposed to be. Right now I:

However, if I want to edit the Nim code, it seems like I need to recompile it completely with nimporter compile and then python setup.py install again. At first I thought running python setup.py install again would work (after changing some of the Nim code), but that doesn't seem to do anything.

Is this the expected workflow? Or am I missing something about my setup? Sorry if I'm missing something obvious 😬

macintacos commented 2 years ago

Oh also, I can't seem to figure out how to get docstrings from Nim files carried over so that my editor can recognize them.

Pebaz commented 2 years ago

Hi @macintacos! Thanks for checking out Nimporter!

The basic workflow is as follows:

  1. Install Nim "somehow" (this will greatly depend on your needs). I recommend installing it by manually downloading and maintaining a Choosenim installation. This is similar to the rustup workflow from the Rust community.
  2. Once you are able to invoke nim from the command line and it works, you'll be able to use Nimporter easily
  3. I see you already have a main.py. One thing to note is that Nimporter must be imported before any Nim extension modules. I usually go: import nimporter, the_nim_ext_mod all on one line
  4. In order to develop and test extension modules, you don't need to invoke pip or nimporter compile. Nimporter keeps track of changes to the Nim extensions automatically and will recompile next time it is imported when it sees a change in the Nim code. For all normal development, just have a Python module import Nimporter and then the Nim extension. This will trigger an initial build and then rebuilds when changes are detected.
  5. Now here's where things get interesting. When it's time to deploy/package, you need a setup.py that is used to hookup all the Nim extensions in the project. Nimporter was designed to support any number of deeply-nested Nim extension within a Python package heirarchy. When you specify ext_modules=nimporter.build_nim_extensions() in the setup.py, you are telling Nimporter to go and compile all the extensions in the project when a source or binary distribution is built from Pip.
  6. Note that nimporter compile is intended in situations where a setup.py cannot easily run. It performs all of the Nim extension compilation the exact same as if they were imported and should not be used except when packaging for restrictive environments.
  7. Now that you have a setup.py, you can python setup.py sdist to get a source distribution of the entire project. Alternatively, you can package into a platform-specific bundle using python setup.py bdist_wheel.
  8. About the docstrings, not sure if this will help, but here and here is the test code from Nimpy that addresses them.

If you run into any other issues, please let me or @SekouDiaoNlp know and we will do our best to help out! 😄

Pebaz commented 2 years ago

Closing this for now but definitely reopen if you run into any other issues! :)