Add Jupyter kernel for MacroPy

[Technologicat]

Macros in the IPython REPL! There was already macropy.console, but I really like readline, tab completion, and the other IPython features. Therefore, here's a first cut at a Jupyter kernel for MacroPy.

Once installed, the following works in jupyter console --kernel imacropy:

from unpythonic.syntax import macros, let
let((x, 1),
    (y, 2))[
      (x, y)]

(The startup command is too long to type, so I have aliased it to mipython3 in my .bashrc.)

This inherits from IPythonKernel and plugs in an AST transformer, basically borrowing everything else from the standard IPython kernel.

Note this requires renaming macropy/logging.py to avoid shadowing the stdlib module with the same name; otherwise installation of the new kernel fails due to Jupyter loading the wrong logging module.

Comments?

[edit] kernel name.

[Technologicat]

Applying some polish.

• Jupyter help some_macro? now works for macros (that are currently imported to the REPL). This works by importing the macro stubs as normal Python imports behind the scenes.
• Macro expansion errors are properly reported by Jupyter.
• If the user tells the REPL to import some macros from the same macro definition module again, the system reloads the module. This should allow semi-live updates to macro definitions: hack on your macros, re-import, re-test, without needing to exit the REPL session.

In the REPL, the set of macros available from mymodule, at any given time, are those specified in the most recent from mymodule import macros, ... import statement. This is a design decision that I might still change later.

(The original basic REPL didn't consider the case of importing the same names multiple times over the same REPL session; attempting to do that would introduce duplicate entries to the stored macro bindings. A first version at avoiding that produced the above solution.)

[Technologicat]

Turns out the configuration is loaded pretty deep in the IPython app. So instead of a custom Jupyter kernel, it's better to make a custom IPython extension.

Most of the code stayed as-is, but the new version interfaces with IPython a bit differently.

As a bonus, there's no need for the install process that was confused by the presence of macropy.logging, so we can now revert that.

I'll update the PR now.

[edit] Oh, and in this version, there's no need for the shebang, either, so I got rid of it.

[edti2] One more thing: to load this new version, use %load_ext macropy.imacro, or to autoload whenever IPython starts, in ipython_config.py, add "macropy.imacro" to the list c.InteractiveShellApp.extensions.

Now this is feature-complete, and seems to be working - now it needs no further changes from my side. Sorry for initially missing the bug!

[catb0t]

These are documented in the file (nicely done):

• Jupyter help some_macro? now works for macros

• to load this new version, use %load_ext macropy.imacro, or to autoload whenever IPython starts, in ipython_config.py, add "macropy.imacro" to the list c.InteractiveShellApp.extensions

but these aren't (should they be?):

• Macro expansion errors are properly reported by Jupyter.

• If the user tells the REPL to import some macros from the same macro definition module again, the system reloads the module. This should allow semi-live updates to macro definitions: hack on your macros, re-import, re-test, without needing to exit the REPL session.

• In the REPL, the set of macros available from mymodule, at any given time, are those specified in the most recent from mymodule import macros

[Technologicat]

New version, taking into account Cat's comments.

• Module docstring updated with all the relevant info.
• Both pre-7.0 and 7.0+ IPython are supported. In 7.0+, the source code for the cell is retrieved after string transformers have run, which gets rid of a corner case.

[azazel75]

Is this code testable in any way? Personally I don't use neither jupiter or IPython so it's impossible for me to judge most of this code, and more importantly to maintain it. I would be more inclined to include it if it contained some test that automatically check its working condition, or maybe it can be a separated module maintained by you, @Technologicat

[Technologicat]

By its nature it's interactive, but I can look into if there's some sensible way to script IPython to load it and run some simple macro-enabled code in the REPL. I'd be surprised if there wasn't.

I'd prefer this to be in the core distribution, because that way the users wouldn't need to be aware of the existence of such a separate add-on, or install a separate package to enable just one additional feature. But I do understand where you're coming from - for one of my projects, I got some Windows-specific fixes and running only Linux, there's no way for me to even test them.

I'll look into possibilities for automatic testing and keep you posted. Let's make the decision once we see how it goes.

[chmp]

With regards to automatic testing, I can recommend nbval. It is a pytest plugin to execute notebooks and compare the generated outputs against a previous run. You can also ignore all outputs and only test that no unexpected exceptions are thrown. I am using it for my own integration tests.

[Technologicat]

I've been thinking about this. @azazel75 , what's your opinion?

On my part, I think the main sensible options are:

1. Write some unit tests. Make MacroPy depend on nbval in order to run them. But do we want a dependency just for this one feature?
2. Split this off into a separate macropy-addons package so that I can maintain it and handle any issues. If you're ok with it, maybe mention the package in the MacroPy docs to help users discover it.

I also have a generic MacroPy3 bootstrapper (e.g. macropy3 my_script_with_macros.py, and if __name__ == '__main__' works as usual) that could go into the addon package, if you're not interested in including that feature in MacroPy itself. I'm currently providing the bootstrapper as part of unpythonic, but I'd like to move it somewhere more logical and more easily discoverable.

I think the REPL integration and bootstrapper together form a set that makes experimenting with macros more agile.

(pydialect also includes a modified version of the bootstrapper that loads the dialect machinery. I'm still weighing the pros and cons of combining these two versions.)

[Technologicat]

(As for the commits, just adding relevant fixes from my local master branch to keep this up to date. Now manually tested on Python 3.6.7, IPython 7.5.0, seems to work.)

[azazel75]

I think that for both the features it's best if they are kept out of it, for now

[azazel75]

I just want to publish a stable release, as requested here lihaoyi/macropy#94 . I would like to avoid new dependencies or disrupting changes.

[Technologicat]

Sure. Perhaps the best option for now is make a separate package, and later worry whether we want to include the additional features into the core or keep them as an addon.

I'll package up the two features and post the link here once done.

[Technologicat]

Separate package made, see imacropy. It's available on PyPI.

[Technologicat]

Now that PyPI has had the separate imacropy package (containing this functionality) for over a year now, I think it's time to close this PR. :)