Experiment in making keywords importable.

[abingham]

This provides a way for build.cfg files to import the various bfg keywords. The primary reason for this is so that IDEs will know about these symbols and not treat them as errors. This addresses #135.

The technique I'm using introduces a new module bfg9000.keywords. This has a module-level attribute for each of the bfg9000 keywords; they all currently point to None, but they could point to something more useful (see notes in the file).

When the build.cfg is executed, bfg9000.build._execute_script replaces the values in keywords with the actual objects.

The result is that you can have a build.cfg like this:

from bfg9000.keywords import alias, default, build_step

# rest of build definition, just like any other.

and any Python-aware IDE will at least recognize that alias, default, and build_step are legitimate identifiers.

This PR is more about getting feedback on the approach and details (e.g. naming). It's certainly missing some details like handling options.cfg.

[jimporter]

I feel like there's got to be a better way to handle this. From an IDE's point of view, a .bfg file should be no different than an embedded Python script that adds some extra builtin functions. Surely someone has run into an issue like this before and resolved it in a way that we could piggyback on. Maybe jedi has something we could hook into...

[abingham]

The other approaches I can think of are ones you've already mentioned. We could probably write IDE extensions to prevent the keywords from being flagged as errors. This has the downside of requiring different extensions for each IDE, and I'm not sure how well or easily this would integrate with e.g. tooltips. I have lots of experience with emacs extensions, and IIRC the syntax checking and docstring tooltips are two entirely separate subsystems. I'm hesitant to go down this road because of the fan-out of work it implies.

The second alternative would be to teach the language servers about these symbols and their documentation. This seems like less work than operating at the IDE extensions level, but it's still more work than just jedi. There's at least Microsoft's Python language server and whatever PyCharm uses (which might be jedi, I'm not sure). And I'm also not sure that injecting symbols is even supported by all/some of them, so we'd need to sort that out.

I guess what appeals to me most about the approach I've coded (and I agree that it's at least a little bit hacky) is that it works at the Python language level and will thus work with any functioning set of tools. It's also pretty simple.

I suppose a middle ground is that bfg could provide a pre-execute hook of some sort. I could replace bfg9000.keywords with an external module and, in that hook, populate it with the live objects. This feels like over-engineering to me (mostly because I can't think of other use cases for that hook), but I think allowing you to keep conceptual integrity over the code is generally more important than any particular feature.

[jimporter]

I'll have to think about this, since it is indeed somewhat painful to have to deal with a variety of different IDEs to make them play nice with bfg files. I was thinking about something like the pre-execute hook too, since it would let a third-party handle this (at least until a better solution presents itself). There may even be some other things a pre-execute hook is good for, such as #48.

I'm not sure how such a hook would look just yet, but I think I'd want to avoid doing it with a setuptools entry point. That feels like it could cause "spooky action at a distance" where some arbitrary third-party hooks into bfg, does some magic, and then stuff "just works"... except when you inevitably forget to tell everyone else about the third-party module you have installed, and now the build doesn't work on their system for non-obvious reasons. Requiring some explicit way of enabling the hook in each project would probably be easier to keep track of...

[abingham]

Requiring some explicit way of enabling the hook in each project would probably be easier to keep track of...

You could introduce another keyword e.g. use_hook that just verifies that a given hook is installed. If you want to ensure a one-to-one correspondence between use_hook() invocations and actual hook activations, I think you'd need to scan build.cfg for the use_hook calls before executing it. On the other hand, if use_hook just verifies that a given hook is installed - but there might be hooks installed without corresponding use_hook calls - then it can just run as part of the script.

Alternatively, you could introduce something like setup.bfg that handles this kind of pre-execution work. This is simple and clear, but it introduces another file to the mix.

[jimporter]

Having looked into this some more, perhaps a https://github.com/palantir/python-language-server plugin would work? The Language Server Protocol is IDE-agnostic, and most IDEs have LSP support either natively or with an IDE plugin. I've tried to dig into how you'd extend python-language-server here, but it's fairly complex, and I'm not really familiar with how LSP works...

[jimporter]

@abingham This is just a thought, but what if you made an empty bfg9000_builtins.py file and then made a bfg9000_builtins.pyi stub file with the appropriate stubs? Based on my (limited) understanding of how LSP/IDEs work, that should make the IDE happy.

I'm still digging into this to come up with a fully-automatic solution, but maybe the above would let you use bfg in your IDE without having to patch bfg itself...

[jimporter]

@abingham I'm not sure if you're still looking for a solution here, but the following works for me when using python-lsp-server, pyls-flake8, and Emacs+Eglot. If you run the following script and redirect the output to /path/to/project/.flake8, you should stop seeing errors about undefined symbols. If you prefer using PyFlakes (which is the default), you can probably do something similar. I'm trying to figure out how to do this in a more-automatic way (maybe by creating a wrapper around python-lsp-server), but hopefully this will make things less painful until then.

from unittest.mock import MagicMock

import bfg9000.builtins
from bfg9000.builtins.builtin import _allbuiltins

bfg9000.builtins.init()
builtins = set()
for i in _allbuiltins.values():
    builtins.update(i.bind(MagicMock()).keys())

print('[flake8]\nbuiltins =')
print('  ' + ',\n  '.join(sorted(builtins)))

[jimporter]

Another solution for this would be to configure your editor to edit *.bfg files as something other than vanilla Python; then they shouldn't get picked up by LSP. I've done this for Emacs here: https://github.com/jimporter/bfg9000-mode. To make life easier on people who don't have that installed, you can set Python to be the fallback language:

# -*- mode: python; mode: bfg9000 -*-