Speed up completion hook shellcode

[kislyuk]

@evanunderscore interest

In the course of looking into #278, I've become concerned about the performance impact of pkg_resources.iter_entry_points() in https://github.com/kislyuk/argcomplete/blob/master/argcomplete/_check_console_script.py.

I'd like us to profile the completion hook shellcode. We might want to use techniques developed by https://github.com/DropD/reentry to speed up entry point scanning, or look for other ways to speed up the shellcode.

[kislyuk]

See this comment for alignment with the development strategy of setuptools.

With this in mind, we should probably stop using pkg_resources, and try to use importlib_metadata.entry_points() for this purpose instead.

[evanunderscore]

For what it's worth, the penalty will only be paid if the thing you're completing is a file with a shebang line for Python. In the case that we don't actually want to complete it, the price is paid once per name per shell, after which point a default completer is registered and the name will not be checked again.

I went with the entry_points solution based on the discussion on https://github.com/pypa/pip/issues/4702. If there were an easier way to tell what a console entry point script looked like ahead of time, we could avoid paying that price in most cases.

It sounds like importlib_metadata will improve performance. Worth noting is that it seems to support 2.7 and 3.5+, however argcomplete still claims support for 3.3+ in the README. We could fall back on the pkg_resources method for 3.3/3.4, or could drop this particular feature for these versions.

Supposing we continue to use argcomplete._check_module, note that everything in argcomplete.__init__ will always be unnecessarily executed in the process of importing it. It's worth profiling and seeing how much of a performance penalty that imposes too. If it's significant, we could avoid it by either moving argcomplete._check_module to a separate script like python-argcomplete-check-easy-install-script, or by moving most of __init__ into a separate submodule.

For scripts we do actually want to complete, we are currently running this script logic every single time. We do have the option to register a separate completer that effectively caches the decision to complete and how, e.g. by running register-python-argcomplete which will avoid having to run the global completion script again for that name for the lifetime of the shell. The downside of this is that if the underlying script is removed, or changes to one that is not intended to be completed, that decision will still be cached and it will attempt to execute it anyway.

Also worth considering, we could make this script completion feature opt-in/out with a flag to activate-global-python-argcomplete.

[evanunderscore]

I forgot I'd actually prototyped the decision caching at some point. You can see the changes here: https://github.com/evanunderscore/argcomplete/tree/cachedecision. Note this won't work for module completion but it would work for entry points.

[kislyuk]

Oh, we only support versions of Python that have upstream support. I'll fix the readme.

[evanunderscore]

Are we similarly considering dropping support for Python 2? It reached EOL at the beginning of this year, but I can understand making a special case for it given how prevalent it still is. However as time goes on, our dependencies will drop support for it and the burden of maintaining it will grow. Do you have any thoughts around this?

[kislyuk]

We are not actively maintaining support for Python 2, and I expect to drop all Python 2-specific compatibility code sometime later this year. In general we will follow EOL announcements from upstream to decide when to drop support, and in case of Python 2.7, EOL was January 1, 2020.