Standardize runner scripts

When running dep11-generator script in our infra from git in a virtualenv, we would get some stacktrace (see below).

The issue is that __spec__ (new in python 3.4) needs to be define when using a pool of process with the multiprocessing module, if initialized with set_start_method('forkserver').

When using the setuptools generated script (in the script section), those are using __import__() to load the executable scripts, which doesn't set those properties. The bug seems thus to be in that tool. However, a more correct/modern way is to use the module entrypoints() for setup.py. This one generate in a similar way the executable script, but does something way more standard like what we have in import <module_name>:

all code is now in the module itself, enabling everything to be pre-compiled in most distribution.
it doesn't use and abuse __import__ to load modules

I thus added this refactoring and moving the code to the dep11 module themselves main() method. I did rewrite the scripts themselves that I guess you are using for testing, but they are not installed/shipped anymore by setuptools. That way, we use a similar way to load and start our local scripts for testing than the one installed on the system.

The stacktrace was:

Traceback (most recent call last):
  File "/home/ubuntu/appstream/bin/dep11-generator", line 4, in <module>
    __import__('pkg_resources').run_script('dep11==0.2', 'dep11-generator')
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/pkg_resources/__init__.py", line 735, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/pkg_resources/__init__.py", line 1652, in run_script
    exec(code, namespace, namespace)
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/dep11-0.2-py3.4.egg/EGG-INFO/scripts/dep11-generator", line 806, in <module>
    main()
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/dep11-0.2-py3.4.egg/EGG-INFO/scripts/dep11-generator", line 763, in main
    gen.process_suite(args[2])
  File "/home/ubuntu/appstream/lib/python3.4/site-packages/dep11-0.2-py3.4.egg/EGG-INFO/scripts/dep11-generator", line 254, in process_suite
    with mp.Pool(maxtasksperchild=16) as pool:
  File "/usr/lib/python3.4/hmultiprocessing/context.py", line 118, in Pool
    context=self.get_context())
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 168, in __init__
    self._repopulate_pool()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 233, in _repopulate_pool
    w.start()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.4/multiprocessing/context.py", line 281, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.4/multiprocessing/popen_forkserver.py", line 36, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.4/multiprocessing/popen_fork.py", line 21, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.4/multiprocessing/popen_forkserver.py", line 43, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/lib/python3.4/multiprocessing/spawn.py", line 173, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: 'module' object has no attribute '__spec__'

ximion / appstream-dep11

Standardize runner scripts #1