Add tasks feature to Craftr 2

[NiklasRosenstein]

Craftr 1 had the "RTS" feature where you could make actual Python functions in your build script be embedded into the Ninja build graph. I say we need this in Craftr 2 as well, it only needs to be implemented a bit better. It should not have the same limitations as the old implementation:

• It should not need to patch sys.stdin and sys.stdout globally (see #102)
• It should allow true parallel execution without threading limitations

We can achieve this by serialising all Python objects involved in the export process, including functions (which are used for tasks). Ninja would invoke a Python command that deserialises the Python objects and call the respective function (i.e. task).

TODO: Think of a design to implement tasks and reflect them as Target objects

[NiklasRosenstein]

So the serialise approach is very troublesome because it depends a lot on Python's import mechanism which Craftr doesn't use. Also there's quite a number of limitations on what can be pickled and what not. IF we're going to pickle a function for a Craftr Task, we need to pickle all of the build environment, which is unfeasable. If only one module exports a member with an unpickable object, pickling fails. And right now, even some of the Craftr built-ins are unpickable.

The current solution is a bit slower. Ninja calls craftr run <task> <task_args...> to execute a specific task. Unfortunately that get's us to re-execute the whole build script every single time. However, there are some optimizations to be made (and are already made) when craftr run is used rather than craftr export. The run command does not export any files (it's not supposed to, modules might still do it if they don't implement it correctly, todo needs to be documented how to avoid writing files in craftr run).

The advantage is that it's pretty straight forward implementation-wise. There's no socket communication happening to the master Craftr process, also we get true multiprocessing because there can be multiple Craftr processes spawned at the same time. With the socket approach from Craftr 1, all tasks end up running in the same Craftr process, and Python is not truly multithreaded.

This could have been circumvented by forking the Craftr process a few times and going back to the way Craftr 1 implemented tasks (socket), but this feature can only be available on Unix-based systems (or systems that truly support process forks).

[NiklasRosenstein]

Craftr 3 does not support "tasks" in the fashion that Craftr 2 and 1 did (that is, declaring Python functions inline with the build script that can then be embedded in the build graph). In Craftr 3, you would write a separate script file that you can target using

import nodepy.runtime
import craftr, {path} from 'craftr'

task_filename = str(module.directory.joinpath('tasks/preprocess.py'))

build_directory = craftr.Namespace.current().build_directory
sources = craftr.glob('*.cpp')
preprocessed_sources = craftr.relocate_files(sources, path.join(build_directory, 'preprocessed_sources'), '.cpp')

craftr.gentarget(
  name = 'mytask',
  commands = [nodepy.runtime.exec_args + [task_filename, '$in', '$out']],
  input_files = sources,
  output_files = preprocessed_sources,
  foreach = True
)

# ...

Maybe a simpler interface to embedd Python scripts into the build graph would be nice in the future.