Add a parameter to display the progress of the kernel compilation

[FabienZa]

Problem? Depending on the configuration, different elements are compiled, this may complicate the estimation of the progress

[acherm]

Displaying the progress of the kernel compilation is not straightforward... But definitely interesting!

A way to achieve this is to predict the compilation time of a kernel. If you have the total time a configuration kernel is supposed to take, then you can easily infer the progression ;) We are actually investigating how to predict the compilation time (in the same spirit we have predicted the kernel size). A challenge is that we're using different kinds of machine, and thus it can have a strong influence (eg a 4-core machine is not the same as a 64-core machine). But there is hope since we have used a quite homogenous set of machines with similar hardware. So we will experiment with our data, and in general it can be part of the data processing axis.

Another approach is to have more fine-grained information about compilation time. Right now, it's "just" the overall time that we record. It would be super nice is to have the time at each step of the build process, in the logs.
I think we have though about it in the past, and it's not that complicated to implement I guess.

Something like (numbers invented):

[00:00:00] make: Entering directory '/TuxML/linux-4.13.3'
[00:00:00] scripts/kconfig/conf  --silentoldconfig Kconfig
[00:00:01]  SYSTBL  arch/x86/entry/syscalls/../../include/generated/asm/syscalls_32.h
[00:00:04]  SYSHDR  arch/x86/entry/syscalls/../../include/generated/asm/unistd_32_ia32.h
...
[00:00:56]  HOSTCXX -fPIC scripts/gcc-plugins/cyc_complexity_plugin.o

As such, we could identify parts of the build process that takes time (and some parts are specific to a configuration).

It can be a nice contribution to TUXML: basically we need to change the way we store logs (especially std_out_logfile), perhaps with logging facilities of Python

[FabienZa]

https://github.com/TuxML/tuxml/blob/master/compilation/compiler.py#L240-L249

       self.__logger.timed_print_output("Compilation in progress")
        failure = subprocess.call(
            args="make -C {} -j{}".format(
                self.__kernel_path,
                self.__nb_core
            ),
            shell=True,
            stdout=self.__logger.get_stdout_pipe(),
            stderr=self.__logger.get_stderr_pipe()
        )

[alxvqold]

To build the image:

(First I removed all my tuxml-related docker images: sudo docker images and sudo docker rmi ids of images to remove)

cd docker_management python3 docker_image_tuxml.py (with sudo if you have an authorisation issue) cd .. python3 kernel_generator.py --dev --local --tiny 1 (--tiny is just to have a faster compilation time)

[FabienZa]

The regeneration of docker images doesnt't work for me, I opened an issue : https://github.com/TuxML/tuxml/issues/21

[alxvqold]

Time added to stdout logs: Add time in compilation logs and print compilation logs on output

Now working on stderr logs.

[alxvqold]

Time added to stderr logs too: Added stderr logs and compilation logs no longer printed on tuxml's stdout..