New idea: using docker containers for mappers

[magicDGS]

I though that using docker containers for Distmap might be a good idea for remove the dependency of binary files that are system dependent. The main idea is:

• Substitute the --mapper-path option for a --mapper-docker one. This might allow to use exactly the same binaries independent of the computers on the Hadoop cluster (e.g., allowing machines with different systems being part of the same cluster).
• Every node should have installed docker, and also every local computer running distmap. Distmap should be sure that the docker container contains the expected mapper binary in the root (e.g., bwa or bowtie2).
• Each map task script (e.g., bwa_mapping.pl command line will use docker run for running the command. The difference will be in arguments. This might allow to pull out common code constructing the command line and running it.
• The container is distributed to each node that will perform the map task the same as the genome reference index.
• Actually, we can even create a docker image on the fly for the requested mapper in the user PATH if no docker is provided; the base image can be linux/macos depending on the caller system, allowing to support calls from both environments. This is in line with @robmaz idea of using the PATH for looking for binaries but adding also the operating system to the base image to allow compatibility.

What do you think about this, @robmaz?

[robmaz]

While a good idea in principle, and maybe the way to go in the future, the current problem is that recent Docker versions are not supported on our many MacOS 10.8.5 nodes.

[magicDGS]

Oops, I didn't know that!

On the other hand, I suggest that our MacOS system should be updated anyway soon, mostly due to security issues from the spectre/meltdown bug. Thus, the problem with incompatibilities will be solve soon.

[magicDGS]

Another problem: it looks that it is unsafe to run docker on Hadoop, but 3.0.0 would bring support for that (still experimental): https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/DockerContainers.html

Another reference: https://thenewstack.io/docker-hadoop-theres-good-bad-ugly/