Add a more comprehensive reference for writing manifests

[ndusek]

The documentation on writing your own manifest files is a little sparse. In reading it, a couple questions come to mind:

1. What if you have your container engine set to singularity in your Bulker config? Do you still use docker_image: and docker_command: to specify the container information? Or should this be singularity_image: and singularity_command:?
2. The documentation on configuring Bulker indicates that you can use a local singularity_image_folder from which to run containers. Do you need to do anything differently with docker_image: in your package manifest to get Bulker to use a local image, rather than pull from a registry?
3. Any other differences users should be aware of when using singularity as the container engine vs docker? Most of the examples reference docker. Even having a singularity version of the cowsay manifest might help users understand the differences between the two (if any).

[nsheff]

Thanks for the questions. I will work on the documentation.

What if you have your container engine set to singularity in your Bulker config? Do you still use docker_image: and docker_command: to specify the container information? Or should this be singularity_image: and singularity_command:?

Yes, in fact the same manifest is intended to be used with either docker or singularity. That's one of the points of bulker. So, we have a bunch of manifests available on hub.bulker.io/... For example, this pepatac manifest. I use this with singularity all the time.

The documentation on configuring Bulker indicates that you can use a local singularity_image_folder from which to run containers. Do you need to do anything differently with docker_image: in your package manifest to get Bulker to use a local image, rather than pull from a registry?

I've always just used the docker images and had bulker build singularity images from them automatically. The point of the package manifest is to be portable, so it wasn't really intended to point at a local thing. As I think about it, I think the manifest really can only point to docker containers, which are then just built by singularity. If you wanted to use your own local singularity containers, you could do that as well, just by putting the containers in the folder structure that bulker would build automatically -- that is {simages}/{namespace}/{image_name}:{tag} -- where simages is the path in your bulker config (may be relative to the bulker config file, namespace is the namespace for the docker/singularity image, and tag is optional.

Any other differences users should be aware of when using singularity as the container engine vs docker? Most of the examples reference docker. Even having a singularity version of the cowsay manifest might help users understand the differences between the two (if any).

Everything should be the same. I think bulker with singularity will work exactly the same as all the tutorials showing it working with docker. I use bulker with singularity all the time just following the same functions and it works like a charm. So if you run into particular issues, let me know.

[nsheff]

Here is the new explanation I added to the docs:

Manifests and singularity

The examples above are docker-centric, but if you're using singularity, you may wonder if you have to do anything different in your manifest. The answer is not really -- one of the goals of bulker is to be able to work with a single manifest, and have that manifest work with either docker or singularity. So the idea is to create a single manifest, and it should just work with both.

That said, there is some potential for confusion. First, the bulker manifest points at docker images. When you're using singularity, bulker will use singularity to convert these into singularity images -- but they have to be docker images to begin with. This is how bulker manages to use a single manifest for either system. If I allowed you to use singularity images instead of docker images in the bulker manifest, then I'd need to convert those into docker images for the manifest to work with docker. It's just easier the other way around, because singularity has easy built-in methods to use docker containers, so that's what bulker uses.

Second, the docker_args argument is only used with docker. In my use cases, I've never had a need to do something like this with singularity, because the nuances of the docker flags I use like -i and -t aren't relevant in a singularity command. Nevertheless, there is an analogous singularity_args argument if you find there are some singularity-specific flags you need to pass.

Similarly, in an ideal world, you'd never need to use docker_command because the command would be the same. But for some containers, depending on how it uses a docker ENTRYPOINT, it might be required to tweak the command that is sent to docker. For singularity, bulker will try to use the same command specified under docker_command, but if you really have a situation where you need the command to differ between docker and singularity, the you can use singularity_command to specify the difference. I don't believe I've ever had to use this, though.

In conclusion, all these example manifests should work in the exact same way with either docker or singularity. The reason the manifests appear docker-centric is that bulker considers the docker image the ground truth, and then uses singularity to build from that image, rather than the other way around -- but in no way does this mean you can't use singularity with bulker; as mentioned, that's one of the main strengths of bulker.

[ndusek]

@nsheff Great explanation, this clears things up significantly.