Requirements for Neurodesk container / Matlab compilation development environment

[civier]

Hi @stebo85,

Today I worked with @JD-Zhu (Judy), @neuro-sevas (Steff) and @DavidjWhite33 (Dave) on compiling fieldtrip and building a container (we had to rebuild because there were some issues with the version compiled by the CI), and we came up with a shortlist of things that need to be in an ideal development environment.

You told me that you'll work on the environment, so let us know if one of those would be a limitation.

0. Possible to run on cloud (as some people don't have strong enough computers)
1. Docker (the same version that is installed in the CI)
2. Python 3 (the same version that is installed in the CI)
3. Graphical environment (to test graphics)
4. Singularity (the same version that is installed in current Neurodesktop; to test graphics, as graphics doesn't work well with docker)
5. Large root partition to hold Docker and Singularity temp files (how large can the root partition on the commercial clouds be?) OR an environment when Docker and Singularity temp files are saved to a storage not on root partition. Judy used a NECTAR instance with the default 30G and ran into "out of space" errors all the time.
6. An option to install Matlab for people that have MATLAB license

Thanks ahead, Oren

[civier]

Hi @stebo85 @aswinnarayanan @air2310

I don't know if you ended up having some standard development environment, but we are now going to do some container development work, and started to think on the best platform for such an environment.

I am considering two main options: 1) A NECTAR instance 2) A Neurodesktop version for container development

For number 2, we will have to run docker within docker. Do you have any experience with that? Any known issues? Do you foresee any other issues with developing containers within Neurodesktop, other than the fact that changes that developers would introduce in their personal environment won't be persistent, and that adding things in a persistent way would require adding stuff to the "development Neurodesktop" container recipe (we won't be able to install and configure software through GUI, e.g. MATLAB)?

I also copy @DavidjWhite33, @TomEmotion, @JD-Zhu, and @neuro-sevas as future users of such an environment.

Best, Oren

[stebo85]

I use a nectar instance for development purposes of containers and that works very well.

I also tested to run docker inside a neurodesk environment and that also works - the trick is to mount -v /var/run/docker.sock:/var/run/docker.sock into the neurodesktop container and then you can install docker inside the container and use the host's docker

[civier]

Great. I'm happy to hear that both options are feasible.

I do have some questions regarding the root partition of the NECTAR instance, if you don't mind (relevant to 1, and also to 2, if we run Neurodesktop on NECTAR): How large is the root partition you use? The default of NECTAR is 30G, which I usually find too small. If you have a larger one, who should I contact in NECTAR in order to have the option for images with larger root partition? If you do use a small root partition, did you change the location where Docker saves internal files? If you can refer me to somewhere that explains the best way to do it, I'd be happy (I once physically moved the folders to another drive, replacing them with symbolic links, but I doubt that is the recommended way).

Best, Oren

[stebo85]

I am not sure if this is the best way of doing it, but moving the docker image cache to an attached ephemeral disk works well: https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169

sudo service docker stop sudo mv /var/lib/docker /storage/scratch/docker sudo ln -s /storage/scratch/docker /var/lib/docker sudo service docker start

[aswinnarayanan]

Hi @civier, I use a nectar instance as well for development. Tbh, if I run out of space, I just prune unused docker images and layers (they can rack up quite a bit on development instances). So I haven't needed more than 30GB so far https://docs.docker.com/engine/reference/commandline/image_prune/

Agreed with @stebo85's points above.

I think Nectar images have limited root partitions. You can request a Volume allocation. Then create a Volume from your nectar project's volume panel, and attach it to your instance. It attaches as a block volume device, so shouldn't be any performance penalties. You'll also need to mount the volume block from the OS, maybe using fstab. Ive pasted a example /etc/fstab on ubuntu for volume vdb below.

[stebo85]

Feedback why building containers is hard for neurodesk @korbinian90 @iishiishii:

• writing dockerfiles is hard -> neurodocker helps us, but it is confusing for users because they don’t know how it works and what it does; idea: hide this process better :? Make dockerfiles available, so people understand what neurodocker is building?
• building containers needs a lot of storage -> developers run out of disk space; build on neurodesk play; build singularity container fully interactively -> extract history -> submit as recipe -> look at BUILDAH (podman?)
• multiple workflows how to build and test -> #should there only be one?
• deploy path, deploy bin is confusing -> better documentation?
• functionality testing: automate testing -> we need some test data -> we need test scripts -> test at end of interactive session

[stebo85]

we build a prototype and will document in a github repository

[stebo85]

completed: https://www.neurodesk.org/docs/neurocontainers/contribute/