Course build performance: Kubernetes container setup changes

[markkuriekkinen]

1. NFS (network file system) is slow for writing hundreds of files

   • NFS is needed for sharing the course files across multiple K8S pods: gitmanager, huey, multiple nodes and containers
   • Consider a different storage than NFS for the course build directory
   • Does multiple pods using the same NFS directory cause trouble?

2. Experiment with RAM filesystem in the Kubernetes mounted volumes

   • Instead of reading the repo over NFS and writing build output to NFS, do a clean clone in a RAM disk and only copy output to NFS.
   • Alternatively, clone once and store a tarball on NFS, during build unpack tarball to RAM.
   • Problem: git clone and pull are run before starting the build container, which runs the course build scripts like build.sh. RAM disk can not be shared between different containers or pods (or mainly nodes?), thus the git pull and clone should be run in the same container that runs the course build script. (Currently, it is possible that the git pull phase in the gitmanager is run on a different node than the build container that follows. RAM disks can not be shared between nodes since they are on different hardware/physical servers.)
   • Consider shallow git clones: could the start of the build be faster if the course git repo were not fully cloned? Consider shallow clones of only the latest state of the target branch without the commit history.

[lainets]

The build directory is to be changed to be on a local disk. The necessary changes have been done but is yet to be deployed on production. The directory is emptied whenever the pod stops, and the files need to be cloned from git after a restart. Recloning the whole git repo is still faster on some courses when using the local disk as opposed to doing the buildi on a NFS disk.