Add shallow checkout for azure pipelines

[Trzs]

• bootstrap.py always checks out the full repository
• the azure runner is severely disk limited, so the '.git' directories are deleted again
• kokkos needs git information to be build -> unwieldy work around to delete the '.git' dir in an extra step
• I propose to add --depth=1 to git clone commands in bootstrap.py, so that only the necessary data is downloaded in the first place
• can be reversed, where necessary, with git fetch --unshallow

Note to @nksauter, @bkpoon

[phyy-nx]

$ time git clone git@github.com:cctbx/cctbx_project.git
real    0m44.325s
$ du -hs cctbx_project/
255M    cctbx_project/

$ time git clone --depth=1 git@github.com:cctbx/cctbx_project.git
real    0m12.546s
$ du -hs cctbx_project/
157M    cctbx_project/

XFEL CI on Azure is so tight on space this kinda thing can help (plus it's good dinosaur management). But how does this affect developers? With depth=1, the old commit history is literally not there. Git log shows only the latest commit. So I think we'd only want this for Azure, right? Maybe a flag to bootstrap update that we can set in Azure?

[bkpoon]

How much more disk space is needed? There is a disk clean up step in

https://github.com/cctbx/cctbx_project/blob/998327540111c7be25502f4e65f9453f92af958f/.azure-pipelines/xfel/conda-linux.yml#L112-L128

You essentially have root access to the virtual machine on Azure so you can delete whatever you want. There is sort of a limit in that in this pipeline, you are already inside the Docker image, but you can always make other host directories writeable here

https://github.com/cctbx/cctbx_project/blob/998327540111c7be25502f4e65f9453f92af958f/.azure-pipelines/xfel/conda-linux.yml#L27-L33

For example, in some pipelines, I clean up more stuff and make a swap file

https://github.com/phenix-project/phenix-installer/blob/main/scripts/clean_linux.sh

But that is done in the normal Azure image, not the Docker image. I get about 47 GB free in / after this step. Also, you do not need to use the 14 GB partition, you can use any partition on the image.

I can probably help more next week once I get back.

[Trzs]

@phyy-nx to get the full git history, you can run either git fetch --unshallow or git pull --unshallow