Set up a docker (or similar) environment, both for running tests and for providing an easy way for a user to get XGA

[DavidT3]

This will be great, largely because of the need for XSPEC and SAS to be installed. That means I will be able to use CI tests easily in theory, and if the environment is setup properly then the user can download it and use it for their analysis

[DavidT3]

This is finally getting done because people are having trouble installing HEASoft + SAS in some cases (don't blame them it's a nightmare) - I think I will create a docker container as you can also use Docker containers with singularity.

I'll then serve it using Docker Hub, which is free for this use case.

The container will be created with the latest versions of SAS and HEASoft available at the time, and then I'll have to come up with a schedule for semi-regularly updating things - I will also have to do the same for updating based on XGA versions

[DavidT3]

I'm approaching this by modifying the official dockerfile that was created for HEASOFT. I will add the abilities to download and install anaconda, SAS, and a stripped down version of HEASOFT (I figured out how to do that without going through the HEASOFT selection process where you select what components you want). The strippped version of HEASoft will probably be downloaded through a Python script:

import requests import shutil

with requests.get('https://heasarc.gsfc.nasa.gov/cgi-bin/Tools/tarit/tarit.pl?mode=download&arch=src&src_pc_linux_centos=Y&src_other_specify=&general=attitude&general=heasarc&general=heasptools&general=heatools&general=heagen&xanadu=xspec', stream=True) as streamo: with open('testo.tar.gz', 'wb') as writo: shutil.copyfileobj(streamo.raw, writo)

This Dockerfile will also have to be multi-platform, able to be built both for AMD64 and ARM64 to account for the fact that many Astronomers are using (or will transition to using) ARM based processors such as the Apple M1 chip.

Currently many of the checks of platform and architecture are done in the Dockerfile, but I apparently best practise is to put things like that in a Bash script, so I might well end up doing that.

[DavidT3]

I'm slowly getting a handle on this, but am running into a problem where the make install command (I think? it's possible it could be the make command) for building/installing HEASoft fails (or rather seems to just hang) when it runs as a result of a command in the dockerfile. However, if I successfully build an image that stops the step before building and install HEASoft it seems to work when I log in and do it the 'normal' way.

I'm somewhat running out of ideas as to why this is, but I'll start by minimising the differences between the official HEASoft dockerfile and the bastardised version I'm now working with. I.e. perhaps I'll stop using conda (though I don't want that to be a permanent solution).

One thing I'm discovering is that it's really hard to diagnose build problems that happen during a dockerfile build...

[DavidT3]

I got it to successfully build an image with HEASoft built + installed. This was achieved with a docker buildx feature that created a build container with a larger limit on the log output. I don't know why this worked when it didn't previously, but when I spun up the image and logged in I could use XSPEC (after adding the HEASoft setup script to the .bashrc, that'll be done automatically once the Dockerfile is finished).

The commands were (I think the first one just sets up the buildkit and only needs to be run once, unless another build container is defined then you might need to switch between them):

docker buildx create --use --name larger_log --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=50000000 docker buildx build . -t xga_first_tests --no-cache

It took ~36 mins to build.

[DavidT3]

Moving onto the problem of SAS...

Unfortunately SAS is only released as pre-compiled binaries, and they are compiled on x86 processors. Apple Macs can use the Rosetta 2 emulation layer to run them, though Rosetta is not available for Linux - some form of emulation is required.

A native ARM version is being worked on, per my talking with the XMM help desk, but is not available yet - and is currently only planned for the Mac version.

The only solution is for me to build an x86 version of the image only, and then anyone with an M1 mac or a ARM server that wants to run the image will just have to use docker emulation.

[DavidT3]

Unfortunately even though the Dockerfile I put together was building yesterday, it doesn't seem to want to build today. Admittedly yesterday was building just for ARM without using buildx, but I don't know why it just seems to hang on the HEASoft build now. I did make some very limited changes to the dockerfile to try and decrease the number of layers, but I think I've undone them now and it still isn't working. This is incredibly frustrating

[DavidT3]

So the x86 (AMD64 specifically) version of the image did eventually build on my M1 Macbook - it just took nearly 5 hours so some patience was required. I'll upload to ICER and see whether it runs in a singularity instance!