Does BuildKit handle the /var directory differently from Kaniko?

[marcodelapierre]

For the record, I have run some Wave tests where, following the upgrade of the container build system, I am having errors accessing files stored under /var. Two examples from the same test with Spack (was working with a previous Kaniko based deployment):

#12 6.122 ==> Error: Command exited with status 2:
#12 6.122 '/usr/bin/gpg2' '--with-colons' '/var/seqera/spack/key'
#12 6.122 gpg: keybox '/opt/spack/opt/spack/gpg/pubring.kbx' created
#12 6.122 gpg: WARNING: no command supplied.  Trying to guess what you mean ...
#12 6.122 gpg: can't open '/var/seqera/spack/key'
#12 6.122 
#12 9.939 ==> Error: [Errno 2] No such file or directory: '/var/run/secrets/[eks.amazonaws.com/serviceaccount/token'](http://eks.amazonaws.com/serviceaccount/token')

The 2 errors are

• gpg: can't open '/var/seqera/spack/key'
• No such file or directory: '/var/run/secrets/eks.amazonaws.com/serviceaccount/token'

I am wondering whether setup of /var requires some extra configuration with BuildKit.

[munishchouhan]

I think we are using rootless buildkit image, which may be causing this issue

[marcodelapierre]

Thank you Munish.

I also have a simple way to reproduce on production Wave, which was not erroring with Kaniko (Wave 1.7.9):

export TOWER_ACCESS_TOKEN=< ... >
nextflow run marcodelapierre/toy-cowsay-nf -profile spackwave

This will result in the following error in the Wave build log:

#12 24.39 ==> Error: Command exited with status 2:
#12 24.39 '/usr/bin/gpg2' '--with-colons' '/var/seqera/spack/key'
#12 24.39 gpg: keybox '/opt/spack/opt/spack/gpg/pubring.kbx' created
#12 24.39 gpg: WARNING: no command supplied.  Trying to guess what you mean ...
#12 24.39 gpg: can't open '/var/seqera/spack/key'
#12 24.39 
#12 27.09 gpg: key A8E0CA3C1C2ADA2F: 7 signatures not checked due to missing keys
#12 27.09 gpg: key A8E0CA3C1C2ADA2F: public key "Spack Project Official Binaries " imported
#12 27.09 gpg: Total number processed: 1
#12 27.09 gpg:               imported: 1
#12 27.09 gpg: no ultimately trusted keys found
#12 27.10 gpg: inserting ownertrust of 6
#12 27.31 ==> Error: [Errno 2] No such file or directory: '/var/run/secrets/[eks.amazonaws.com/serviceaccount/token'](http://eks.amazonaws.com/serviceaccount/token')

[marcodelapierre]

Leaving this issue for the record, as issues in using the /var in Wave container builds might manifest in the future in other scenarios.

At this stage, the cause seems likely to be in the unpriviliged usage mode of BuildKit.