BuildStream refuses to cache artifacts with files that have very strict permissions #429

Open Cynical-Optimist opened 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @tlater] on Jun 18, 2018, 15:24

Summary

When trying to build an element that produces an artifact which contains a file without write permissions, BuildStream will fail to cache the artifact.

This is particularly annoying when trying to depend on a base platform with, say, a /etc/shadow file.

Steps to reproduce

Create a simple BuildStream project.
Create a tarfile containing a file with its permissions set to 400.

Create a .bst file similar to:

   kind: import
   sources:
   - kind: tar
     base-dir: ''
     url: file:///src/src/permissions.tar.gz

Attempt to build the element.

What is the current bug behavior?

BuildStream will fail to cache the resulting files in an artifact.

What is the expected correct behavior?

BuildStream caches the resulting artifact.

Relevant logs and/or screenshots

The error message given when BuildStream attempts to cache:

[--:--:--] START   [readonly.bst] : Staging sources
[00:00:00] SUCCESS [readonly.bst] : Staging sources
[--:--:--] START   [readonly.bst] : Caching artifact
[00:00:00] FAILURE [readonly.bst] : Caching artifact
[00:00:00] FAILURE [readonly.bst] : Failed to commit artifact: openat(none): Permission denied
======================================================================

Other relevant information

BuildStream version affected: /milestone %"BuildStream_v1.1"
BuildStream version affected: /milestone %"BuildStream_v1.2"
BuildStream version affected: /milestone %"BuildStream_v1.3"

In GitLab by [Gitlab user @tlater] on Jun 18, 2018, 15:25

changed the description

In GitLab by [Gitlab user @tristanvb] on Jun 19, 2018, 01:04

Because tar thinks it is allowed to do something if it is uid 0 ?

Or is this OSTree failing to commit readonly files ? I somehow doubt this, i think i've seen similar OSTree issues fixed.

In GitLab by [Gitlab user @tlater] on Jun 19, 2018, 10:53

[Gitlab user @tristanvb] not having had a much deeper look into this, I suspect it's on the OSTree side, since the source has already been checked out. The error happens when the artifact is committed to the cache.

This might also happen when we copy between temporary directories, or something, but I doubt it's an issue with tar, especially as I've seen it happen with docker and file sources as well.

In GitLab by [Gitlab user @tlater] on Jun 25, 2018, 11:34

marked this issue as related to bst-external#3

In GitLab by [Gitlab user @tlater] on Sep 25, 2018, 14:11

This appears to be fixed with the introduction of CAS :)

I'm not sure this can be backported, so I will close it for now.

In GitLab by [Gitlab user @tlater] on Sep 25, 2018, 14:11

closed

In GitLab by [Gitlab user @tlater] on Sep 25, 2018, 15:22

mentioned in merge request bst-external!40

In GitLab by [Gitlab user @hdonnay] on Sep 27, 2018, 16:02

I've just hit this (or an issue very similar) when trying to use the CentOS OSTree repo -- usr/bin/ssh-agent is set to 0111 and causes BuildStream to error out.

In GitLab by [Gitlab user @LaurenceUrhegyi] on Sep 27, 2018, 16:38

[Gitlab user @hdonnay] did you get the exact same error message?

Re-opening.

In GitLab by [Gitlab user @LaurenceUrhegyi] on Sep 27, 2018, 16:38

reopened

In GitLab by [Gitlab user @hdonnay] on Sep 27, 2018, 17:39

No, just a similar permissions message. I've attached a reproducing setup; perhaps someone with more familiarity can determine if this is the same issue or a new one.

In GitLab by [Gitlab user @juergbi] on Sep 27, 2018, 17:53

[Gitlab user @hdonnay] The error in your setup happens when staging sources, not when caching the artifact. It's either an issue in the OSTree source plugin or in BuildStream core, as far as I can tell. As it's not quite the same issue, please open a new one. I'm closing this again.

In GitLab by [Gitlab user @juergbi] on Sep 27, 2018, 17:53

closed

In GitLab by [Gitlab user @hdonnay] on Sep 27, 2018, 17:56

Sorry, the logs printed for me seem to indicate it's the caching:

        [--:--:--] START   centos.bst: Build
        [--:--:--] LOG     centos.bst: Build environment for element centos.bst

            HOME: /tmp
            LC_ALL: C
            LOGNAME: tomjon
            PATH: /usr/bin:/bin:/usr/sbin:/sbin
            SHELL: /bin/sh
            SOURCE_DATE_EPOCH: '1320937200'
            TERM: dumb
            TZ: UTC
            USER: tomjon
            USERNAME: tomjon
        [--:--:--] START   centos.bst: Staging sources
        [--:--:--] START   centos.bst-0: Staging ref: b62513420243def54db7d6c36a79d8f0eda94c5ac410c1f64ee6e0b44e5dca99 from origin: http://mirror.centos.org/centos/7/atomic/x86_64/repo/
        [00:00:08] SUCCESS centos.bst-0: Staging ref: b62513420243def54db7d6c36a79d8f0eda94c5ac410c1f64ee6e0b44e5dca99 from origin: http://mirror.centos.org/centos/7/atomic/x86_64/repo/
        [00:00:08] SUCCESS centos.bst: Staging sources
        [--:--:--] START   centos.bst: Caching artifact
        [00:00:02] FAILURE centos.bst: Caching artifact
        [00:00:12] FAILURE centos.bst: Failed to hash object: [Errno 13] Permission denied: '/home/hank/.cache/buildstream/build/centos-upncikpw/artifact/files/usr/bin/ssh-agent'

In GitLab by [Gitlab user @juergbi] on Sep 27, 2018, 18:02

BuildStream master already fails staging sources while 1.2 fails caching the artifact. I've opened #684 for the former issue (master only). The latter issue (also in 1.2) fits here.

In GitLab by [Gitlab user @juergbi] on Sep 27, 2018, 18:02

reopened

In GitLab by [Gitlab user @toscalix] on Sep 28, 2018, 13:58

marked this issue as related to #684

In GitLab by [Gitlab user @jmacarthur] on Oct 1, 2018, 16:00

mentioned in merge request !837

buildstream-migration / bst-staging