make (in top directory) fails to build container

[yarikoptic]

ended up with

❯ make
podman build .  \
    -f code/opfvta-images/Containerfile \
    -t docker.io/asmacdo/opfvta:2.0.0-alpha
[1/2] STEP 1/1: FROM docker.io/gentoo/portage:20230424 AS portage
--> 52467ee4da7
[2/2] STEP 1/18: FROM docker.io/gentoo/stage3:20230424
...
[2/2] STEP 17/18: RUN pushd /var/db/repos/local_opfvta/sci-misc/opfvta; ebuild opfvta-99999.ebuild manifest; popd
/var/db/repos/local_opfvta/sci-misc/opfvta /
/usr/lib/portage/python3.10/ebuild.sh: line 628: /var/db/repos/local_opfvta/sci-misc/opfvta/opfvta-99999.ebuild: Permission denied
 * ERROR: sci-misc/opfvta-99999::local_opfvta failed (depend phase):
 *   error sourcing ebuild
 * 
 * Call stack:
 *   ebuild.sh, line 628:  Called die
 * The specific snippet of code:
 *              source "${EBUILD}" || die "error sourcing ebuild"
 * 
 * If you need support, post the output of `emerge --info '=sci-misc/opfvta-99999::local_opfvta'`,
 * the complete build log and the output of `emerge -pqv '=sci-misc/opfvta-99999::local_opfvta'`.
 * Working directory: '/usr/lib/python3.10/site-packages'
 * S: '/var/tmp/portage/sci-misc/opfvta-99999/work/opfvta-99999'
/
--> d7ff9500835
[2/2] STEP 18/18: RUN emerge -v opfvta --autounmask-continue

Performing Global Updates
(Could take a couple of minutes if you have a lot of binary packages.)
  .='update pass'  *='binary update'  #='/var/db update'  @='/var/db move'
  s='/var/db SLOT move'  %='binary move'  S='binary SLOT move'
  p='update /etc/portage/package.*'
/var/db/repos/gentoo/profiles/updates/2Q-2023.
/var/db/repos/gentoo/profiles/updates/3Q-2021..............

These are the packages that would be merged, in order:

Calculating dependencies  
 * IMPORTANT: 1 news items need reading for repository 'science'.
 * IMPORTANT: 14 news items need reading for repository 'gentoo'.
 * Use eselect news read to view new items.

. * Manifest not found for '/var/db/repos/local_opfvta/sci-misc/opfvta/opfvta-99999.ebuild'
... done!
Dependency resolution took 9.99 s.

!!! All ebuilds that could satisfy "opfvta" have been masked.
!!! One of the following masked packages is required to complete your request:
- sci-misc/opfvta-99999::local_opfvta (masked by: corruption)

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

Error: building at STEP "RUN emerge -v opfvta --autounmask-continue": while running runtime: exit status 1
make: *** [Makefile:29: build] Error 1
make  237.64s user 197.44s system 74% cpu 9:44.56 total

Shouldn't we have that container image shared and by default just downloaded/used?

[asmacdo]

make run will do that, but make will also run the build. Ill cover that in the README.

Let's leave this issue open though, make build should still work.

[asmacdo]

@TheChymera sci-misc/opfvta-99999::local_opfvta (masked by: corruption)

[TheChymera]

failed to reproduce this bug on commit 3599d57276f0302202d29b5d592ad679716f6e13 @yarikoptic can you please check the output of the following commands match?

[deco]/tmp/opfvta-replication-2023 ❱ ls -lah code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild
-rw-r--r-- 1 chymera chymera 756 May 25 23:51 code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild
[deco]/tmp/opfvta-replication-2023 ❱ cat code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild
# Copyright 1999-2023 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=8

DESCRIPTION="Whole-Brain Map and Assay Analysis of Mouse VTA Dopaminergic Activation"
HOMEPAGE="https://bitbucket.org/TheChymera/opfvta"

LICENSE="GPL-3"
SLOT="0"
IUSE="scanner-data"
KEYWORDS=""

DEPEND=""
RDEPEND="
    >=dev-python/repsep_utils-0.3.1
    >=dev-python/seaborn-0.9.0
    >=dev-python/statsmodels-0.9.0
    >=dev-tex/pythontex-0.16
    app-text/texlive[publishers,science,xetex]
    dev-python/matplotlib
    dev-python/numpy
    dev-python/pandas
    dev-tex/biblatex
    dev-tex/rubber
    media-gfx/graphviz
    sci-biology/ABI-connectivity-data
    sci-biology/samri
    scanner-data? ( sci-biology/opfvta_data )
    !scanner-data? ( sci-biology/opfvta_bidsdata )
"

[yarikoptic]

here is what I see if I go to that folder now (wow -- time stamp from May 1st, awhile back)

```shell ❯ cd ~/proj/CON/opfvta-replication-2023 Makefile README.md code/ inputs/ outputs/ paper/ ❯ ls -lah code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild -rw------- 1 yoh yoh 756 May 1 10:10 code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild ❯ cat code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild # Copyright 1999-2023 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 EAPI=8 DESCRIPTION="Whole-Brain Map and Assay Analysis of Mouse VTA Dopaminergic Activation" HOMEPAGE="https://bitbucket.org/TheChymera/opfvta" LICENSE="GPL-3" SLOT="0" IUSE="scanner-data" KEYWORDS="" DEPEND="" RDEPEND=" >=dev-python/repsep_utils-0.3.1 >=dev-python/seaborn-0.9.0 >=dev-python/statsmodels-0.9.0 >=dev-tex/pythontex-0.16 app-text/texlive[publishers,science,xetex] dev-python/matplotlib dev-python/numpy dev-python/pandas dev-tex/biblatex dev-tex/rubber media-gfx/graphviz sci-biology/ABI-connectivity-data sci-biology/samri scanner-data? ( sci-biology/opfvta_data ) !scanner-data? ( sci-biology/opfvta_bidsdata ) " ```

[TheChymera]

❯ ls -lah code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild
-rw------- 1 yoh yoh 756 May  1 10:10 code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild

vs

[deco]/tmp/opfvta-replication-2023 ❱ ls -lah code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild
-rw-r--r-- 1 chymera chymera 756 May 25 23:51 code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild

I am pretty sure it's the permissions

[yarikoptic]

Indeed my default umask is quite restrictive (077) and by default disables group/others permissions to even read. But then it means that there is no proper uid mapping of mine into podman instance invocation, or there is a need for an explicit umask call within Makefile to ensure some expected "wider" permissions than just for an original user... (would not help since we are talking about permissions of the files in the original git clon'ed copy outside of the container). I will give some tries now.

[TheChymera]

@yarikoptic in case you don't get to this I'll also try, just that I ran into another issue on this repo that needs fixing first. If you get anything regarding flit_core/flit-core just ignore it for now.

[TheChymera]

Ok, so make builds the image fine on c648945fe72bf36d0aa09201aa62d85b915965ae (aforementioned flit{_,-}core now irrelevant). Testing again with 600 permissions.

[TheChymera]

@yarikoptic

ok, was able to reproduce the bug, it's the file permissions. Set the ebuild file to 644 or higher. No idea why it's more restricted on your user... It's 644 for me out of the box.

These are the packages that would be merged, in order:

Calculating dependencies
 * IMPORTANT: 1 news items need reading for repository 'science'.
 * IMPORTANT: 13 news items need reading for repository 'gentoo'.
 * Use eselect news read to view new items.

.. . * Manifest not found for '/var/db/repos/local_opfvta/sci-misc/opfvta/opfvta-99999.ebuild'
... done!
Dependency resolution took 28.93 s.

!!! All ebuilds that could satisfy "opfvta" have been masked.
!!! One of the following masked packages is required to complete your request:
- sci-misc/opfvta-99999::local_opfvta (masked by: corruption)

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

Error: building at STEP "RUN emerge -v opfvta --autounmask-continue": while running runtime: exit status 1
make: *** [Makefile:29: build] Error 1
[typhon]~/docsrc/opfvta-replication-2023 ❱ ls code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild  -lah
-rw------- 1 chymera chymera 756 May 25 23:37 code/opfvta/.gentoo/overlay/sci-misc/opfvta/opfvta-99999.ebuild

[yarikoptic]

It is such because of my umask. May be make is doomed to set permissions right where it needs files to be accessible by any other user

[TheChymera]

We can chmod in either the Makefile or the Containerfile if we want to specifically address this issue. I think the best way is to do this in the Containerfile (testing right now). The core problem here is that in the image the file is accessible to root but not to portage which is the sandbox user that builds everything.

More generally, not sure what the standard operating procedure is with respect to managing permissions across git, since it only tracks the executable bit. I guess this is where we encounter the issue that git is not really a package management tool, and this is one of the cases where it is inferior to tarballs :3 Alternatively, maybe your umask is “broken” with regard to multi-user usage? I can think of a lot of failure points for a lot of software with umask 077.

[TheChymera]

@yarikoptic I am fairly sure the current issue is solved by b3bc390c23c721d9c530f9acb98b7db17ae4dbea . Please confirm.

Having said that, I think the general notion of umask-resistant permissions needs some more thought, but probably better deferred to when it crops up again.

[yarikoptic]

Left a comment on the commit

[yarikoptic]

FTR, running now

yoh@typhon:/data/tmp/opfvta-replication-2023$ make

on a fresh clone... if succeeds, will close this issue

[yarikoptic]

ok, it did complete after first failing since I didn't do git submodule update --init or datalad -r so it was lacking some submodules needed for the build. So this issue should be closed either by adding some explicit Makefile git submodule init on needed submodule(s) or instructions in README (attn @asmacdo ref #2 )

[TheChymera]

It no longer fails, and if it should fail again it would probably be another issue, as the make system has changed substantially.