CommandError / permission denied error when running deb-bootstrap-builder (on Singularity 2)

[mslw]

I was running

datalad deb-configure-builder -d builder dockerbase=debian:bullseye  # all ok
datalad deb-bootstrap-builder -d builder

The second command prompted for sudo password (*within run command and printed all the regular datalad run & singularity outputs, ending with:

Singularity container built: envs/singularity-amd64.sif
Cleaning up...
[INFO   ] == Command exit (modification check follows) ===== 
run(ok): /home/mszczepanik/Documents/hackathon-datalad-debian/mydistrib/builder (dataset) [sudo singularity build --force envs/sing...]
[WARNING] Received an exception CommandError(CommandError: 'git -c diff.ignoreSubmodules=none annex add --json --json-error-messages -c annex.dotfiles=true -- envs/singularity-amd64.sif' failed with exitcode 1 under /home/mszczepanik/Documents/hackathon-datalad-debian/mydistrib/builder [info keys: stdout_json]
>  envs/singularity-amd64.sif: setFileMode: permission denied (Operation not permitted) [err: 'add: 1 failed']). Canceling not-yet running jobs and waiting for completion of running. You can force earlier forceful exit by Ctrl-C. 
[INFO   ] Canceled 0 out of 0 jobs. 0 left running. 
Total: 0.00 datasets [00:00, ? datasets/s]CommandError: 'git -c diff.ignoreSubmodules=none annex add --json --json-error-messages -c annex.dotfiles=true -- envs/singularity-amd64.sif' failed with exitcode 1 under /home/mszczepanik/Documents/hackathon-datalad-debian/mydistrib/builder [info keys: stdout_json]
>  envs/singularity-amd64.sif: setFileMode: permission denied (Operation not permitted)
add: 1 failed

The singularity-amd64.sif file ends up created, but untracked by DataLad. It's owned by root:

ls -lh builder/envs/        
-rw-r--r-- 1 mszczepanik mszczepanik   46 Jul 11 12:35 README.md
-rwxr-xr-x 1 root        root        201M Jul 11 12:38 singularity-amd64.sif

I'm on Singularity 2.6.1-dist, DataLad 0.17 and git-annex 10.20220322-1

Full WTF output

``` # WTF ## configuration ## credentials - keyring: - active_backends: - SecretService Keyring - PlaintextKeyring with no encyption v.1.0 at /home/mszczepanik/.local/share/python_keyring/keyring_pass.cfg - config_file: /home/mszczepanik/.config/python_keyring/keyringrc.cfg - data_root: /home/mszczepanik/.local/share/python_keyring ## datalad - version: 0.17.0 ## dataset - branches: - git-annex@7247bd6 - main@035d21a - id: 863397f4-3252-4592-aa75-c22c1c4b29fc - metadata: - path: /home/mszczepanik/Documents/hackathon-datalad-debian/mydistrib - repo: AnnexRepo ## dependencies - annexremote: 1.6.0 - boto: 2.49.0 - cmd:7z: 16.02 - cmd:annex: 10.20220322-1~ndall+1 - cmd:bundled-git: 2.30.2 - cmd:git: 2.30.2 - cmd:ssh: 8.4p1 - cmd:system-git: 2.30.2 - cmd:system-ssh: 8.4p1 - humanize: 4.2.3 - iso8601: 1.0.2 - keyring: 23.6.0 - keyrings.alt: 4.1.0 - msgpack: 1.0.4 - platformdirs: 2.5.2 - requests: 2.28.1 ## environment - LANG: en_US.UTF-8 - LANGUAGE: en_US.UTF-8 - LC_ALL: en_US.UTF-8 - LC_CTYPE: en_US.UTF-8 - PATH: /home/mszczepanik/.virtualenvs/datalad-debian-dev/bin:/home/mszczepanik/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games:/home/mszczepanik/.local/bin:/home/mszczepanik/.local/share/gem/ruby/2.7.0/bin ## extensions - container: - description: Containerized environments - entrypoints: - datalad_container.containers_add.ContainersAdd: - class: ContainersAdd - load_error: None - module: datalad_container.containers_add - names: - containers-add - containers_add - datalad_container.containers_list.ContainersList: - class: ContainersList - load_error: None - module: datalad_container.containers_list - names: - containers-list - containers_list - datalad_container.containers_remove.ContainersRemove: - class: ContainersRemove - load_error: None - module: datalad_container.containers_remove - names: - containers-remove - containers_remove - datalad_container.containers_run.ContainersRun: - class: ContainersRun - load_error: None - module: datalad_container.containers_run - names: - containers-run - containers_run - load_error: None - module: datalad_container - version: 1.1.6 - datalad_debian: - description: DataLad for working with Debian packages - entrypoints: - datalad_debian.bootstrap_builder.BootstrapBuilder: - class: BootstrapBuilder - load_error: None - module: datalad_debian.bootstrap_builder - names: - deb-bootstrap-builder - deb_bootstrap_builder - datalad_debian.build_package.BuildPackage: - class: BuildPackage - load_error: None - module: datalad_debian.build_package - names: - deb-build-package - deb_build_package - datalad_debian.configure_builder.ConfigureBuilder: - class: ConfigureBuilder - load_error: None - module: datalad_debian.configure_builder - names: - deb-configure-builder - deb_configure_builder - datalad_debian.new_distribution.NewDistribution: - class: NewDistribution - load_error: None - module: datalad_debian.new_distribution - names: - deb-new-distribution - deb_new_distribution - datalad_debian.new_package.NewPackage: - class: NewPackage - load_error: None - module: datalad_debian.new_package - names: - deb-new-package - deb_new_package - load_error: None - module: datalad_debian - version: 0+untagged.56.g73e9d09 ## git-annex - build flags: - Assistant - Webapp - Pairing - Inotify - DBus - DesktopNotify - TorrentParser - MagicMime - Feeds - Testsuite - S3 - WebDAV - dependency versions: - aws-0.22 - bloomfilter-2.0.1.0 - cryptonite-0.26 - DAV-1.3.4 - feed-1.3.0.1 - ghc-8.8.4 - http-client-0.6.4.1 - persistent-sqlite-2.10.6.2 - torrent-10000.1.1 - uuid-1.3.13 - yesod-1.6.1.0 - key/value backends: - SHA256E - SHA256 - SHA512E - SHA512 - SHA224E - SHA224 - SHA384E - SHA384 - SHA3_256E - SHA3_256 - SHA3_512E - SHA3_512 - SHA3_224E - SHA3_224 - SHA3_384E - SHA3_384 - SKEIN256E - SKEIN256 - SKEIN512E - SKEIN512 - BLAKE2B256E - BLAKE2B256 - BLAKE2B512E - BLAKE2B512 - BLAKE2B160E - BLAKE2B160 - BLAKE2B224E - BLAKE2B224 - BLAKE2B384E - BLAKE2B384 - BLAKE2BP512E - BLAKE2BP512 - BLAKE2S256E - BLAKE2S256 - BLAKE2S160E - BLAKE2S160 - BLAKE2S224E - BLAKE2S224 - BLAKE2SP256E - BLAKE2SP256 - BLAKE2SP224E - BLAKE2SP224 - SHA1E - SHA1 - MD5E - MD5 - WORM - URL - X* - local repository version: 8 - operating system: linux x86_64 - remote types: - git - gcrypt - p2p - S3 - bup - directory - rsync - web - bittorrent - webdav - adb - tahoe - glacier - ddar - git-lfs - httpalso - borg - hook - external - supported repository versions: - 8 - 9 - 10 - upgrade supported from repository versions: - 0 - 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 - version: 10.20220322-1~ndall+1 ## location - path: /home/mszczepanik/Documents/hackathon-datalad-debian/mydistrib - type: dataset ## metadata_extractors - annex: - distribution: datalad 0.17.0 - load_error: None - module: datalad.metadata.extractors.annex - audio: - distribution: datalad 0.17.0 - load_error: ModuleNotFoundError(No module named 'mutagen') - module: datalad.metadata.extractors.audio - datacite: - distribution: datalad 0.17.0 - load_error: None - module: datalad.metadata.extractors.datacite - datalad_core: - distribution: datalad 0.17.0 - load_error: None - module: datalad.metadata.extractors.datalad_core - datalad_rfc822: - distribution: datalad 0.17.0 - load_error: None - module: datalad.metadata.extractors.datalad_rfc822 - exif: - distribution: datalad 0.17.0 - load_error: ModuleNotFoundError(No module named 'exifread') - module: datalad.metadata.extractors.exif - frictionless_datapackage: - distribution: datalad 0.17.0 - load_error: None - module: datalad.metadata.extractors.frictionless_datapackage - image: - distribution: datalad 0.17.0 - load_error: ModuleNotFoundError(No module named 'PIL') - module: datalad.metadata.extractors.image - xmp: - distribution: datalad 0.17.0 - load_error: ModuleNotFoundError(No module named 'libxmp') - module: datalad.metadata.extractors.xmp ## metadata_indexers ## python - implementation: CPython - version: 3.9.2 ## system - distribution: debian/11/bullseye - encoding: - default: utf-8 - filesystem: utf-8 - locale.prefered: UTF-8 - filesystem: - CWD: - path: /home/mszczepanik/Documents/hackathon-datalad-debian/mydistrib - HOME: - path: /home/mszczepanik - TMP: - path: /tmp - max_path_length: 318 - name: Linux - release: 5.10.0-15-amd64 - type: posix - version: #1 SMP Debian 5.10.120-1 (2022-06-09) Hint: install psutil to get filesystem information ```

[mih]

Possibly an indication that we need singularity v3. If so, this issues should be closed with some kind of documentation on that, somewhere.

From my POV there is little point in supporting singularity v2.

[mslw]

I can confirm that updating to Singularity 3 (specifically 3.10.0) solved this issue. All commands return with ok result.

ls -lh builder/envs 
total 8.0K
-rw-r--r-- 1 mszczepanik mszczepanik  46 Jul 11 13:19 README.md
lrwxrwxrwx 1 mszczepanik mszczepanik 135 Jul 11 13:21 singularity-amd64.sif -> ../.git/annex/objects/V9/Kp/MD5E-s210337792--ba90081997dbd7244e08401e150a438f.sif/MD5E-s210337792--ba90081997dbd7244e08401e150a438f.sif

FTR I'm on Debian 11.4 (bullseye) and I installed Singularity from their GitHub release page (`singularity-ce_3.10.0-focal_amd64.deb)

Do you think that a short section "requirements" with a comment (saying that Singularity 3 is required) in ReST docs and / or the README file would be enough?

[mslw]

Now superseded by an issue related to documentation.