search

sylabs / singularity

SingularityCE is the Community Edition of Singularity, an open source container platform designed to be simple, fast, and secure.
https://sylabs.io/docs/
Other
749 stars 96 forks source link

2023-01-03 main branch e2e-failures RHEL 7 AMD64 #2473

Closed dtrudg closed 8 months ago

dtrudg commented 8 months ago

A large number of failures are expected due to inability to use buildkit on a distro this old.

These need to be investigated and gated appropriately.

   --- FAIL: TestE2E (551.32s)
        --- FAIL: TestE2E/PAR (0.00s)
            --- FAIL: TestE2E/PAR/DOCKER (0.00s)
                --- FAIL: TestE2E/PAR/DOCKER/ordered (166.77s)
                    --- FAIL: TestE2E/PAR/DOCKER/ordered/user (12.17s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/user_df_build (1.62s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_df_build (1.80s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif (0.84s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OCIUser (0.09s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OCIFakeroot (0.08s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OCIRoot (0.06s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OrigHomeOCIUser (0.10s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OrigHomeOCIFakeroot (0.10s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/root_oci-sif/OrigHomeOCIRoot (0.09s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/user/user_oci-sif (0.62s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/user_oci-sif/OCIUser (0.08s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/user_oci-sif/OCIFakeroot (0.07s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/user_oci-sif/OCIRoot (0.05s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/user_oci-sif/OrigHomeOCIUser (0.09s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/user_oci-sif/OrigHomeOCIFakeroot (0.07s)
                            --- FAIL: TestE2E/PAR/DOCKER/ordered/user/user_oci-sif/OrigHomeOCIRoot (0.04s)
                    --- FAIL: TestE2E/PAR/DOCKER/ordered/scif (5.12s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/scif/build (4.06s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/scif/run_echo (0.11s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/scif/exec_echo (0.18s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/scif/run_script (0.09s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/scif/exec_script (0.28s)
                        --- FAIL: TestE2E/PAR/DOCKER/ordered/scif/exec_script2 (0.40s)
            --- FAIL: TestE2E/PAR/ACTIONS (0.01s)
                --- FAIL: TestE2E/PAR/ACTIONS/bind_image (0.01s)
                --- FAIL: TestE2E/PAR/ACTIONS/persistent_overlay (0.01s)
                --- FAIL: TestE2E/PAR/ACTIONS/issue_4755 (0.36s)
                --- FAIL: TestE2E/PAR/ACTIONS/ociBindImage (0.01s)
        --- FAIL: TestE2E/SEQ (243.58s)
            --- FAIL: TestE2E/SEQ/BUILD (38.04s)
                --- FAIL: TestE2E/SEQ/BUILD/dockerfile (34.35s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User (16.52s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_none (1.37s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_none/build (1.14s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_none/act (0.23s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_wrong (1.36s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_wrong/build (1.14s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_wrong/act (0.22s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_wrong_and_right (1.36s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_wrong_and_right/build (1.14s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_wrong_and_right/act (0.22s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_right (1.35s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_right/build (1.14s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_right/act (0.21s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_none (1.36s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_none/build (1.14s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_wrong (1.36s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_wrong/build (1.14s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_wrong_and_right (1.35s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_wrong_and_right/build (1.14s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_wrong_and_right/act (0.21s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_right (1.36s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_right/build (1.14s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/ba_nd_right/act (0.22s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/add (1.38s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/add/build (1.15s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/User/add/act (0.24s)
                    --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root (17.82s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_none (1.45s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_none/build (1.23s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_none/act (0.22s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong (1.45s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong/build (1.22s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong/act (0.22s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong_and_right (1.45s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong_and_right/build (1.23s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_wrong_and_right/act (0.21s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_right (1.46s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_right/build (1.23s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_right/act (0.23s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_none (1.43s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_none/build (1.22s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong (1.43s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong/build (1.21s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong_and_right (1.45s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong_and_right/build (1.23s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_wrong_and_right/act (0.22s)
                        --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_right (1.45s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_right/build (1.23s)
                            --- FAIL: TestE2E/SEQ/BUILD/dockerfile/Root/ba_nd_right/act (0.22s)
dtrudg commented 8 months ago

Definitely gate buildkit - not supported on RHEL7.

extfs overlay error may be due to fuse2fs failure to mount?

=== NAME  TestE2E/PAR/ACTIONS/ociOverlayExtfsPerms/OCIUser/FirstWrite
    oci.go:1548: 
        "/usr/local/bin/singularity exec --oci --overlay /tmp/stest.602905161/oci_overlay_extfs_perms-4198888422/extfs-perms-test.img oci-sif:/tmp/stest.602905161/oci-sif.sif sh -c echo my_test_string > /my_test_file" output:
        FATAL:   could not open image oci-sif:/tmp/stest.602905161/oci-sif.sif: failed to get absolute path: getwd: no such file or directory

    oci.go:1548: got 255 as exit code and was expecting 0: exit status 255
        waiting for command "/usr/local/bin/singularity exec --oci --overlay /tmp/stest.602905161/oci_overlay_extfs_perms-4198888422/extfs-perms-test.img oci-sif:/tmp/stest.602905161/oci-sif.sif sh -c echo my_test_string > /my_test_file"
        github.com/sylabs/singularity/v4/e2e/internal/e2e.TestEnv.RunSingularity.func1
                github.com/sylabs/singularity/v4/e2e/internal/e2e/singularitycmd.go:673
        testing.tRunner
                testing/testing.go:1595
        runtime.goexit
                runtime/asm_amd64.s:1650

FUSEImage failures appeard due to missing fusermount3 for squashfuse...

        DEBUG   [U=1000,P=31601]   Mount()                       Executing FUSE mount command: "/usr/local/libexec/singularity/bin/squashfuse_ll -o uid=1000,gid=1000,ro,offset=40960 /tmp/stest.602905161/test.sif /tmp/stest.602905161/e2e-imgcache-2557647099/rootfs-2098309186/root"
        fuse: failed to exec fusermount3: No such file or directory
        WARNING [U=1000,P=31601]   prepareSquashfs()             squashfuse mount failed, falling back to extraction: encountered error while trying to mount image "/tmp/stest.602905161/test.sif" with FUSE at /tmp/stest.602905161/e2e-imgcache-2557647099/rootfs-2098309186/root: exit status 1
dtrudg commented 8 months ago

Default PATH issue on EL7?

    === CONT  TestE2E/PAR/ACTIONS/bind_image
        exec.go:28: Running cmd "mksquashfs" with args ["/tmp/stest.1372663981/bind-image-3302174588/root-squash-dir-1762999573" "/tmp/stest.1372663981/bind-image-3302174588/squashfs.simg" "-noappend" "-all-root"]
        actions.go:1939: Unexpected error while running command.

            Command:  mksquashfs /tmp/stest.1372663981/bind-image-3302174588/root-squash-dir-1762999573 /tmp/stest.1372663981/bind-image-3302174588/squashfs.simg -noappend -all-root
            ExitCode: 127
            Error:    exec: "mksquashfs": executable file not found in $PATH
            Stdout:
            Stderr:
        actions.go:1907: Not removing directory /tmp/stest.1372663981/bind-image-3302174588 for test TestE2E/PAR/ACTIONS/bind_image
dtrudg commented 8 months ago

Reduced as of 6959d1e to....

    --- FAIL: TestE2E (345.99s)
        --- FAIL: TestE2E/PAR (0.00s)
            --- FAIL: TestE2E/PAR/ACTIONS (0.01s)
                --- FAIL: TestE2E/PAR/ACTIONS/issue_4755 (0.27s)
                --- FAIL: TestE2E/PAR/ACTIONS/bind_image (0.01s)
                --- FAIL: TestE2E/PAR/ACTIONS/ociBindImage (0.02s)
                --- FAIL: TestE2E/PAR/ACTIONS/persistent_overlay (0.02s)
dtrudg commented 8 months ago

The issues in the above listing have been addressed by #2511 and #2513