search

JuliaTime / TimeZones.jl

IANA time zone database access for the Julia programming language
Other
86 stars 52 forks source link

Build error: `IOError: sendfile: operation not supported on socket (ENOTSUP)` #341

Open SebastianM-C opened 3 years ago

SebastianM-C commented 3 years ago

I got the following error while building TimeZones v1.5.5:

Building TimeZones → `/mnt/storage/sebastian/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/960099aed321e05ac649c90d583d59c9309faee1/build.log`
ERROR: Error building `TimeZones`: 
 Downloading artifact: tzdata2021a
[ Info: Installing 2021a tzdata region data
ERROR: LoadError: IOError: sendfile: operation not supported on socket (ENOTSUP)
Stacktrace:
  [1] uv_error
    @ ./libuv.jl:97 [inlined]
  [2] sendfile(dst::Base.Filesystem.File, src::Base.Filesystem.File, src_offset::Int64, bytes::Int64)
    @ Base.Filesystem ./filesystem.jl:119
  [3] sendfile(src::String, dst::String)
    @ Base.Filesystem ./file.jl:960
  [4] cp(src::String, dst::String; force::Bool, follow_symlinks::Bool)
    @ Base.Filesystem ./file.jl:355
  [5] build(version::String, regions::Vector{String}, archive_dir::String, tz_source_dir::String, compiled_dir::String; verbose::Bool)
    @ TimeZones.TZData /mnt/storage/sebastian/.julia/packages/TimeZones/y3gf6/src/tzdata/build.jl:64
  [6] build(version::String)
    @ TimeZones.TZData /mnt/storage/sebastian/.julia/packages/TimeZones/y3gf6/src/tzdata/build.jl:121
  [7] build(version::String; force::Bool)
    @ TimeZones /mnt/storage/sebastian/.julia/packages/TimeZones/y3gf6/src/build.jl:11
  [8] build (repeats 2 times)
    @ /mnt/storage/sebastian/.julia/packages/TimeZones/y3gf6/src/build.jl:11 [inlined]
  [9] top-level scope
    @ /mnt/storage/sebastian/.julia/packages/TimeZones/y3gf6/deps/build.jl:3
 [10] include(fname::String)
    @ Base.MainInclude ./client.jl:444
 [11] top-level scope
    @ none:5
in expression starting at /mnt/storage/sebastian/.julia/packages/TimeZones/y3gf6/deps/build.jl:3

This is on:

julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_DEPOT_PATH = /mnt/storage/sebastian/.julia
  JULIA_EDITOR = "/home/sebastian/.vscode-server/bin/054a9295330880ed74ceaedda236253b4f39a335/node"
  JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
  JULIA_NUM_THREADS = 20
  JULIA_PATH = /usr/local/julia
  JULIA_VERSION = 1.6.1

I'm not sure if it helps, but my system info is the following: I'm using the julia docker image (in a singularity container) with

shell> uname -a
Linux cn1 3.10.0-1127.19.1.el7.x86_64 #1 SMP Thu Aug 20 14:39:03 CDT 2020 x86_64 GNU/Linux

shell> cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

on a host with

$ cat /etc/os-release 
NAME="Scientific Linux"
VERSION="7.9 (Nitrogen)"
ID="scientific"
ID_LIKE="rhel centos fedora"
VERSION_ID="7.9"
PRETTY_NAME="Scientific Linux 7.9 (Nitrogen)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:scientificlinux:scientificlinux:7.9:GA"
HOME_URL="http://www.scientificlinux.org//"
BUG_REPORT_URL="mailto:scientific-linux-devel@listserv.fnal.gov"

REDHAT_BUGZILLA_PRODUCT="Scientific Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Scientific Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"
omus commented 3 years ago

Thanks for reporting this. This appears to be a bug with Julia itself. I suspect it may have something to do with copying files across file systems. Can you outline what your file system setup is in the container?

SebastianM-C commented 3 years ago

I'm not sure if it's related, but as shown above, I have a custom DEPOT_PATH, which is bind mounted in the container from the host. On the host this path is a NFS share made available to multiple servers.

Besides that, I think that singularity auto-mounts the current working directory (which was inside /mnt/storage/) and the home directory, which is local to each server.

What other details could I provide?

omus commented 3 years ago

What other details could I provide?

Can you run the following code in your container?

using Pkg
Pkg.add(PackageSpec(name="TimeZones", rev="cv/build-debug"))
using TimeZones
TimeZones.build()

Also the output from mount may be useful.

SebastianM-C commented 3 years ago

The add command triggered a build, which errored like this:

Resolving package versions...
    Updating `/mnt/storage/epoch/dev/Project.toml`
  [f269a46b] + TimeZones v1.5.5 `https://github.com/JuliaTime/TimeZones.jl.git#cv/build-debug`
    Updating `/mnt/storage/epoch/dev/Manifest.toml`
  [8f5d6c58] + EzXML v1.1.0
  [78c3b35d] + Mocking v0.7.1
  [f269a46b] + TimeZones v1.5.5 `https://github.com/JuliaTime/TimeZones.jl.git#cv/build-debug`
    Building TimeZones → `/mnt/storage/sebastian/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/8856fa551085f6bbb29e0e9800d3be7ecf8e0db8/build.log`
ERROR: Error building `TimeZones`: 
[ Info: Installing 2021a tzdata region data
[ Info: /mnt/storage/sebastian/.julia/artifacts/6d94ada27957590cbd0d7678f5ae711232a4d714/africa -> /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/deps/tzsource/africa
ERROR: LoadError: IOError: sendfile: operation not supported on socket (ENOTSUP)
Stacktrace:
  [1] uv_error
    @ ./libuv.jl:97 [inlined]
  [2] sendfile(dst::Base.Filesystem.File, src::Base.Filesystem.File, src_offset::Int64, bytes::Int64)
    @ Base.Filesystem ./filesystem.jl:119
  [3] sendfile(src::String, dst::String)
    @ Base.Filesystem ./file.jl:960
  [4] cp(src::String, dst::String; force::Bool, follow_symlinks::Bool)
    @ Base.Filesystem ./file.jl:355
  [5] build(version::String, regions::Vector{String}, archive_dir::String, tz_source_dir::String, compiled_dir::String; verbose::Bool)
    @ TimeZones.TZData /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/src/tzdata/build.jl:67
  [6] build(version::String)
    @ TimeZones.TZData /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/src/tzdata/build.jl:124
  [7] build(version::String; force::Bool)
    @ TimeZones /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/src/build.jl:11
  [8] build (repeats 2 times)
    @ /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/src/build.jl:11 [inlined]
  [9] top-level scope
    @ /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/deps/build.jl:3
 [10] include(fname::String)
    @ Base.MainInclude ./client.jl:444
 [11] top-level scope
    @ none:5
in expression starting at /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/deps/build.jl:3
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:55
  [2] (::Pkg.Operations.var"#82#87"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec})()
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1044
  [3] withenv(::Pkg.Operations.var"#82#87"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, ::Pair{String, String}, ::Vararg{Pair{String, B} where B, N} where N)
    @ Base ./env.jl:161
  [4] (::Pkg.Operations.var"#109#113"{String, Pkg.Operations.var"#82#87"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, Pkg.Types.PackageSpec})()
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1542
  [5] with_temp_env(fn::Pkg.Operations.var"#109#113"{String, Pkg.Operations.var"#82#87"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, Pkg.Types.PackageSpec}, temp_env::String)
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1444
  [6] (::Pkg.Operations.var"#108#112"{Pkg.Operations.var"#82#87"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String})(tmp::String)
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1517
  [7] mktempdir(fn::Pkg.Operations.var"#108#112"{Pkg.Operations.var"#82#87"{Bool, Pkg.Types.Context, String, Pkg.Types.PackageSpec}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:729
  [8] mktempdir(fn::Function, parent::String) (repeats 2 times)
    @ Base.Filesystem ./file.jl:727
  [9] sandbox(fn::Function, ctx::Pkg.Types.Context, target::Pkg.Types.PackageSpec, target_path::String, sandbox_path::String, sandbox_project_override::Pkg.Types.Project)
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1483
 [10] build_versions(ctx::Pkg.Types.Context, uuids::Vector{Base.UUID}; verbose::Bool)
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1025
 [11] build_versions
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:952 [inlined]
 [12] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Vector{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1241
 [13] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Iterators.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:203
 [14] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:79
 [15] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:77
 [16] #add#22
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:74 [inlined]
 [17] add(pkg::Pkg.Types.PackageSpec)
    @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:74
 [18] top-level scope
    @ REPL[2]:1

I'm not sure if it helps, but I also followed with the following commands

julia> using TimeZones

julia> TimeZones.build()
[ Info: Installing 2021a tzdata region data
[ Info: /mnt/storage/sebastian/.julia/artifacts/6d94ada27957590cbd0d7678f5ae711232a4d714/africa -> /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/deps/tzsource/africa
ERROR: IOError: sendfile: operation not supported on socket (ENOTSUP)
Stacktrace:
 [1] uv_error
   @ ./libuv.jl:97 [inlined]
 [2] sendfile(dst::Base.Filesystem.File, src::Base.Filesystem.File, src_offset::Int64, bytes::Int64)
   @ Base.Filesystem ./filesystem.jl:119
 [3] sendfile(src::String, dst::String)
   @ Base.Filesystem ./file.jl:960
 [4] cp(src::String, dst::String; force::Bool, follow_symlinks::Bool)
   @ Base.Filesystem ./file.jl:355
 [5] build(version::String, regions::Vector{String}, archive_dir::String, tz_source_dir::String, compiled_dir::String; verbose::Bool)
   @ TimeZones.TZData /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/src/tzdata/build.jl:67
 [6] build(version::String)
   @ TimeZones.TZData /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/src/tzdata/build.jl:124
 [7] build(version::String; force::Bool)
   @ TimeZones /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/src/build.jl:11
 [8] build (repeats 2 times)
   @ /mnt/storage/sebastian/.julia/packages/TimeZones/UQZUE/src/build.jl:11 [inlined]
 [9] top-level scope
   @ REPL[4]:1

The output from mount form the julia shell (inside the container):

shell> mount
overlay on / type overlay (ro,nodev,relatime,seclabel,lowerdir=/var/singularity/mnt/session/overlay-lowerdir:/var/singularity/mnt/session/rootfs)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=65840508k,nr_inodes=16460127,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda2 on /usr/share/zoneinfo/UCT type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /etc/hosts type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
172.16.9.1:/data1 on /mnt/storage type nfs4 (rw,nosuid,nodev,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.9.11,local_lock=none,addr=172.16.9.1)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=374563956)
sysfs on /sys type sysfs (rw,nosuid,nodev,relatime,seclabel)
/dev/sda5 on /home/sebastian type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /tmp type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /var/tmp type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /.singularity.d/libs type tmpfs (ro,nosuid,nodev,relatime,seclabel,size=16384k,uid=1000,gid=1000)
/dev/sda2 on /.singularity.d/libs/libcuda.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libcuda.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libEGL_nvidia.so.0 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libEGL.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libEGL.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLdispatch.so.0 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLdispatch.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLESv1_CM_nvidia.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLESv1_CM.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLESv1_CM.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLESv2_nvidia.so.2 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLESv2.so.2 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLESv2.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGL.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGL.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLX_nvidia.so.0 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLX.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libGLX.so.0 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvcuvid.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvcuvid.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-cbl.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-cbl.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-cfg.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-cfg.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-compiler.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-compiler.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-eglcore.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-eglcore.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-encode.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-encode.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-fatbinaryloader.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-fbc.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-fbc.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-glcore.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-glcore.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-glsi.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-glsi.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-glvkspirv.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-glvkspirv.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-gtk2.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-gtk3.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-gtk3.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-ifr.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-ifr.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-ml.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-ml.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-opencl.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-opticalflow.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-ptxjitcompiler.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-ptxjitcompiler.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-rtcore.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-rtcore.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-tls.so.410.48 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvidia-tls.so.460.32.03 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libnvoptix.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libOpenCL.so.1 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libOpenCL.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libOpenGL.so type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /.singularity.d/libs/libOpenGL.so.0 type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /usr/bin/nvidia-smi type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /usr/bin/nvidia-debugdump type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /usr/bin/nvidia-persistenced type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /usr/bin/nvidia-cuda-mps-control type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /usr/bin/nvidia-cuda-mps-server type xfs (ro,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /etc/resolv.conf type tmpfs (rw,nosuid,relatime,seclabel,size=16384k,uid=1000,gid=1000)
tmpfs on /etc/passwd type tmpfs (rw,nosuid,relatime,seclabel,size=16384k,uid=1000,gid=1000)
tmpfs on /etc/group type tmpfs (rw,nosuid,relatime,seclabel,size=16384k,uid=1000,gid=1000)
172.16.9.1:/data1 on /mnt/storage type nfs4 (rw,nosuid,nodev,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.9.11,local_lock=none,addr=172.16.9.1)
/dev/sda2 on /usr/local/cuda type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)

From outside the container form bash:

$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=65840508k,nr_inodes=16460127,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda2 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sda5 on /home type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
172.16.9.1:/data1 on /mnt/storage type nfs4 (rw,nosuid,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.9.11,local_lock=none,addr=172.16.9.1,_netdev)
tmpfs on /run/user/42 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=13171284k,mode=700,uid=42,gid=42)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=13171284k,mode=700,uid=1000,gid=1000)
tmpfs on /run/user/1005 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=13171284k,mode=700,uid=1005,gid=900)
tmpfs on /run/user/1002 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=13171284k,mode=700,uid=1002,gid=900)
tmpfs on /run/user/1003 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=13171284k,mode=700,uid=1003,gid=900)
tmpfs on /run/user/1009 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=13171284k,mode=700,uid=1009,gid=900)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=374563956)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /run/user/1004 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=13171284k,mode=700,uid=1004,gid=900)
tmpfs on /run/user/1008 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=13171284k,mode=700,uid=1008,gid=900)