forkserver fail to fork initial namespaces process

[Amandaynzhou]

Hi!

I run the sandbox2 in linux example from https://developers.google.com/code-sandboxing/sandbox2/examples?hl=zh-cn and get the clone error: [util.cc : 199] RAW: clone(): Invalid argument [22]. If it is convenient, could you please give some suggestions on solving this?

bazel run //sandboxed_api/sandbox2/examples/tool:sandbox2tool -- \
--sandbox2tool_resolve_and_add_libraries \
--sandbox2tool_additional_bind_mounts /etc \
/bin/cat /etc/hostname
Starting local Bazel server and connecting to it...
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of gcc... none
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... none
checking whether make sets $(MAKE)... (cached) yes
checking for ar... ar
checking the archiver (ar) interface... ar
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... no
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking dependency style of gcc... none
checking for __uc_get_grs in -luca... no
checking for library containing dlopen... -ldl
checking for ANSI C header files... (cached) yes
checking asm/ptrace_offsets.h usability... no
checking asm/ptrace_offsets.h presence... no
checking for asm/ptrace_offsets.h... no
checking asm/ptrace.h usability... yes
checking asm/ptrace.h presence... yes
checking for asm/ptrace.h... yes
checking endian.h usability... yes
checking endian.h presence... yes
checking for endian.h... yes
checking sys/endian.h usability... no
checking sys/endian.h presence... no
checking for sys/endian.h... no
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking execinfo.h usability... yes
checking execinfo.h presence... yes
checking for execinfo.h... yes
checking ia64intrin.h usability... no
checking ia64intrin.h presence... no
checking for ia64intrin.h... no
checking sys/uc_access.h usability... no
checking sys/uc_access.h presence... no
checking for sys/uc_access.h... no
checking for unistd.h... (cached) yes
checking signal.h usability... yes
checking signal.h presence... yes
checking for signal.h... yes
checking for sys/types.h... (cached) yes
checking sys/procfs.h usability... yes
checking sys/procfs.h presence... yes
checking for sys/procfs.h... yes
checking sys/ptrace.h usability... yes
checking sys/ptrace.h presence... yes
checking for sys/ptrace.h... yes
checking sys/syscall.h usability... yes
checking sys/syscall.h presence... yes
checking for sys/syscall.h... yes
checking byteswap.h usability... yes
checking byteswap.h presence... yes
checking for byteswap.h... yes
checking elf.h usability... yes
checking elf.h presence... yes
checking for elf.h... yes
checking sys/elf.h usability... no
checking sys/elf.h presence... no
checking for sys/elf.h... no
checking link.h usability... yes
checking link.h presence... yes
checking for link.h... yes
checking sys/link.h usability... no
checking sys/link.h presence... no
checking for sys/link.h... no
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking for size_t... yes
checking size of off_t... 8
checking for struct dl_phdr_info.dlpi_subs... yes
checking for struct elf_prstatus... yes
checking for struct prstatus... no
checking whether PTRACE_POKEUSER is declared... yes
checking whether PTRACE_POKEDATA is declared... yes
checking whether PTRACE_SETREGSET is declared... yes
checking whether PTRACE_TRACEME is declared... yes
checking whether PTRACE_CONT is declared... yes
checking whether PTRACE_SINGLESTEP is declared... yes
checking whether PTRACE_SYSCALL is declared... yes
checking whether PT_IO is declared... no
checking whether PT_GETREGS is declared... yes
checking whether PT_GETFPREGS is declared... yes
checking whether PT_CONTINUE is declared... yes
checking whether PT_TRACE_ME is declared... yes
checking whether PT_STEP is declared... yes
checking whether PT_SYSCALL is declared... yes
checking for dl_iterate_phdr... yes
checking for dl_phdr_removals_counter... no
checking for dlmodinfo... no
checking for getunwind... no
checking for ttrace... no
checking for mincore... yes
checking for pipe2... yes
checking if building with AltiVec... no
checking for Android... no
checking if we should build libunwind-coredump... yes
checking if we should build libunwind-ptrace... yes
checking if we should export unwind.h... yes
checking if we should build libunwind-setjmp... yes
checking for build architecture... x86_64
checking for host architecture... x86_64
checking for target architecture... x86_64
checking for target operating system... linux-gnu
checking for ELF helper width... 64
checking whether to include DWARF support... yes
checking whether to restrict build to remote support... no
checking whether to enable debug support... 
checking whether to enable C++ exception support... no
checking whether to load .debug_frame sections... no
checking whether to block signals during mutex ops... yes
checking whether to validate memory addresses before use... yes
checking whether to enable msabi support... 
checking whether to support LZMA-compressed symbol tables... no
checking whether to support ZLIB-compressed symbol tables... auto
checking for uncompress in -lz... yes
checking whether to support UNW_CACHE_PER_THREAD... 
checking for Intel compiler... no
checking if building on Solaris then define __EXTENSIONS__ macro... yes
checking for QCC compiler... no
checking for __builtin___clear_cache... yes
checking for __builtin_unreachable... yes
checking for library containing backtrace... none required
checking for latex2man... no
configure: WARNING: latex2man not found. Install latex2man. Disabling docs.
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating tests/Makefile
config.status: creating tests/check-namespace.sh
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating include/libunwind-common.h
config.status: creating include/libunwind.h
config.status: creating include/tdep/libunwind_i.h
config.status: creating src/unwind/libunwind.pc
config.status: creating src/coredump/libunwind-coredump.pc
config.status: creating src/ptrace/libunwind-ptrace.pc
config.status: creating src/setjmp/libunwind-setjmp.pc
config.status: creating src/libunwind-generic.pc
config.status: creating include/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
bash: line 1: config.guess: No such file or directory
mv: missing destination file operand after 'configure-bazel-gen'
Try 'mv --help' for more information.
INFO: Analyzed target //sandboxed_api/sandbox2/examples/tool:sandbox2tool (71 packages loaded, 1557 targets configured).
INFO: Found 1 target...
Target //sandboxed_api/sandbox2/examples/tool:sandbox2tool up-to-date:
  bazel-bin/sandboxed_api/sandbox2/examples/tool/sandbox2tool
INFO: Elapsed time: 96.239s, Critical Path: 23.87s
INFO: 563 processes: 14 internal, 549 processwrapper-sandbox.
INFO: Build completed successfully, 563 total actions
INFO: Running command line: bazel-bin/sandboxed_api/sandbox2/examples/tool/sandbox2tool --sandbox2tool_resolve_and_add_libraries --sandbox2tool_additional_bind_mounts /etc /bin/cat /etc/hostname
[global_forkclient.cc : 153] RAW: Starting global forkserver
[util.cc : 199] RAW: clone(): Invalid argument [22]
[forkserver.cc : 580] RAW: Check pid != -1 failed: failed to fork initial namespaces process: Invalid argument [22]
E0321 13:43:35.448775  200609 fork_client.cc:55] Receiving init PID from the ForkServer failed
E0321 13:43:35.448842  200609 global_forkclient.cc:303] Global forkserver connection terminated
[global_forkclient.cc : 227] RAW: forkserver (pid=207271) terminated by signal 6
E0321 13:43:35.448949  200609 sandbox2tool.cc:233] Sandbox failed
E0321 13:43:35.448976  200609 sandbox2tool.cc:239] Sandbox error: SETUP_ERROR - Code: FAILED_SUBPROCESS

Best

[robertswiecki]

Hi, what's your kernel version? uname -a

Also, can you run everything under strace, and upload the results?

strace -f -o /tmp/output.txt bazel-bin/sandboxed_api/sandbox2/examples/tool/sandbox2tool --sandbox2tool_resolve_and_add_libraries --sandbox2tool_additional_bind_mounts /etc /bin/cat /etc/hostname

[Amandaynzhou]

Hi, Thanks for the reply! I am using the Linux system developed within the enterprise (similar to CentOS), and the kernel may be 5.4. I find the problem may be caused by the permission or something in Cloud Virtual Machine ( I try both inside docker with --privileged and outside docker in CVM, not work). Currently, I bypass it by running it on the local machine (Ubuntu).

Here is the output: https://drive.google.com/file/d/1nTvRS7-DJw8qV0H_jDrx3joBOVu0PEYB/view?usp=share_link

[cblichmann]

1964804 clone(child_stack=0x7ffeabeba7b0, flags=CLONE_NEWNS|CLONE_NEWUSER|SIGCHLD) = -1 EINVAL (Invalid argument)

That line looks like unprivileged user namespace are not permitted. If you're on a CentOS derivative, that might be the default config. Note that Docker by default uses a daemon that runs as root, so it will not have this issue.

Can you check if

echo 10000 > /proc/sys/user/max_user_namespaces

does anything for you?

On a Debian kernel, this would be

sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone

[Amandaynzhou]

After I enter echo 10000 > /proc/sys/user/max_user_namespaces

It still gets the same error

INFO: Running command line: bazel-bin/sandboxed_api/sandbox2/examples/tool/sandbox2tool --sandbox2tool_resolve_and_add_libraries --sandbox2tool_additional_bind_mounts /etc /bin/cat /etc/hostname
[global_forkclient.cc : 153] RAW: Starting global forkserver
[util.cc : 199] RAW: clone(): Invalid argument [22]
[forkserver.cc : 580] RAW: Check pid != -1 failed: failed to fork initial namespaces process: Invalid argument [22]
E0324 15:10:23.602776 2614517 fork_client.cc:55] Receiving init PID from the ForkServer failed
E0324 15:10:23.602837 2614517 global_forkclient.cc:303] Global forkserver connection terminated
[global_forkclient.cc : 227] RAW: forkserver (pid=2621296) terminated by signal 6
E0324 15:10:23.602924 2614517 sandbox2tool.cc:233] Sandbox failed
E0324 15:10:23.602943 2614517 sandbox2tool.cc:239] Sandbox error: SETUP_ERROR - Code: FAILED_SUBPROCESS

[cblichmann]

Ok, this might still mean that the unprivileged namespace feature is not active. What's the output of uname -a? Are you running a custom kernel or the one that ships with your distribution?

Also, to rule out other issues, can you try to run sandbox2tool as root?

[Amandaynzhou]

Yes, I use the custom kernel like: Linux VM-252-28-centos 5.4.32-1-sometag I tried sudo + command and met the same error.

[cblichmann]

Do you have a kernel config for me? Is CONFIG_USER_NS actually enabled?

[Amandaynzhou]

Sorry I could not provide the config file. But yes it seems that the CONFIG_USER_NS is not enabled.

./kernel/Makefile:75:obj-$(CONFIG_USER_NS) += user_namespace.o
./include/config/auto.conf:173:# CONFIG_USER_NS is not set
./include/linux/cred.h:391:#ifdef CONFIG_USER_NS
./include/linux/user_namespace.h:106:#ifdef CONFIG_USER_NS
./include/linux/uidgid.h:121:#ifdef CONFIG_USER_NS
./include/linux/uidgid.h:189:#endif /* CONFIG_USER_NS */
./include/linux/seq_file.h:165:#ifdef CONFIG_USER_NS
./include/linux/projid.h:51:#ifdef CONFIG_USER_NS
./include/linux/projid.h:88:#endif /* CONFIG_USER_NS */

➜  config grep CONFIG_USER_NS /boot/config-$(uname -r)
# CONFIG_USER_NS is not set

[cblichmann]

That is very likely the root of this issue. Can you try with/rebuilt a kernel that has this setting enabled?

[cblichmann]

I think we got to the bottom of this. Closing. Feel free to reopen if you have more questions.