flyctl hangs on startup when run under github-runner

antifuchs commented 1 year ago

Describe the bug

I'm seeing the flyctl version that lives in nixpkgs-unstable (1.0.8) hang on startup in a really early FUTEX_WAIT_PRIVATE call - but only when invoked under a private github-runner installation. The version of flyctl that is in nixpkgs-22.11 works fine. I can run both versions of flyctl in a terminal on the machine where it hangs.

This is annoying to reproduce because it needs a self-hosted github runner, but here goes:

Steps To Reproduce

I've made a test repo https://github.com/antifuchs/nixpkgs-bug-repro-flyctl that pins the versions of nixpkgs (and thus flyctl) that I'm reporting. I have a little github workflow here that invokes it:

name: "ZZZ: Debug something"
on:
  workflow_dispatch:
    inputs:
      args:
        description: "Commandline args to pass to flyctl. Normal shell quoting rules apply."
        default: "version"

concurrency: debug
jobs:
  debug:
    name: "flyctl"
    runs-on: self-hosted
    timeout-minutes: 1
    strategy:
      matrix:
        version:
          - stable
          - unstable
      fail-fast: false
    steps:
      - run: |
          nix run github:antifuchs/nixpkgs-bug-repro-flyctl#${{ matrix.version }} ${{github.event.inputs.args}}

Steps to reproduce the behavior:

setup a self-hosted github runner under nixos
Install this workflow
Invoke the workflow (using version as the commandline arg).
Observe that stable succeeds immediately and that unstable gets cancelled after 1min, the workflow timeout.

Expected behavior

I would expect both versions of flyctl to succeed and run.

Additional context

I straced the flyctl startup (with a different commandline), and it looks like it's hanging at the very beginning of startup. Here's the redacted strace output:

Lots of strace output

```console set_robust_list(0x7f370b4c8a20, 24) = 0 getpid() = 175529 close(255) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigaction(SIGTSTP, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, 8) = 0 rt_sigaction(SIGTTIN, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, 8) = 0 rt_sigaction(SIGTTOU, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, 8) = 0 rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, 8) = 0 rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f370b508bf0}, 8) = 0 rt_sigaction(SIGCHLD, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f370b508bf0}, {sa_handler=0x44a480, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f370b508bf0}, 8) = 0 openat(AT_FDCWD, "/dev/null", O_RDONLY) = 3> dup2(3>, 0) = 0> close(3>) = 0 dup2(1, 2) = 2 fcntl(1, F_GETFD) = 0 execve("/nix/store/5w5cj1paa47cy0q1h110698ccx6mq693-flyctl-0.1.8/bin/flyctl", ["flyctl", "deploy", "--verbose", "--auto-confirm", "-t", "REDACTED-api-key", "-c", "fly/fly.REDACTED-config-name.toml", "-i", "registry.fly.io/REDACTED-config-name:REDACTED-tag", "--remote-only"], 0xf0e010 /* 69 vars */) = 0 brk(NULL) = 0x3f63000 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe994d8080) = -1 EINVAL (Invalid argument) mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f036cdcf000 access("/etc/ld-nix.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/opengl-driver/lib/glibc-hwcaps/x86-64-v4/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/opengl-driver/lib/glibc-hwcaps/x86-64-v4", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/opengl-driver/lib/glibc-hwcaps/x86-64-v3/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/opengl-driver/lib/glibc-hwcaps/x86-64-v3", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/opengl-driver/lib/glibc-hwcaps/x86-64-v2/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/opengl-driver/lib/glibc-hwcaps/x86-64-v2", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/opengl-driver/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/opengl-driver/lib", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v4/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v4", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v3/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v3", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v2/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v2", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/opengl-driver-32/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/opengl-driver-32/lib", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v4/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v4", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v3/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v3", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v2/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v2", 0x7ffe994d72b0, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0@\0\0\0\0\0\0\0p\25\1\0\0\0\0\0\0\0\0\0@\08\0\v\0@\0 \0\37\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\330/\0\0\0\0\0\0\330/\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0000\0\0\0\0\0\0\0000\0\0\0\0\0\0\0000\0\0\0\0\0\0\371u\0\0\0\0\0\0\371u\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\260\0\0\0\0\0\0\0\260\0\0\0\0\0\0\0\260\0\0\0\0\0\0p(\0\0\0\0\0\0p(\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\6\0\0\0000\344\0\0\0\0\0\0000\344\0\0\0\0\0\0000\344\0\0\0\0\0\0p\f\0\0\0\0\0\0X4\0\0\0\0\0\0\0\20\0\0\0\0\0\0\2\0\0\0\6\0\0\0\310\353\0\0\0\0\0\0\310\353\0\0\0\0\0\0\310\353\0\0\0\0\0\0 \2\0\0\0\0\0\0 \2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\310\2\0\0\0\0\0\0\310\2\0\0\0\0\0\0\310\2\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0S\345td\4\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0\360\276\0\0\0\0\0\0\360\276\0\0\0\0\0\0\360\276\0\0\0\0\0\0\254\2\0\0\0\0\0\0\254\2\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0000\344\0\0\0\0\0\0000\344\0\0\0\0\0\0000\344\0\0\0\0\0\0\320\v\0\0\0\0\0\0\320\v\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\200\0\300\4\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0z\355<\363\260F\1c\361\361ek%\25`g\230|\321n\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\201\0\0\0>\0\0\0\10\0\0\0\t\0\0\0\10\0\310\240,R \10\201\0\22\225\201\5\344\222\202\300\200\0B\1 \1\4\0\230\0000\220\25D", 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0555, st_size=73072, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 71816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f036cdbd000 mmap(0x7f036cdc0000, 32768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f036cdc0000 mmap(0x7f036cdc8000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb000) = 0x7f036cdc8000 mmap(0x7f036cdcb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f036cdcb000 mmap(0x7f036cdcd000, 6280, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f036cdcd000 close(3) = 0 openat(AT_FDCWD, "/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\209\0\0\0\0\0\0\0\0\0\0@\08\0\v\0@\0\35\0\34\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\230\n\0\0\0\0\0\0\230\n\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\20\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\20\0\0\0\0\0\0\25\1\0\0\0\0\0\0\25\1\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\210\1\0\0\0\0\0\0\210\1\0\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\6\0\0\0\310-\0\0\0\0\0\0\310=\0\0\0\0\0\0\310=\0\0\0\0\0\0@\2\0\0\0\0\0\0H\2\0\0\0\0\0\0\0\20\0\0\0\0\0\0\2\0\0\0\6\0\0\0\330-\0\0\0\0\0\0\330=\0\0\0\0\0\0\330=\0\0\0\0\0\0\360\1\0\0\0\0\0\0\360\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\310\2\0\0\0\0\0\0\310\2\0\0\0\0\0\0\310\2\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0S\345td\4\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0$\0\0\0\0\0\0\0$\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\310-\0\0\0\0\0\0\310=\0\0\0\0\0\0\310=\0\0\0\0\0\08\2\0\0\0\0\0\08\2\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\200\0\300\4\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\255\223}c\214\257\243\10\3v42\37y\324<\231\246\353\321\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\25\0\0\0\5\0\0\0\4\0\0\0\10\0\0\0\0\3\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\2 \0\0\0\f\200A\0\2\0\200\0'", 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0555, st_size=16464, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 16400, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f036cdb8000 mmap(0x7f036cdb9000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f036cdb9000 mmap(0x7f036cdba000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f036cdba000 mmap(0x7f036cdbb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f036cdbb000 close(3) = 0 openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\224\2\0\0\0\0\0@\0\0\0\0\0\0\0X\221$\0\0\0\0\0\0\0\0\0@\08\0\16\0@\0@\0?\0\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\20\3\0\0\0\0\0\0\20\3\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0 ?\34\0\0\0\0\0 ?\34\0\0\0\0\0 ?\34\0\0\0\0\0T\0\0\0\0\0\0\0T\0\0\0\0\0\0\0 \0\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\270\177\2\0\0\0\0\0\270\177\2\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\200\2\0\0\0\0\0\0\200\2\0\0\0\0\0\0\200\2\0\0\0\0\0\207P\27\0\0\0\0\0\207P\27\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\340\31\0\0\0\0\0\0\340\31\0\0\0\0\0\0\340\31\0\0\0\0\0dy\5\0\0\0\0\0dy\5\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\6\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0xO\0\0\0\0\0\0\240&\1\0\0\0\0\0\0\20\0\0\0\0\0\0\2\0\0\0\6\0\0\0\340\231\37\0\0\0\0\0\340\231\37\0\0\0\0\0\340\231\37\0\0\0\0\0\340\1\0\0\0\0\0\0\340\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\3\0\0\0\0\0\0p\3\0\0\0\0\0\0p\3\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0\20\0\0\0\0\0\0\0\220\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0S\345td\4\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0t?\34\0\0\0\0\0t?\34\0\0\0\0\0t?\34\0\0\0\0\0$u\0\0\0\0\0\0$u\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0\3208\0\0\0\0\0\0", 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\20\3\0\0\0\0\0\0\20\3\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0 ?\34\0\0\0\0\0 ?\34\0\0\0\0\0 ?\34\0\0\0\0\0T\0\0\0\0\0\0\0T\0\0\0\0\0\0\0 \0\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\270\177\2\0\0\0\0\0\270\177\2\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\200\2\0\0\0\0\0\0\200\2\0\0\0\0\0\0\200\2\0\0\0\0\0\207P\27\0\0\0\0\0\207P\27\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\340\31\0\0\0\0\0\0\340\31\0\0\0\0\0\0\340\31\0\0\0\0\0dy\5\0\0\0\0\0dy\5\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\6\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0xO\0\0\0\0\0\0\240&\1\0\0\0\0\0\0\20\0\0\0\0\0\0\2\0\0\0\6\0\0\0\340\231\37\0\0\0\0\0\340\231\37\0\0\0\0\0\340\231\37\0\0\0\0\0\340\1\0\0\0\0\0\0\340\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\3\0\0\0\0\0\0p\3\0\0\0\0\0\0p\3\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0\20\0\0\0\0\0\0\0\220\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0S\345td\4\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0t?\34\0\0\0\0\0t?\34\0\0\0\0\0t?\34\0\0\0\0\0$u\0\0\0\0\0\0$u\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0\3208\0\0\0\0\0\0\3208\0\0\0\0\0\0\1\0\0\0\0\0\0\0", 784, 64) = 784 newfstatat(3, "", {st_mode=S_IFREG|0555, st_size=2400600, ...}, AT_EMPTY_PATH) = 0 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\20\3\0\0\0\0\0\0\20\3\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0 ?\34\0\0\0\0\0 ?\34\0\0\0\0\0 ?\34\0\0\0\0\0T\0\0\0\0\0\0\0T\0\0\0\0\0\0\0 \0\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\270\177\2\0\0\0\0\0\270\177\2\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\200\2\0\0\0\0\0\0\200\2\0\0\0\0\0\0\200\2\0\0\0\0\0\207P\27\0\0\0\0\0\207P\27\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\340\31\0\0\0\0\0\0\340\31\0\0\0\0\0\0\340\31\0\0\0\0\0dy\5\0\0\0\0\0dy\5\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\6\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0xO\0\0\0\0\0\0\240&\1\0\0\0\0\0\0\20\0\0\0\0\0\0\2\0\0\0\6\0\0\0\340\231\37\0\0\0\0\0\340\231\37\0\0\0\0\0\340\231\37\0\0\0\0\0\340\1\0\0\0\0\0\0\340\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\3\0\0\0\0\0\0p\3\0\0\0\0\0\0p\3\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0\20\0\0\0\0\0\0\0\220\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0S\345td\4\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0P\3\0\0\0\0\0\0 \0\0\0\0\0\0\0 \0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0t?\34\0\0\0\0\0t?\34\0\0\0\0\0t?\34\0\0\0\0\0$u\0\0\0\0\0\0$u\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0000g\37\0\0\0\0\0\3208\0\0\0\0\0\0\3208\0\0\0\0\0\0\1\0\0\0\0\0\0\0", 784, 64) = 784 mmap(NULL, 2133456, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f036cbaf000 mmap(0x7f036cbd7000, 1531904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f036cbd7000 mmap(0x7f036cd4d000, 360448, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19e000) = 0x7f036cd4d000 mmap(0x7f036cda5000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1f6000) = 0x7f036cda5000 mmap(0x7f036cdab000, 52688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f036cdab000 close(3) = 0 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f036cbac000 arch_prctl(ARCH_SET_FS, 0x7f036cbac740) = 0 set_tid_address(0x7f036cbaca10) = 175529 set_robust_list(0x7f036cbaca20, 24) = 0 rseq(0x7f036cbad060, 0x20, 0, 0x53053053) = 0 mprotect(0x7f036cda5000, 16384, PROT_READ) = 0 mprotect(0x7f036cdbb000, 4096, PROT_READ) = 0 mprotect(0x7f036cdcb000, 4096, PROT_READ) = 0 mprotect(0x7f036ce02000, 8192, PROT_READ) = 0 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 getrandom("\xc2\x3e\x8c\x74\x96\xec\xdf\xe4", 8, GRND_NONBLOCK) = 8 brk(NULL) = 0x3f63000 brk(0x3f84000) = 0x3f84000 sched_getaffinity(0, 8192, [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39]) = 48 openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3 read(3, "2097152\n", 20) = 8 close(3) = 0 mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f036cb6c000 mmap(NULL, 131072, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f036cb4c000 mmap(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f036ca4c000 mmap(NULL, 8388608, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f036c24c000 mmap(NULL, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f036824c000 mmap(NULL, 536870912, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f034824c000 mmap(NULL, 8388608, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0347a4c000 mmap(0xc000000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000 mmap(NULL, 33554432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0345a4c000 mmap(NULL, 1133584, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0345937000 mmap(0xc000000000, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xc000000000 mmap(0x7f036cb4c000, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f036cb4c000 mmap(0x7f036cacc000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f036cacc000 mmap(0x7f036c652000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f036c652000 mmap(0x7f036a27c000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f036a27c000 mmap(0x7f03583cc000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f03583cc000 mmap(0x7f0347a4c000, 4222976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f0347a4c000 madvise(0x7f0347c00000, 2097152, MADV_HUGEPAGE) = 0 mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0345837000 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0345827000 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0345817000 rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0 sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0 sigaltstack({ss_sp=0xc000008000, ss_flags=0, ss_size=32768}, NULL) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 gettid() = 175529 rt_sigaction(SIGHUP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGHUP, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGINT, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGQUIT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGQUIT, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGILL, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGILL, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGTRAP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGTRAP, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGABRT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGABRT, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGBUS, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGBUS, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGFPE, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGFPE, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGUSR1, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGUSR1, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGSEGV, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGSEGV, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGUSR2, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGUSR2, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGPIPE, NULL, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGPIPE, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGALRM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGALRM, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGTERM, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGSTKFLT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGSTKFLT, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGCHLD, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGCHLD, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGURG, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGURG, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGXCPU, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGXCPU, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGXFSZ, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGXFSZ, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGVTALRM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGVTALRM, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGPROF, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGPROF, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGWINCH, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGWINCH, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGIO, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGIO, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGPWR, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGPWR, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGSYS, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGSYS, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRTMIN, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_1, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_2, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_3, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_3, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_4, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_4, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_5, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_5, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_6, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_6, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_7, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_7, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_8, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_8, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_9, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_9, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_10, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_10, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_11, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_11, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_12, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_12, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_13, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_13, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_14, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_14, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_15, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_15, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_16, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_16, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_17, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_17, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_18, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_18, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_19, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_19, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_20, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_20, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_21, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_21, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_22, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_22, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_23, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_23, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_24, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_24, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_25, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_25, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_26, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_26, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_27, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_27, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_28, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_28, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_29, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_29, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_30, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_30, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_31, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_31, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigaction(SIGRT_32, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGRT_32, {sa_handler=0x46c380, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0 rt_sigaction(SIGRT_1, {sa_handler=0x7f036cc350d0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f036cbecbf0}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 futex(0x7f036ce050c8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? +++ killed by SIGKILL +++ ```

The github-runner systemd service is running with pretty strict restrictions; here's that list:

$ sudo systemctl cat github-runner
# /etc/systemd/system/github-runner.service
[Unit]
After=network.target network-online.target
Description=GitHub Actions runner
Wants=network-online.target

[Service]
Environment="DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true"
Environment="HOME=/tmp/actions-runner-src"
Environment="LOCALE_ARCHIVE=/nix/store/4jdpbi210fnw59kxgjn16l41b9l80bvm-glibc-locales-2.35-224/lib/locale/locale-archive"
Environment="PATH=/nix/store/kga2r02rmyxl14sg96nxbdhifq3rb8lc-bash-5.1-p16/bin:/nix/store/l6jgwxkc3jhr029vfzfwzcy28iyckwsj-coreutils-9.1/bin:/nix/store/1gz8qns2nlk9l3rw7079g55r64idn7sn-git-2.38.5/bin:/nix/s>
Environment="RUNNER_ROOT=/tmp/actions-runner-src"
Environment="TZDIR=/nix/store/z0kg1c0f8fx6r4rgg5bdy01lb2b9izqg-tzdata-2023a/share/zoneinfo"

DeviceAllow=
DynamicUser=true
ExecStart=/nix/store/vs5rx7j91p40w6mv8y724rg5viywa1g2-actions-runner
ExecStartPre=!/nix/store/95q8r1j5ri55bxx5y4qjfbxqpac9r60g-fix-perms.sh
ExecStartPre=!/nix/store/hggp0ir2lmy233fs7hfmkp6xx14wmq0j-copy-token.sh
ExecStartPre=/nix/store/jkgmb7a8z1y8acn7fb6nzhy9g96nymm4-setup-runner.sh
ExecStartPre=!/nix/store/gix2x02ycylqsyylw8hjbv87i6r1yc5g-remove-token.sh
ExecStop=/nix/store/lizw71m65rcbwcpg7m7j7apdx5r9sakh-github-runner-stop
KillMode=mixed
KillSignal=SIGTERM
LockPersonality=true
LogsDirectory=github-runner/gloria
MemoryDenyWriteExecute=false
NoNewPrivileges=true
PIDFile=github-runner/gloria/runsvc.pid
PrivateDevices=true
PrivateMounts=true
PrivateNetwork=false
PrivateTmp=true
PrivateUsers=true
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=strict
RemoveIPC=true
RestrictRealtime=true
RestrictSUIDSGID=true
RuntimeDirectory=github-runner/gloria
StateDirectory=github-runner/gloria
StateDirectoryMode=0700

Notify maintainers

@aaronjanse @jsierles @techknowlogick @viraptor

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.


[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.112, NixOS, 22.11 (Raccoon), 22.11.20230521.b0671cb`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.1`
 - nixpkgs: `/nix/store/0a9mxx52l6y6smb1jpnc6b6m78mac3l8-source````

antifuchs commented 1 year ago

Updating to mention that this also fails with the latest released version of flyctl, 0.1.12: https://github.com/antifuchs/nixpkgs-bug-repro-flyctl/commit/a3eb32abc5d6fb8c04c2d08c5abe7e09d8f5e581 - invoking the #latest flake app also hangs indefinitely, without output. My next suspicion would be that it's maybe related to the version of the golang compiler used to build it (in combination with resource limits in place, maybe?), but I haven't been able to replicate this at all using a systemd-run invocation with the given service options in place.

viraptor commented 1 year ago

Since plain systemd-run works, that means it's likely not the environment itself / app but some specific unit options that break it. Have you tried commenting out every single protect setting / restriction from your runner service and reenabling one by one?

Also, a basic question, but have you got anything relevant in the journal?

antifuchs commented 1 year ago

Nothing relevant in the journal for github-runner, it's generally super quiet. I'll try to binary-search the settings that cause the hangup now.

antifuchs commented 1 year ago

OK, so that is interesting. I have disabled all the restrictions on the github-runner unit, and it doesn't affect the behavior at all: The process still crashes. Since I'm running github-runner in an fhs user env, next I'm going to test whether flyctl survives being launched in one (which should also allow for much faster iteration times, if it fails there).

antifuchs commented 1 year ago

OMG so now I've got this reproduced quite a bit and it's a bit more insidious, even. To trigger this, you need to:

Run under the github-runner (still TODO how to reduce that)
Run a cgo-built dynamically linked go program (doesn't matter if it's 1.20 or 1.19) with the unstable toolchain - a cgo-built dynamically linked go program from the nixpkgs-22.11 branch works correctly!

I added a test program to the repro repo, under https://github.com/antifuchs/nixpkgs-bug-repro-flyctl/tree/main/testapp - when I run the testapp-cgo attribute (built to ensure cgo builds a dynamically-linked binary), it hangs exactly like flyctl does (which also is a cgo-built dynamically linked binary).

That binary correctly runs under the github-runner when built with pkgs-stable.buildGoModule (or buildGo119Module); it fails to start up within 1 minute built with pkgs-unstable.buildGo*Module.

So that's great(ℹ️) news, I'll work on getting github-runner out of the way of figuring this out now.

viraptor commented 1 year ago

So I'm curious if github-runner does anything with sandboxing / syscall filtering / resource limiting... An idea for further debugging (sorry, don't have time to dig into it myself): instead of running fly itself, run a script which dumps the output of export, ulimit -a, and maybe readlink /proc/$$/task/*/ns/* | sort -u Then try to run fly directly (without the github-runner) in a similar environment - same variables, same limits... the namespaces may be a bit harder to reproduce, but at least you'll know if cgroups limit you in some way.

antifuchs commented 1 year ago

Your suggestion brought an idea up - since the github runner is running in an FHS userenv, which includes a chroot-like env, and the program is loading dynamic libraries, might having mismatched libraries cause issues? Concretely, this is an FHS userenv built by nixpkgs release-22.11, used by a dynamically-linked program built with a nixpkgs-unstable toolchain. I believe the versions match up, but that seems like a factor I hadn't tested yet.

Update: Nope, I had tried that, and it didn't trigger the bug. Back to the drawing board!

antifuchs commented 1 year ago

Some updates: I used nsenter to run the programs in the github-runner's namespaces (validated with the diagnostics tool that they're all the same), and could not repro the hang. Next up would be the process environment and mayyyyybeee the file descriptor configuration.

NixOS / nixpkgs