libfuse / sshfs

A network filesystem client to connect to SSH servers
GNU General Public License v2.0
5.88k stars 494 forks source link

Process xxx (sshfs) of user 0 killed by SIGABRT - dumping core #310

Open mayelle02 opened 1 month ago

mayelle02 commented 1 month ago

Hi team,

I figured out an issue today. Since os migration to almalinux 8.9, the sshfs process killed after few minutes and the mountpoints aren't unavailable. In production, it's very terrible because, many clients are concerned.

In below, you'll see the report from abrt tool:

reason: sshfs killed by SIGABRT time: dim. 21 juil. 2024 21:14:21 CEST cmdline: sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,allow_other [user]@server01:/var/ftp /mnt uid: 0 Directory: /var/spool/abrt/ccpp-2024-07-21-21:14:21-14228 abrt_version: 2.10.9 analyzer: abrt-ccpp architecture: x86_64 executable: /usr/bin/sshfs global_pid: 14228 hostname: server01 kernel: 4.18.0-513.24.1.el8_9.x86_64 last_occurrence: 1721589261 os_release: AlmaLinux release 8.9 (Midnight Oncilla) pid: 14228 pwd: / tid: 852300 type: CCpp coredump: Binary file, 2442788864 bytes environ: Text file, 4868 bytes maps: Text file, 13514 bytes mountinfo: Text file, 20871 bytes cgroup: :12:freezer:/ :11:perf_event:/ :10:rdma:/ :9:memory:/ :8:cpuset:/ :7:net_cls,net_prio:/ :6:pids:/ :5:hugetlb:/ :4:cpu,cpuacct:/ :3:blkio:/ :2:devices:/ :1:name=systemd:/ limits: :Limit Soft Limit Hard Limit Units :Max cpu time unlimited unlimited seconds :Max file size unlimited unlimited bytes :Max data size unlimited unlimited bytes :Max stack size 8388608 unlimited bytes :Max core file size 0 0 bytes :Max resident set unlimited unlimited bytes :Max processes 512881 512881 processes :Max open files 1024 262144 files :Max locked memory 65536 65536 bytes :Max address space unlimited unlimited bytes :Max file locks unlimited unlimited locks :Max pending signals 512881 512881 signals :Max msgqueue size 819200 819200 bytes :Max nice priority 0 0 :Max realtime priority 0 0 :Max realtime timeout unlimited unlimited us

namespaces: :ipc : default :mnt : default :net : default :pid : default :uts : default :user : default :cgroup : default :pid_for_children : default

open_fds: :0:/dev/null :pos: 0 :flags: 0100002 :mnt_id: 24 : :1:/dev/null :pos: 0 :flags: 0100002 :mnt_id: 24 : :2:/dev/null :pos: 0 :flags: 0100002 :mnt_id: 24 : :3:/dev/fuse :pos: 0 :flags: 02100002 :mnt_id: 24 : :4:socket:[104258] :pos: 0 :flags: 02 :mnt_id: 10

os_info: :NAME="AlmaLinux" :VERSION="8.9 (Midnight Oncilla)" :ID="almalinux" :ID_LIKE="rhel centos fedora" :VERSION_ID="8.9" :PLATFORM_ID="platform:el8" :PRETTY_NAME="AlmaLinux 8.9 (Midnight Oncilla)" :ANSI_COLOR="0;34" :LOGO="fedora-logo-icon" :CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos" :HOME_URL="https://almalinux.org/" :DOCUMENTATION_URL="https://wiki.almalinux.org/" :BUG_REPORT_URL="https://bugs.almalinux.org/" : :ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8" :ALMALINUX_MANTISBT_PROJECT_VERSION="8.9" :REDHAT_SUPPORT_PRODUCT="AlmaLinux" :REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

proc_pid_status: :Name: sshfs :Umask: 0027 :State: D (disk sleep) :Tgid: 14228 :Ngid: 0 :Pid: 14228 :PPid: 1 :TracerPid: 0 :Uid: 0 0 0 0 :Gid: 0 0 0 0 :FDSize: 64 :Groups: 0 :NStgid: 14228 :NSpid: 14228 :NSpgid: 14228 :NSsid: 14228 :VmPeak: 3584572 kB :VmSize: 3387968 kB :VmLck: 0 kB :VmPin: 0 kB :VmHWM: 2724564 kB :VmRSS: 2628412 kB :RssAnon: 2626452 kB :RssFile: 1960 kB :RssShmem: 0 kB :VmData: 2782796 kB :VmStk: 136 kB :VmExe: 76 kB :VmLib: 9452 kB :VmPTE: 5376 kB :VmSwap: 0 kB :HugetlbPages: 0 kB :CoreDumping: 1 :THP_enabled: 1 :Threads: 12 :SigQ: 5/512881 :SigPnd: 0000000000000000 :ShdPnd: 0000000000000000 :SigBlk: 0000000000000000 :SigIgn: 0000000000001000 :SigCgt: 0000000180004003 :CapInh: 0000000000000000 :CapPrm: 000001ffffffffff :CapEff: 000001ffffffffff :CapBnd: 000001ffffffffff :CapAmb: 0000000000000000 :NoNewPrivs: 0 :Seccomp: 0 :Speculation_Store_Bypass: thread vulnerable :Cpus_allowed: ffff,ffffffff :Cpus_allowed_list: 0-47 :Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 :Mems_allowed_list: 0-1 :voluntary_ctxt_switches: 2 :nonvoluntary_ctxt_switches: 0

Could you help me please?

Regards,

h4sh5 commented 1 month ago

you probably need to use the coredump to get a backtrace of what the program was doing when SIGABRT happened.

don't upload the entire coredump here since there might be sensitive data in memory - but you can run gdb on the coredump like this to get a backtrace:

gdb /usr/bin/sshfs /path/to/coredump

then in the gdb prompt, type bt and paste the output here.

mayelle02 commented 1 week ago

Hi there,

Here the bt result: (gdb) bt

0 (memory address) in ?? ()

Backtrace stopped: Cannot access memory at address (memory address)

That's it

I don't know if this helpful?