search

superfly / litefs

FUSE-based file system for replicating SQLite databases across a cluster of machines
Apache License 2.0
3.78k stars 89 forks source link

Specifying same data & mount directory causes system to hang #135

Closed gkyildirim closed 1 year ago

gkyildirim commented 1 year ago

Hello,

I tried latest litefs 0.2.0 and it seems to stuck at some point. It is working with 0.1.1 though. When I start with litefs I can not access database and even list the mount directory.

This is how I start:

litefs -config litefs_SPA.yml

litefs_SPA.yml:

mount-dir: '/mnt/db'
data-dir: '/mnt/db'

http:
  addr: ':20202'

consul:
  url: 'http://10.16.18.230:8500'
  advertise-url: 'http://10.16.18.228:20202'

Also there is kernel info message:

Tue Oct 25 09:15:15 2022]  </TASK>
[Tue Oct 25 09:15:15 2022] INFO: task ls:2307 blocked for more than 120 seconds.
[Tue Oct 25 09:15:15 2022]       Tainted: P           OE     5.15.0-52-generic #58~20.04.1-Ubuntu
[Tue Oct 25 09:15:15 2022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Oct 25 09:15:15 2022] task:ls              state:D stack:    0 pid: 2307 ppid:  1986 flags:0x00004004
[Tue Oct 25 09:15:15 2022] Call Trace:
[Tue Oct 25 09:15:15 2022]  <TASK>
[Tue Oct 25 09:15:15 2022]  __schedule+0x2cd/0x8a0
[Tue Oct 25 09:15:15 2022]  schedule+0x4e/0xc0
[Tue Oct 25 09:15:15 2022]  request_wait_answer+0x136/0x210
[Tue Oct 25 09:15:15 2022]  ? wait_woken+0x60/0x60
[Tue Oct 25 09:15:15 2022]  fuse_simple_request+0x1ac/0x2f0
[Tue Oct 25 09:15:15 2022]  fuse_do_getattr+0xd7/0x340
[Tue Oct 25 09:15:15 2022]  fuse_getattr+0xa9/0x130
[Tue Oct 25 09:15:15 2022]  vfs_getattr_nosec+0xba/0xe0
[Tue Oct 25 09:15:15 2022]  vfs_getattr+0x37/0x50
[Tue Oct 25 09:15:15 2022]  vfs_statx+0x89/0x110
[Tue Oct 25 09:15:15 2022]  __do_sys_newlstat+0x3e/0x80
[Tue Oct 25 09:15:15 2022]  __x64_sys_newlstat+0x16/0x20
[Tue Oct 25 09:15:15 2022]  do_syscall_64+0x59/0xc0
[Tue Oct 25 09:15:15 2022]  ? exit_to_user_mode_prepare+0x3d/0x1c0
[Tue Oct 25 09:15:15 2022]  ? do_user_addr_fault+0x1e0/0x660
[Tue Oct 25 09:15:15 2022]  ? irqentry_exit_to_user_mode+0x9/0x20
[Tue Oct 25 09:15:15 2022]  ? irqentry_exit+0x1d/0x30
[Tue Oct 25 09:15:15 2022]  ? exc_page_fault+0x89/0x170
[Tue Oct 25 09:15:15 2022]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[Tue Oct 25 09:15:15 2022] RIP: 0033:0x7f7244ed557a
[Tue Oct 25 09:15:15 2022] RSP: 002b:00007ffca76b3b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000006
[Tue Oct 25 09:15:15 2022] RAX: ffffffffffffffda RBX: 00005555e6434550 RCX: 00007f7244ed557a
[Tue Oct 25 09:15:15 2022] RDX: 00005555e6434568 RSI: 00005555e6434568 RDI: 00007ffca76b3b10
[Tue Oct 25 09:15:15 2022] RBP: 00007ffca76b3ec0 R08: 0000000000000001 R09: 00000000e643ab00
[Tue Oct 25 09:15:15 2022] R10: 00007ffca76b3b14 R11: 0000000000000246 R12: 00005555e643ab43
[Tue Oct 25 09:15:15 2022] R13: 0000000000000003 R14: 00007ffca76b3b10 R15: 00005555e6434568
[Tue Oct 25 09:15:15 2022]  </TASK>

Maybe this has something to do with fuse but dont know how to dig it. As I said it is working with 0.1.1.

Thanks for this great project and let me know if I can help more.

benbjohnson commented 1 year ago

@gkyildirim Thanks for reporting the bug. Can you try sending a SIGQUIT signal (CTRL-\) to the litefs process and paste the reported stack trace in here?

I'm surprised you're having issues on Ubuntu 20.04. My development environment for LiteFS is Ubuntu 20.04.4 LTS and I haven't seen this issue before.

gkyildirim commented 1 year ago

Maybe 20.04.5 makes the difference? Or maybe there is a something dummy absent at virtual machine. I honestly do not have a clue.

Here is the stack trace. And also m

root@SPA:~$ /usr/local/bin/litefs-0.2.0 -config /root/moonstor/configs/litefs_SPA.yml
Using Consul to determine primary
initializing consul: key= url=http://10.16.18.230:8500 hostname=SPA advertise-url=http://10.16.18.228:20202
LiteFS mounted to: /mnt/db
http server listening on: http://localhost:20202
waiting to connect to cluster
cannot acquire lease or find primary, retrying: no primary
cannot acquire lease or find primary, retrying: no primary
cannot acquire lease or find primary, retrying: no primary
primary lease acquired, advertising as http://10.16.18.228:20202
connected to cluster, ready
SIGQUIT: quit
PC=0x46fb21 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/sys_linux_amd64.s:559 +0x21 fp=0x7ffd34bf4aa0 sp=0x7ffd34bf4a98 pc=0x46fb21
runtime.futexsleep(0x448d19?, 0x36500?, 0xc000034000?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/os_linux.go:69 +0x36 fp=0x7ffd34bf4af0 sp=0x7ffd34bf4aa0 pc=0x436056
runtime.notesleep(0xf07b48)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/lock_futex.go:160 +0x87 fp=0x7ffd34bf4b28 sp=0x7ffd34bf4af0 pc=0x40c107
runtime.mPark()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:1457 +0x25 fp=0x7ffd34bf4b48 sp=0x7ffd34bf4b28 pc=0x43f345
runtime.stoplockedm()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:2440 +0x65 fp=0x7ffd34bf4b98 sp=0x7ffd34bf4b48 pc=0x440f25
runtime.schedule()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:3185 +0x3d fp=0x7ffd34bf4bd0 sp=0x7ffd34bf4b98 pc=0x442bfd
runtime.park_m(0xc0001a6340?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:3355 +0x14d fp=0x7ffd34bf4c00 sp=0x7ffd34bf4bd0 pc=0x4431ad
runtime.mcall()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:448 +0x43 fp=0x7ffd34bf4c10 sp=0x7ffd34bf4c00 pc=0x46bb03

goroutine 1 [select]:
runtime.gopark(0xc000119e40?, 0x2?, 0xc0?, 0x2b?, 0xc000119d7c?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc0004c5bf8 sp=0xc0004c5bd8 pc=0x43c9b6
runtime.selectgo(0xc0004c5e40, 0xc000119d78, 0xc00010adc0?, 0x0, 0x2?, 0x1)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/select.go:328 +0x7bc fp=0xc0004c5d38 sp=0xc0004c5bf8 pc=0x44c83c
main.main()
    /home/runner/work/litefs/litefs/cmd/litefs/main.go:58 +0x42f fp=0xc0004c5f80 sp=0xc0004c5d38 pc=0x94672f
runtime.main()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:250 +0x212 fp=0xc0004c5fe0 sp=0xc0004c5f80 pc=0x43c5f2
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0004c5fe8 sp=0xc0004c5fe0 pc=0x46dc01

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000046fb0 sp=0xc000046f90 pc=0x43c9b6
runtime.goparkunlock(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:369
runtime.forcegchelper()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:302 +0xad fp=0xc000046fe0 sp=0xc000046fb0 pc=0x43c84d
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x46dc01
created by runtime.init.6
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000047790 sp=0xc000047770 pc=0x43c9b6
runtime.goparkunlock(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc0000477c8 sp=0xc000047790 pc=0x427177
runtime.gcenable.func1()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:178 +0x26 fp=0xc0000477e0 sp=0xc0000477c8 pc=0x41bde6
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x46dc01
created by runtime.gcenable
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc000066000?, 0xb55938?, 0x0?, 0x0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000047f70 sp=0xc000047f50 pc=0x43c9b6
runtime.goparkunlock(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:369
runtime.(*scavengerState).park(0xf070c0)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000047fa0 sp=0xc000047f70 pc=0x4251d3
runtime.bgscavenge(0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc000047fc8 sp=0xc000047fa0 pc=0x4257c5
runtime.gcenable.func2()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:179 +0x26 fp=0xc000047fe0 sp=0xc000047fc8 pc=0x41bd86
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x46dc01
created by runtime.gcenable
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0xab7ac8?, 0x0?, 0x0?, 0x2000000020?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000046628 sp=0xc000046608 pc=0x43c9b6
runtime.goparkunlock(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:369
runtime.runfinq()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mfinal.go:180 +0x10f fp=0xc0000467e0 sp=0xc000046628 pc=0x41aeef
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x46dc01
created by runtime.createfing
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mfinal.go:157 +0x45

goroutine 10 [select, locked to thread]:
runtime.gopark(0xc0000487a8?, 0x2?, 0x37?, 0xcd?, 0xc0000487a4?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000048618 sp=0xc0000485f8 pc=0x43c9b6
runtime.selectgo(0xc0000487a8, 0xc0000487a0, 0x0?, 0x0, 0xc000048701?, 0x1)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/select.go:328 +0x7bc fp=0xc000048758 sp=0xc000048618 pc=0x44c83c
runtime.ensureSigM.func1()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/signal_unix.go:991 +0x1b0 fp=0xc0000487e0 sp=0xc000048758 pc=0x450af0
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000487e8 sp=0xc0000487e0 pc=0x46dc01
created by runtime.ensureSigM
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/signal_unix.go:974 +0xbd

goroutine 11 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/lock_futex.go:236 +0x34 fp=0xc000048fa0 sp=0xc000048f68 pc=0x40c414
os/signal.signal_recv()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/sigqueue.go:152 +0x2f fp=0xc000048fc0 sp=0xc000048fa0 pc=0x46a36f
os/signal.loop()
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/signal/signal_unix.go:23 +0x19 fp=0xc000048fe0 sp=0xc000048fc0 pc=0x707b99
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000048fe8 sp=0xc000048fe0 pc=0x46dc01
created by os/signal.Notify.func1.1
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/signal/signal.go:151 +0x2a

goroutine 15 [select]:
runtime.gopark(0xc000283ea0?, 0x2?, 0x0?, 0x30?, 0xc000283e3c?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000283cb8 sp=0xc000283c98 pc=0x43c9b6
runtime.selectgo(0xc000283ea0, 0xc000283e38, 0x0?, 0x0, 0xc000285e40?, 0x1)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/select.go:328 +0x7bc fp=0xc000283df8 sp=0xc000283cb8 pc=0x44c83c
github.com/superfly/litefs.(*Store).monitorLeaseAsPrimary(0xc0001c2200, {0xb5cb70, 0xc00010afc0}, {0xb5ce80, 0xc00034acc0})
    /home/runner/work/litefs/litefs/store.go:531 +0x1f0 fp=0xc000283ee0 sp=0xc000283df8 pc=0x808750
github.com/superfly/litefs.(*Store).monitorLease(0xc0001c2200, {0xb5cb70, 0xc00010afc0})
    /home/runner/work/litefs/litefs/store.go:458 +0x1dd fp=0xc000283f50 sp=0xc000283ee0 pc=0x80807d
github.com/superfly/litefs.(*Store).Open.func1()
    /home/runner/work/litefs/litefs/store.go:131 +0x25 fp=0xc000283f78 sp=0xc000283f50 pc=0x805325
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x64 fp=0xc000283fe0 sp=0xc000283f78 pc=0x7fb404
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000283fe8 sp=0xc000283fe0 pc=0x46dc01
created by golang.org/x/sync/errgroup.(*Group).Go
    /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0xa5

goroutine 16 [semacquire]:
runtime.gopark(0x7fab1a4d1fff?, 0x203000?, 0x60?, 0x3?, 0x203000?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000049bf8 sp=0xc000049bd8 pc=0x43c9b6
runtime.goparkunlock(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:369
runtime.semacquire1(0xc0001c2204, 0x80?, 0x3, 0x1)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/sema.go:150 +0x1fe fp=0xc000049c60 sp=0xc000049bf8 pc=0x44d8fe
sync.runtime_SemacquireMutex(0x58?, 0xb8?, 0x60?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/sema.go:77 +0x25 fp=0xc000049c90 sp=0xc000049c60 pc=0x469d25
sync.(*Mutex).lockSlow(0xc0001c2200)
    /opt/hostedtoolcache/go/1.19.0/x64/src/sync/mutex.go:171 +0x165 fp=0xc000049ce0 sp=0xc000049c90 pc=0x477b05
sync.(*Mutex).Lock(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/sync/mutex.go:90
github.com/superfly/litefs.(*Store).DBs(0xc0001c2200)
    /home/runner/work/litefs/litefs/store.go:303 +0x77 fp=0xc000049df0 sp=0xc000049ce0 pc=0x806857
github.com/superfly/litefs.(*Store).EnforceRetention(0xc0001c2200, {0xb5cb70, 0xc00010afc0})
    /home/runner/work/litefs/litefs/store.go:630 +0xd4 fp=0xc000049eb0 sp=0xc000049df0 pc=0x8095f4
github.com/superfly/litefs.(*Store).monitorRetention(0xc0001c2200, {0xb5cb70, 0xc00010afc0})
    /home/runner/work/litefs/litefs/store.go:619 +0x109 fp=0xc000049f50 sp=0xc000049eb0 pc=0x809409
github.com/superfly/litefs.(*Store).Open.func2()
    /home/runner/work/litefs/litefs/store.go:135 +0x25 fp=0xc000049f78 sp=0xc000049f50 pc=0x8052e5
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x64 fp=0xc000049fe0 sp=0xc000049f78 pc=0x7fb404
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000049fe8 sp=0xc000049fe0 pc=0x46dc01
created by golang.org/x/sync/errgroup.(*Group).Go
    /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0xa5

goroutine 41 [IO wait]:
runtime.gopark(0xd0?, 0x20?, 0x48?, 0xcc?, 0xc00007cc30?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc00007cbc0 sp=0xc00007cba0 pc=0x43c9b6
runtime.netpollblock(0xd0?, 0x4a400?, 0xc0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/netpoll.go:526 +0xf7 fp=0xc00007cbf8 sp=0xc00007cbc0 pc=0x435437
internal/poll.runtime_pollWait(0x7fab1a01b0e8, 0x72)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/netpoll.go:305 +0x89 fp=0xc00007cc18 sp=0xc00007cbf8 pc=0x468349
internal/poll.(*pollDesc).wait(0xc0001ac500?, 0x4?, 0x0)
    /opt/hostedtoolcache/go/1.19.0/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00007cc40 sp=0xc00007cc18 pc=0x4dcd12
internal/poll.(*pollDesc).waitRead(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0001ac500)
    /opt/hostedtoolcache/go/1.19.0/x64/src/internal/poll/fd_unix.go:614 +0x234 fp=0xc00007ccd8 sp=0xc00007cc40 pc=0x4e1f94
net.(*netFD).accept(0xc0001ac500)
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/fd_unix.go:172 +0x35 fp=0xc00007cd90 sp=0xc00007ccd8 pc=0x5dcbd5
net.(*TCPListener).accept(0xc0000108b8)
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/tcpsock_posix.go:142 +0x28 fp=0xc00007cdc0 sp=0xc00007cd90 pc=0x5f21e8
net.(*TCPListener).Accept(0xc0000108b8)
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/tcpsock.go:288 +0x3d fp=0xc00007cdf0 sp=0xc00007cdc0 pc=0x5f13bd
net/http.(*onceCloseListener).Accept(0xb5cb70?)
    <autogenerated>:1 +0x2a fp=0xc00007ce08 sp=0xc00007cdf0 pc=0x6fdd0a
net/http.(*Server).Serve(0xc0001ae0f0, {0xb5c268, 0xc0000108b8})
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3070 +0x385 fp=0xc00007cf38 sp=0xc00007ce08 pc=0x6d7fc5
github.com/superfly/litefs/http.(*Server).Serve.func1()
    /home/runner/work/litefs/litefs/http/server.go:74 +0x2d fp=0xc00007cf78 sp=0xc00007cf38 pc=0x90f3cd
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x64 fp=0xc00007cfe0 sp=0xc00007cf78 pc=0x7fb404
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x46dc01
created by golang.org/x/sync/errgroup.(*Group).Go
    /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0xa5

goroutine 40 [syscall]:
syscall.Syscall(0x47812e?, 0xef48c0?, 0x2?, 0x9ea0c0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/syscall_linux.go:68 +0x27 fp=0xc000281458 sp=0xc0002813e8 pc=0x483987
syscall.read(0xef48c0?, {0xc0004a2000?, 0x8?, 0x0?})
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/zsyscall_linux_amd64.go:696 +0x45 fp=0xc000281498 sp=0xc000281458 pc=0x481645
syscall.Read(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/syscall_unix.go:183
bazil.org/fuse.(*Conn).ReadRequest(0xc00006ab40)
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fuse.go:580 +0xc6 fp=0xc000281eb0 sp=0xc000281498 pc=0x86a6e6
bazil.org/fuse/fs.(*Server).Serve(0xc000290000, {0xb57ba0?, 0xc000074eb0})
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:501 +0x3ef fp=0xc000281f98 sp=0xc000281eb0 pc=0x87c50f
github.com/superfly/litefs/fuse.(*FileSystem).Mount.func2()
    /home/runner/work/litefs/litefs/fuse/file_system.go:75 +0x28 fp=0xc000281fe0 sp=0xc000281f98 pc=0x8881e8
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000281fe8 sp=0xc000281fe0 pc=0x46dc01
created by github.com/superfly/litefs/fuse.(*FileSystem).Mount
    /home/runner/work/litefs/litefs/fuse/file_system.go:74 +0x2b0

goroutine 22 [IO wait]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0xa?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc0000d5af8 sp=0xc0000d5ad8 pc=0x43c9b6
runtime.netpollblock(0x481645?, 0x48f6f7?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/netpoll.go:526 +0xf7 fp=0xc0000d5b30 sp=0xc0000d5af8 pc=0x435437
internal/poll.runtime_pollWait(0x7fab1a01ad28, 0x72)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/netpoll.go:305 +0x89 fp=0xc0000d5b50 sp=0xc0000d5b30 pc=0x468349
internal/poll.(*pollDesc).wait(0xc000220800?, 0xc0000cf000?, 0x0)
    /opt/hostedtoolcache/go/1.19.0/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0000d5b78 sp=0xc0000d5b50 pc=0x4dcd12
internal/poll.(*pollDesc).waitRead(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000220800, {0xc0000cf000, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.19.0/x64/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0000d5bf8 sp=0xc0000d5b78 pc=0x4de07a
net.(*netFD).Read(0xc000220800, {0xc0000cf000?, 0x406271?, 0x4?})
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/fd_posix.go:55 +0x29 fp=0xc0000d5c40 sp=0xc0000d5bf8 pc=0x5daac9
net.(*conn).Read(0xc0000ac028, {0xc0000cf000?, 0x0?, 0x0?})
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/net.go:183 +0x45 fp=0xc0000d5c88 sp=0xc0000d5c40 pc=0x5e9625
net/http.(*persistConn).Read(0xc00027e120, {0xc0000cf000?, 0xc0000cc120?, 0xc0000d5d30?})
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/transport.go:1929 +0x4e fp=0xc0000d5ce8 sp=0xc0000d5c88 pc=0x6ed94e
bufio.(*Reader).fill(0xc0000b0420)
    /opt/hostedtoolcache/go/1.19.0/x64/src/bufio/bufio.go:106 +0xff fp=0xc0000d5d20 sp=0xc0000d5ce8 pc=0x65c79f
bufio.(*Reader).Peek(0xc0000b0420, 0x1)
    /opt/hostedtoolcache/go/1.19.0/x64/src/bufio/bufio.go:144 +0x5d fp=0xc0000d5d40 sp=0xc0000d5d20 pc=0x65c8fd
net/http.(*persistConn).readLoop(0xc00027e120)
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/transport.go:2093 +0x1ac fp=0xc0000d5fc8 sp=0xc0000d5d40 pc=0x6ee76c
net/http.(*Transport).dialConn.func5()
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/transport.go:1751 +0x26 fp=0xc0000d5fe0 sp=0xc0000d5fc8 pc=0x6ecf46
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000d5fe8 sp=0xc0000d5fe0 pc=0x46dc01
created by net/http.(*Transport).dialConn
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/transport.go:1751 +0x173e

goroutine 23 [select]:
runtime.gopark(0xc00007df90?, 0x2?, 0xd8?, 0xdd?, 0xc00007df24?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc00007dd90 sp=0xc00007dd70 pc=0x43c9b6
runtime.selectgo(0xc00007df90, 0xc00007df20, 0xc0000ba040?, 0x0, 0xc00009a960?, 0x1)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/select.go:328 +0x7bc fp=0xc00007ded0 sp=0xc00007dd90 pc=0x44c83c
net/http.(*persistConn).writeLoop(0xc00027e120)
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/transport.go:2392 +0xf5 fp=0xc00007dfc8 sp=0xc00007ded0 pc=0x6f03f5
net/http.(*Transport).dialConn.func6()
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/transport.go:1752 +0x26 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0x6ecee6
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x46dc01
created by net/http.(*Transport).dialConn
    /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/transport.go:1752 +0x1791

goroutine 54 [GC worker (idle)]:
runtime.gopark(0xde74ea8f13?, 0x0?, 0x0?, 0x0?, 0xc000296fd0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000296f50 sp=0xc000296f30 pc=0x43c9b6
runtime.gcBgMarkWorker()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc000296fe0 sp=0xc000296f50 pc=0x41df31
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000296fe8 sp=0xc000296fe0 pc=0x46dc01
created by runtime.gcBgMarkStartWorkers
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1159 +0x25

goroutine 66 [GC worker (idle)]:
runtime.gopark(0xde74e7762b?, 0x2?, 0xca?, 0x37?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000298750 sp=0xc000298730 pc=0x43c9b6
runtime.gcBgMarkWorker()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc0002987e0 sp=0xc000298750 pc=0x41df31
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0002987e8 sp=0xc0002987e0 pc=0x46dc01
created by runtime.gcBgMarkStartWorkers
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1159 +0x25

goroutine 53 [GC worker (idle)]:
runtime.gopark(0xde74ea494b?, 0x0?, 0x0?, 0x0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000297750 sp=0xc000297730 pc=0x43c9b6
runtime.gcBgMarkWorker()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc0002977e0 sp=0xc000297750 pc=0x41df31
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0002977e8 sp=0xc0002977e0 pc=0x46dc01
created by runtime.gcBgMarkStartWorkers
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1159 +0x25

goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000297f50 sp=0xc000297f30 pc=0x43c9b6
runtime.gcBgMarkWorker()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc000297fe0 sp=0xc000297f50 pc=0x41df31
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000297fe8 sp=0xc000297fe0 pc=0x46dc01
created by runtime.gcBgMarkStartWorkers
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1159 +0x25

goroutine 45 [syscall]:
syscall.Syscall6(0xc00034e450?, 0x14?, 0x7fab41603f18?, 0xd0?, 0xc000080c00?, 0xc0003801a0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/syscall_linux.go:90 +0x36 fp=0xc0000d0740 sp=0xc0000d06b8 pc=0x483a16
syscall.fstatat(0x14?, {0xc00034e450?, 0x1?}, 0xc0003801d8, 0xc00034e450?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/zsyscall_linux_amd64.go:1439 +0x9a fp=0xc0000d07b8 sp=0xc0000d0740 pc=0x482d1a
syscall.Stat(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/syscall_linux_amd64.go:59
os.statNolog.func1(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/stat_unix.go:32
os.ignoringEINTR(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/file_posix.go:245
os.statNolog({0xc00034e450, 0x14})
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/stat_unix.go:31 +0x58 fp=0xc0000d0808 sp=0xc0000d07b8 pc=0x4edef8
os.Stat({0xc00034e450, 0x14})
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/stat.go:13 +0x34 fp=0xc0000d0828 sp=0xc0000d0808 pc=0x4eda34
os.MkdirAll({0xc00034e450, 0x14}, 0x219020?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/path.go:20 +0x2b fp=0xc0000d0868 sp=0xc0000d0828 pc=0x4ed20b
github.com/superfly/litefs.(*Store).CreateDB(0xc0001c2200, {0xc000219020, 0x8})
    /home/runner/work/litefs/litefs/store.go:327 +0x19b fp=0xc0000d09c0 sp=0xc0000d0868 pc=0x806c3b
github.com/superfly/litefs/fuse.(*RootNode).createDatabase(0xc000010948, {0x8?, 0x9d19a0?}, {0xc000219020?, 0x40a8f3?}, 0x7fab19f52888?, 0xb0?)
    /home/runner/work/litefs/litefs/fuse/root_node.go:155 +0x3e fp=0xc0000d0a40 sp=0xc0000d09c0 pc=0x88ac1e
github.com/superfly/litefs/fuse.(*RootNode).Create(0xc000010948, {0xb5cb70, 0xc0003548c0}, 0xc0002228c0, 0x8?)
    /home/runner/work/litefs/litefs/fuse/root_node.go:137 +0x23d fp=0xc0000d0ae0 sp=0xc0000d0a40 pc=0x88a9fd
bazil.org/fuse/fs.(*Server).handleRequest(0xc000290000, {0xb5cb70, 0xc0003548c0}, {0xb57c20?, 0xc000010948}, 0xc00010b240, {0xb5b608?, 0xc0002228c0?}, 0xc0000d0f08)
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:1244 +0x2acd fp=0xc0000d0de0 sp=0xc0000d0ae0 pc=0x88264d
bazil.org/fuse/fs.(*Server).serve(0xc000290000, {0xb5b608, 0xc0002228c0})
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:1015 +0x625 fp=0xc0000d0f98 sp=0xc0000d0de0 pc=0x87f425
bazil.org/fuse/fs.(*Server).Serve.func1()
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:512 +0x69 fp=0xc0000d0fe0 sp=0xc0000d0f98 pc=0x87c729
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000d0fe8 sp=0xc0000d0fe0 pc=0x46dc01
created by bazil.org/fuse/fs.(*Server).Serve
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:510 +0x3df

goroutine 46 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000298f50 sp=0xc000298f30 pc=0x43c9b6
runtime.gcBgMarkWorker()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc000298fe0 sp=0xc000298f50 pc=0x41df31
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000298fe8 sp=0xc000298fe0 pc=0x46dc01
created by runtime.gcBgMarkStartWorkers
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1159 +0x25

goroutine 47 [GC worker (idle)]:
runtime.gopark(0xde74ea4d8e?, 0x3?, 0xc?, 0x51?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000299750 sp=0xc000299730 pc=0x43c9b6
runtime.gcBgMarkWorker()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc0002997e0 sp=0xc000299750 pc=0x41df31
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0002997e8 sp=0xc0002997e0 pc=0x46dc01
created by runtime.gcBgMarkStartWorkers
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/mgc.go:1159 +0x25

rax    0xca
rbx    0x0
rcx    0x46fb23
rdx    0x0
rdi    0xf07b48
rsi    0x80
rbp    0x7ffd34bf4ae0
rsp    0x7ffd34bf4a98
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7ffd34bf4ad8
r13    0x58
r14    0xf07460
r15    0x7fab1a2d7815
rip    0x46fb21
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

And here is my kernel if it helps:

root@SPA:~$ uname -a
Linux SPA 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
benbjohnson commented 1 year ago

@gkyildirim Ah, I think I see the issue. The data-dir and mount-dir can't be the same location. LiteFS v0.1.1 automatically created the data directory but now it has to be specified. It looks like LiteFS is trying to create a directory but it hangs on this goroutine:

goroutine 45 [syscall]:
syscall.Syscall6(0xc00034e450?, 0x14?, 0x7fab41603f18?, 0xd0?, 0xc000080c00?, 0xc0003801a0?, 0x0?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/syscall_linux.go:90 +0x36 fp=0xc0000d0740 sp=0xc0000d06b8 pc=0x483a16
syscall.fstatat(0x14?, {0xc00034e450?, 0x1?}, 0xc0003801d8, 0xc00034e450?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/zsyscall_linux_amd64.go:1439 +0x9a fp=0xc0000d07b8 sp=0xc0000d0740 pc=0x482d1a
syscall.Stat(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/syscall/syscall_linux_amd64.go:59
os.statNolog.func1(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/stat_unix.go:32
os.ignoringEINTR(...)
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/file_posix.go:245
os.statNolog({0xc00034e450, 0x14})
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/stat_unix.go:31 +0x58 fp=0xc0000d0808 sp=0xc0000d07b8 pc=0x4edef8
os.Stat({0xc00034e450, 0x14})
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/stat.go:13 +0x34 fp=0xc0000d0828 sp=0xc0000d0808 pc=0x4eda34
os.MkdirAll({0xc00034e450, 0x14}, 0x219020?)
    /opt/hostedtoolcache/go/1.19.0/x64/src/os/path.go:20 +0x2b fp=0xc0000d0868 sp=0xc0000d0828 pc=0x4ed20b
github.com/superfly/litefs.(*Store).CreateDB(0xc0001c2200, {0xc000219020, 0x8})
    /home/runner/work/litefs/litefs/store.go:327 +0x19b fp=0xc0000d09c0 sp=0xc0000d0868 pc=0x806c3b
github.com/superfly/litefs/fuse.(*RootNode).createDatabase(0xc000010948, {0x8?, 0x9d19a0?}, {0xc000219020?, 0x40a8f3?}, 0x7fab19f52888?, 0xb0?)
    /home/runner/work/litefs/litefs/fuse/root_node.go:155 +0x3e fp=0xc0000d0a40 sp=0xc0000d09c0 pc=0x88ac1e
github.com/superfly/litefs/fuse.(*RootNode).Create(0xc000010948, {0xb5cb70, 0xc0003548c0}, 0xc0002228c0, 0x8?)
    /home/runner/work/litefs/litefs/fuse/root_node.go:137 +0x23d fp=0xc0000d0ae0 sp=0xc0000d0a40 pc=0x88a9fd
bazil.org/fuse/fs.(*Server).handleRequest(0xc000290000, {0xb5cb70, 0xc0003548c0}, {0xb57c20?, 0xc000010948}, 0xc00010b240, {0xb5b608?, 0xc0002228c0?}, 0xc0000d0f08)
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:1244 +0x2acd fp=0xc0000d0de0 sp=0xc0000d0ae0 pc=0x88264d
bazil.org/fuse/fs.(*Server).serve(0xc000290000, {0xb5b608, 0xc0002228c0})
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:1015 +0x625 fp=0xc0000d0f98 sp=0xc0000d0de0 pc=0x87f425
bazil.org/fuse/fs.(*Server).Serve.func1()
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:512 +0x69 fp=0xc0000d0fe0 sp=0xc0000d0f98 pc=0x87c729
runtime.goexit()
    /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000d0fe8 sp=0xc0000d0fe0 pc=0x46dc01
created by bazil.org/fuse/fs.(*Server).Serve
    /home/runner/go/pkg/mod/bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:510 +0x3df

That has a lock that it obtains (typically momentarily) but since it hangs then it locks out most other requests.

If you change the directories to be different then it should work. I'll fix it in v0.3.0 so that it validates that they're not the same.

Thanks for the bug report!