search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.05k stars 3.01k forks source link

panic in go-unixfs@v0.3.1/hamt/hamt.go #9063

Closed lidel closed 1 year ago

lidel commented 2 years ago

Version

0.12.2

(updated to 0.13.0, now waiting to see if it reproduces there too)

Description

Extracted from https://github.com/ipfs/go-ipfs/issues/8694#issuecomment-1167353280

Running ipfs files mkdir -p followed by ipfs files write -c in a tight loop sometimes produces a panic:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x18688c6]

goroutine 441888916 [running]:
github.com/ipfs/go-unixfs/hamt.(*Shard).childLinkType(0xc04e3ff420, 0x0, 0x40, 0x40, 0x41)
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:293 +0x26
github.com/ipfs/go-unixfs/hamt.(*Shard).walkChildren(0xc04e3ff420, 0xc037be7880, 0x0, 0x1, 0x1)
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:400 +0x27d
github.com/ipfs/go-unixfs/hamt.parallelShardWalk.func1(0x1, 0xc03f03fc00)
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:467 +0x107
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc014813b30, 0xc0ee853f20)
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x66

Relevant code calling /api/v0/files/mkdir and /api/v0/files/write over HTTP RPC API client is here. The panic happens randomly for them, every 2-3 days.

Asked them to update to 0.13.0 and report if it happens again.

cc @schomatis

schomatis commented 2 years ago

This is on my radar but will wait for confirmation on the new version and any possible reproduction of the issue.

kuzdogan commented 2 years ago

Although it seemed to be stable for longer time than usual, we encountered the same issue today. Here's the output below. I added some comments starting with #. The relevant code is fully open-sourced and can be found here.

It is basically a Docker container with ipfs. Initiall it runs the entrypoint.sh which adds the whole repository of verified contracts on Sourcify, copies it to /contracts in MFS, and publishes the MFS root under this IPNS. For each new verified contract added to the repository, the files are added to their respective directory in MFS over the IPFS API here. Also, the IPNS is updated every 6 hours with a cron job with publish.sh

Here's the output of the container:

sourcify@sourcify:~$ docker logs ipfs-stable
generating ED25519 keypair...done
peer identity: 12D3KooWPT7etYMSLhG1YLjWHdQqdvEKdP8ku4y4WyHmMpoUBaTq
initializing IPFS node at /root/.ipfs
to get started, enter:

    ipfs cat /ipfs/QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc/readme

2022-07-11T15:49:39.153Z    ERROR   provider.queue  queue/queue.go:124  Failed to enqueue cid: datastore closed
k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p
# Sleeping to let daemon initialize. Otherwise can't start adding.
Sleeping 30 seconds
Initializing daemon...
go-ipfs version: 0.13.0
Repo version: 12
System version: amd64/linux
Golang version: go1.18.3
2022/07/11 15:49:44 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/172.18.0.7/tcp/4001
Swarm listening on /ip4/172.18.0.7/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/10.10.42.2/tcp/4003
Swarm announcing /ip4/10.10.42.2/udp/4003/quic
Swarm announcing /ip4/178.19.221.38/tcp/4003
Swarm announcing /ip4/178.19.221.38/udp/4003/quic
API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
Daemon is ready
Sleeped 30 seconds
Mon Jul 11 15:50:13 UTC 2022
# Add takes a long time. Start time above. See below the finish time
Starting ipfs add
2022-07-11T15:53:40.036Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-11T17:08:04.920Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.720Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.773Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.774Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.896Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.898Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:08:16.899Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:08:39.323Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:09:14.892Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:09:14.895Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.078Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.079Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.079Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.079Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.079Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.542Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.542Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.542Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.611Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:10:28.611Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:12:16.557Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-11T17:12:16.557Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-11T17:12:25.387Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:12:25.390Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:12:32.087Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:12:32.087Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:13:34.328Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:14:08.773Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:14:16.242Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:14:16.242Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:14:16.243Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:14:25.599Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.617Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.617Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.618Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.618Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:15:11.619Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:15:36.299Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:18:51.774Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:18:51.774Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-11T17:20:22.176Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-11T17:20:23.173Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-11T17:20:23.320Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-11T17:20:42.076Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
Finished ipfs add! New ipfs hash: QmXYqk3Uc93xLrBtCqMixiNgeUT1gYS8eeQxVHPwuNDFF9
# Add finish time below. Took around 10 hours.
Tue Jul 12 01:51:43 UTC 2022
# Updating the old /contracts hash with the newly added one
Removing /contracts from MFSc
Removed /contracts from MFSc
Copying QmXYqk3Uc93xLrBtCqMixiNgeUT1gYS8eeQxVHPwuNDFF9 to MFS at /contracts
Copied QmXYqk3Uc93xLrBtCqMixiNgeUT1gYS8eeQxVHPwuNDFF9 to MFS at /contracts
Tue Jul 12 01:51:45 UTC 2022
Adding manifest and stats
Added manifest: QmNt8QffJBHpXpYinMScaZEwtDsiZr7bvz8iprhuCMUxbq and stats: Qma8Q9R7q3YAxyQzMX5t6yXaf7NLChcjudtLhw4omnZ7dE
Publishing rootHash QmTc7LAHwiwFZN6vVKAbN127SVv1dnt8rEGJtN25BvLase under ipns key
2022-07-12T01:51:58.792Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmTc7LAHwiwFZN6vVKAbN127SVv1dnt8rEGJtN25BvLase
Published rootHash QmTc7LAHwiwFZN6vVKAbN127SVv1dnt8rEGJtN25BvLase under ipns key
Tue Jul 12 01:52:02 UTC 2022
Successfully added and published the repository
2022-07-12T06:59:01.779Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:01.779Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:01.779Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:01.780Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:02.516Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:02.517Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:02.517Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:02.517Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:02.517Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:02.558Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T06:59:02.559Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-12T23:05:54.800Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.038Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.172Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.173Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.173Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.256Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.362Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.618Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.619Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.652Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.658Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.676Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.677Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.686Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.687Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.687Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.704Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.725Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.725Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.731Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.731Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.731Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.732Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.737Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.737Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.738Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.743Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.743Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.749Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.749Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.749Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.891Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.891Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.892Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.892Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:55.907Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:56.751Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:56.751Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:56.976Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:57.603Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:57.610Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:05:59.463Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-12T23:10:01.579Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-12T23:10:01.610Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-12T23:10:38.139Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-12T23:10:38.140Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-12T23:11:40.279Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-13T18:01:58.354Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-13T18:01:58.354Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-13T18:01:58.355Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-13T18:01:58.355Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-13T18:01:58.355Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-13T18:01:58.355Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-13T18:01:58.355Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-13T18:01:58.355Z    ERROR   fullrtdht   fullrt/dht.go:431   key not found in map
2022-07-14T20:02:35.241Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-14T20:02:36.642Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-14T20:07:46.990Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-14T20:08:58.739Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-14T20:08:58.740Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-14T20:09:45.324Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-14T20:09:45.325Z    ERROR   autorelay   autorelay/autorelay.go:104  failed to start relay finder
2022-07-14T20:09:49.192Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-14T20:09:49.484Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-14T20:11:38.343Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
2022-07-14T20:11:38.784Z    ERROR   dht go-libp2p-kad-dht@v0.16.0/dht_net.go:47 ignoring incoming dht message while not in server mode
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x19e3bf8]

goroutine 17608019870 [running]:
github.com/ipfs/go-unixfs/hamt.(*Shard).childLinkType(0x23d59a0?, 0xc0f2a5b800?)
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:293 +0x18
github.com/ipfs/go-unixfs/hamt.(*Shard).walkChildren(0xc0a43cd730, 0xc074e53700)
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:400 +0x20c
github.com/ipfs/go-unixfs/hamt.parallelShardWalk.func1()
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:467 +0x25a
golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x8d
Terminated

# Here the container starts automatically after crashing
Sleeping 30 seconds
Initializing daemon...
go-ipfs version: 0.13.0
Repo version: 12
System version: amd64/linux
Golang version: go1.18.3
Sleeped 30 seconds
Fri Jul 15 02:26:24 UTC 2022
Starting ipfs add
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Finished ipfs add! New ipfs hash:
Fri Jul 15 02:26:25 UTC 2022
Removing /contracts from MFSc
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Removed /contracts from MFSc
Copying  to MFS at /contracts
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Copied  to MFS at /contracts
Fri Jul 15 02:26:27 UTC 2022
Adding manifest and stats
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Error: lock /root/.ipfs/repo.lock: someone else has the lock
2022/07/15 02:26:29 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Error: lock /root/.ipfs/repo.lock: someone else has the lock
Added manifest:  and stats:
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/172.18.0.7/tcp/4001
Swarm listening on /ip4/172.18.0.7/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/10.10.42.2/tcp/4003
Swarm announcing /ip4/10.10.42.2/udp/4003/quic
Swarm announcing /ip4/178.19.221.38/tcp/4003
Swarm announcing /ip4/178.19.221.38/udp/4003/quic
API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
Daemon is ready
Publishing rootHash QmQKdndr4NGrnoBcuHkTRkLjqqieTbzhEMxZmBMF7WFNYZ under ipns key
2022-07-15T02:26:30.964Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Error: failed to complete put
Published rootHash QmQKdndr4NGrnoBcuHkTRkLjqqieTbzhEMxZmBMF7WFNYZ under ipns key
Fri Jul 15 02:26:31 UTC 2022
Successfully added and published the repository
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x19e3bf8]

goroutine 224787303 [running]:
github.com/ipfs/go-unixfs/hamt.(*Shard).childLinkType(0xc04dc29490?, 0xc025cdb8d8?)
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:293 +0x18
github.com/ipfs/go-unixfs/hamt.(*Shard).walkChildren(0xc04dc29490, 0xc05bdf39c0)
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:400 +0x20c
github.com/ipfs/go-unixfs/hamt.parallelShardWalk.func1()
    github.com/ipfs/go-unixfs@v0.3.1/hamt/hamt.go:467 +0x25a
golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x8d
Terminated
# Auto-restart container after crash again
Sleeping 30 seconds
Initializing daemon...
go-ipfs version: 0.13.0
Repo version: 12
System version: amd64/linux
Golang version: go1.18.3
2022/07/15 09:27:44 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/172.18.0.7/tcp/4001
Swarm listening on /ip4/172.18.0.7/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/10.10.42.2/tcp/4003
Swarm announcing /ip4/10.10.42.2/udp/4003/quic
Swarm announcing /ip4/178.19.221.38/tcp/4003
Swarm announcing /ip4/178.19.221.38/udp/4003/quic
API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
Daemon is ready
Sleeped 30 seconds
Fri Jul 15 09:27:46 UTC 2022
Starting ipfs add

Also adding the output of the periodic publish operations:

root@e7ebab206713:/app# cat /var/log/cron.log
Tue Jul 12 06:00:01 UTC 2022
# Running a find script to publish the verified contract stats. This gets published in stats.json
Started find in repository for stats
# ignore these as we take the superset of `full_match` folder and not all of them are found in partial_match
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats 
Tue Jul 12 06:00:47 UTC 2022
Publishing rootHash QmZKpmqNsUFBJ4tErBfL7wanvwtRiKTWeo1J3isgMxCK6o under ipns key
2022-07-12T06:14:15.966Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmZKpmqNsUFBJ4tErBfL7wanvwtRiKTWeo1J3isgMxCK6o
Published rootHash QmZKpmqNsUFBJ4tErBfL7wanvwtRiKTWeo1J3isgMxCK6o under ipns key
Tue Jul 12 06:14:16 UTC 2022
Tue Jul 12 12:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Tue Jul 12 12:00:42 UTC 2022
Publishing rootHash QmRV5j64E5xJV5RxXScuiakcc9n9o1prxjxNWByuzpyixY under ipns key
2022-07-12T12:25:18.670Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmRV5j64E5xJV5RxXScuiakcc9n9o1prxjxNWByuzpyixY
Published rootHash QmRV5j64E5xJV5RxXScuiakcc9n9o1prxjxNWByuzpyixY under ipns key
Tue Jul 12 12:25:20 UTC 2022
Tue Jul 12 18:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Tue Jul 12 18:00:45 UTC 2022
Publishing rootHash QmV4kAkBjCWbKAkvM54wHftVhUyPnDLeLBipiueLLXpi2F under ipns key
2022-07-12T18:43:17.596Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmV4kAkBjCWbKAkvM54wHftVhUyPnDLeLBipiueLLXpi2F
Published rootHash QmV4kAkBjCWbKAkvM54wHftVhUyPnDLeLBipiueLLXpi2F under ipns key
Tue Jul 12 18:43:18 UTC 2022
Wed Jul 13 00:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Wed Jul 13 00:00:42 UTC 2022
Publishing rootHash QmTt4Sh8HjzRie4wDCjVES2qqxKtEEWv5VzezXWATLtZ2z under ipns key
2022-07-13T00:41:56.020Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmTt4Sh8HjzRie4wDCjVES2qqxKtEEWv5VzezXWATLtZ2z
Published rootHash QmTt4Sh8HjzRie4wDCjVES2qqxKtEEWv5VzezXWATLtZ2z under ipns key
Wed Jul 13 00:41:58 UTC 2022
Wed Jul 13 06:00:05 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Wed Jul 13 06:00:46 UTC 2022
Publishing rootHash QmecqaAh9boMrrAXdPhuEtDrra1GdTrYGiDRsAkfczzYzk under ipns key
2022-07-13T06:51:54.754Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmecqaAh9boMrrAXdPhuEtDrra1GdTrYGiDRsAkfczzYzk
Published rootHash QmecqaAh9boMrrAXdPhuEtDrra1GdTrYGiDRsAkfczzYzk under ipns key
Wed Jul 13 06:51:59 UTC 2022
Wed Jul 13 12:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Wed Jul 13 12:00:49 UTC 2022
Publishing rootHash QmNXhrdk9p7VZN3YXLDtvgN8bXGpAbQLgfG9wMgb4yV4wA under ipns key
2022-07-13T13:31:12.516Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmNXhrdk9p7VZN3YXLDtvgN8bXGpAbQLgfG9wMgb4yV4wA
Published rootHash QmNXhrdk9p7VZN3YXLDtvgN8bXGpAbQLgfG9wMgb4yV4wA under ipns key
Wed Jul 13 13:31:13 UTC 2022
Wed Jul 13 18:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Wed Jul 13 18:00:43 UTC 2022
Publishing rootHash QmQjBWuVyRvL3jqwVGd9Jb4931hHdFGm1xDr4kGGuov3NP under ipns key
2022-07-13T19:38:46.913Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmQjBWuVyRvL3jqwVGd9Jb4931hHdFGm1xDr4kGGuov3NP
Published rootHash QmQjBWuVyRvL3jqwVGd9Jb4931hHdFGm1xDr4kGGuov3NP under ipns key
Wed Jul 13 19:38:47 UTC 2022
Thu Jul 14 00:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Thu Jul 14 00:00:48 UTC 2022
Publishing rootHash QmUGJR8aHNfQ5wAUKTcZrawcWcz3X5CECm4jy9422PQr9e under ipns key
2022-07-14T01:21:09.421Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmUGJR8aHNfQ5wAUKTcZrawcWcz3X5CECm4jy9422PQr9e
Published rootHash QmUGJR8aHNfQ5wAUKTcZrawcWcz3X5CECm4jy9422PQr9e under ipns key
Thu Jul 14 01:21:10 UTC 2022
Thu Jul 14 06:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Thu Jul 14 06:00:46 UTC 2022
Publishing rootHash QmbWG3hiccqcaWjUpNuTBvoTCHJBeYTyaAMGdoqdY6dArz under ipns key
2022-07-14T08:08:09.758Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmbWG3hiccqcaWjUpNuTBvoTCHJBeYTyaAMGdoqdY6dArz
Published rootHash QmbWG3hiccqcaWjUpNuTBvoTCHJBeYTyaAMGdoqdY6dArz under ipns key
Thu Jul 14 08:08:11 UTC 2022
Thu Jul 14 12:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Thu Jul 14 12:00:52 UTC 2022
Publishing rootHash QmNou1yXp1mJ7rViyzbxzabLe3vesDqPjgBVy4L5mrT6nb under ipns key
2022-07-14T14:17:03.134Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmNou1yXp1mJ7rViyzbxzabLe3vesDqPjgBVy4L5mrT6nb
Published rootHash QmNou1yXp1mJ7rViyzbxzabLe3vesDqPjgBVy4L5mrT6nb under ipns key
Thu Jul 14 14:17:04 UTC 2022
Thu Jul 14 18:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Thu Jul 14 18:00:45 UTC 2022
Publishing rootHash QmZvg25sMyDXpimS4EvhY78jNn4Tjr7cjYdhZL4S4zULmN under ipns key
2022-07-14T20:48:37.973Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmZvg25sMyDXpimS4EvhY78jNn4Tjr7cjYdhZL4S4zULmN
Published rootHash QmZvg25sMyDXpimS4EvhY78jNn4Tjr7cjYdhZL4S4zULmN under ipns key
Thu Jul 14 20:48:39 UTC 2022
Fri Jul 15 00:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Fri Jul 15 00:00:46 UTC 2022
Publishing rootHash QmTQyH1DkdhuEHr4XeyCjQeSv1cF42AEZ9uogqWp1sZLsC under ipns key
2022-07-15T02:03:56.096Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmTQyH1DkdhuEHr4XeyCjQeSv1cF42AEZ9uogqWp1sZLsC
Published rootHash QmTQyH1DkdhuEHr4XeyCjQeSv1cF42AEZ9uogqWp1sZLsC under ipns key
Fri Jul 15 02:03:57 UTC 2022
Fri Jul 15 06:00:01 UTC 2022
Started find in repository for stats
find: '/root/.ipfs/repository/contracts/partial_match/62621/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/5700/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/534/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/57/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1284/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/11111/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1287/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/1313161555/': No such file or directory
find: '/root/.ipfs/repository/contracts/partial_match/103090/': No such file or directory
Finished find in repo for stats
Fri Jul 15 06:00:45 UTC 2022
Publishing rootHash QmYqWuNj4WaGgNsoxo4ff7W8MzKLnag7kmozR6EvJXT4Mo under ipns key
2022-07-15T06:07:11.654Z    DEBUG   cmd/ipfs    ipfs/main.go:140    config path is /root/.ipfs
Published to k51qzi5uqu5dll0ocge71eudqnrgnogmbr37gsgl12uubsinphjoknl6bbi41p: /ipfs/QmYqWuNj4WaGgNsoxo4ff7W8MzKLnag7kmozR6EvJXT4Mo
Published rootHash QmYqWuNj4WaGgNsoxo4ff7W8MzKLnag7kmozR6EvJXT4Mo under ipns key
Fri Jul 15 06:07:15 UTC 2022

One additional thing to note is that the ipfs node is incredibly resource intensive in our case. Our server nearly always run at full capacity with ~350/400% of the CPU and around 8GB of RAM consumed by the ipfs node.

schomatis commented 2 years ago

Thanks for the report @kuzdogan; will take a closer look next week but seems consistent with the original report panicking in our new custom parallel walk.

schomatis commented 2 years ago

Avoiding panic on https://github.com/ipfs/go-unixfs/pull/123 (blocking this on landing it). Other than that I have no idea what to do (or how to investigate this further) other than letting it roll and see if we can get more information through the added log line.

ajnavarro commented 2 years ago

Fixing the problem here: https://github.com/ipfs/go-unixfs/pull/124

schomatis commented 2 years ago

@ajnavarro Could you point me to where did we have the race in our parallel walk?

ajnavarro commented 2 years ago

Any call to these Shard methods (Set, SetLink, Swap, Remove, Take) on different threads can cause a race condition on both cases, reading or writing, and corrupt slice data.

schomatis commented 2 years ago

@ajnavarro Yes, but where are those threads created that manipulate the same shard? By that criteria anything that doesn't have a lock has a race condition, but it depends on how its being used. I need more specifics to understand why is this panic specifically being fixed by https://github.com/ipfs/go-unixfs/pull/124.

ajnavarro commented 2 years ago

@kuzdogan, @schomatis I tried to reproduce the error locally with no luck.

I tried with publish.sh and entrypoint.sh to reproduce the error.

One thing I noticed is the add process takes only 10-15 mins for me instead of 10h (add 13Gb into kubo):

Add finish time below. Took around 10 hours.

Are you using mechanical disks? I try to discard if I'm doing something incorrectly.

BigLep commented 2 years ago

2022-08-04 maintainer conversation: hypothesis is that MFS is related, that it's potentially not following certain concurrency expectations.

Things we can do: fuzzing at the top level (run a hundred time with large sharded directories).

@Jorropo is going to paste in the link for a relevant Discord conversation.

Jorropo commented 2 years ago

https://discord.com/channels/806902334369824788/847893063841349652/1003259317383868426

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1a37eb8]

goroutine 576316133 [running]:
github.com/ipfs/go-unixfs/hamt.(Shard).childLinkType(0xc037cfc690?, 0xc00fc1e8f0?)
        github.com/ipfs/go-unixfs@v0.4.0/hamt/hamt.go:313 +0x18
github.com/ipfs/go-unixfs/hamt.(Shard).walkChildren(0xc037cfc690, 0xc0f335f380)
        github.com/ipfs/go-unixfs@v0.4.0/hamt/hamt.go:420 +0x20c
github.com/ipfs/go-unixfs/hamt.parallelShardWalk.func1()
        github.com/ipfs/go-unixfs@v0.4.0/hamt/hamt.go:487 +0x25a
golang.org/x/sync/errgroup.(Group).Go.func1()
        golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(Group).Go
        golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x8d
ajnavarro commented 2 years ago

Some more context from @aschmahmann on #ipfs-implementers channel:

aschmahmann: I have a suspicion that there were some assumptions here in the past that various functions for go-unixfs were not used concurrently that is not being upheld in go-mfs. I don't know if there are other errors that would've come in a similar form if sharded directories has been enabled by default in the past, or if this error is unique due to adding some concurrency in sharded directory enumeration that came with enabling by default. stebalien any insight on to what the historical concurrency boundaries were supposed to be? It seems like we either:

  1. Need more docs in go-unixfs explaining it's not thread safe, and then add thread-safety into go-mfs
  2. Need to add locking in go-unixfs to enforce thread-safety and then document that functionality of go-unixfs

stebalien: There is no safe way to use these concurrently at the moment.

aschmahmann: So then likely the above panic is due to mfs lack of concurrency controls rather than our introduction of parallelWalk (https://github.com/ipfs/go-unixfs/blob/4571136c15d0678006f09ebde615d0c45c126ff9/hamt/hamt.go#L468) itself being problematic?

BigLep commented 2 years ago

What's the next step here?

Given we've had this open for weeks, do we believe this is a P1?

ajnavarro commented 2 years ago

The following steps should be to do some Fuzz testing on mfs and check if we can reproduce the error.

Fixing attempt here: https://github.com/ipfs/go-unixfs/pull/124, but maybe we want to do the blocking at mfs level.

BigLep commented 2 years ago

Lowering the priority because:

  1. We expect users won't hit this once https://github.com/ipfs/kubo/pull/8927 lands.
  2. We don't have an easy repro case

When we pick this up again, we need to:

  1. Get repro (e.g., using fuzzing as suggested above)
  2. Added locking at the MFS level rather than go-unixfs.
BigLep commented 2 years ago

2022-09-08 conversation: @ajnavarro on 2022-09-09 will:

  1. Get https://github.com/ipfs/kubo/pull/8927 landed
  2. Attempt to reproduce this issue with fuzzing
ajnavarro commented 2 years ago

I have good and bad news.

I was able to reproduce a panic on Shard from MFS here: https://github.com/ipfs/go-mfs/pull/103. The bad news is that is a panic on a different line :disappointed:

ajnavarro commented 2 years ago

Nil pointers are coming when childer is nil. This can only happen on https://github.com/ipfs/go-unixfs/blob/2c23c3ea6fae3ef1b487cfc0c606a4ffc7893676/hamt/hamt.go#L798 , when swapping values that don't exist, so a new value is added.

lidel commented 1 year ago

(iiuc) this was fixed in https://github.com/ipfs/kubo/pull/9402 and will be included in Kubo 0.17.0-rc2