search

nspcc-dev / neofs-node

NeoFS is a decentralized distributed object storage integrated with the Neo blockchain
https://fs.neo.org
GNU General Public License v3.0
32 stars 38 forks source link

object hash command fails with "context deadline exceeded" error #1943

Open vkarak1 opened 2 years ago

vkarak1 commented 2 years ago

I faced issue when node1 was attemted to start but due to issue with ports couldn't:

Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:76        initializing pprof service...
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:78        pprof service has been successfully initialized
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:76        initializing prometheus service...
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:78        prometheus service has been successfully initia>
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:76        initializing tree service...
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:78        tree service has been successfully initialized
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:76        initializing control service...
Oct 20 09:29:11 az neofs-node[1051]: 2022/10/20 09:29:11 listen tcp 127.0.0.1:8091: bind: address already in use
Oct 20 09:29:11 az systemd[1]: neofs-storage.service: Main process exited, code=exited, status=1/FAILURE
Oct 20 09:29:11 az systemd[1]: neofs-storage.service: Failed with result 'exit-code'.

And I tried to issue object hash command to node2 and got the following error:

root@buky:/etc/neofs/storage# neofs-cli object hash -r node2.neofs:8080 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSjrPsqv --oid FqWtXTxQx83bqVNr54nR1AZPh3Mew                  mEmH1TJYbv8eCWh --range 0:11 -w wallet.json
Enter password >
rpc error: read payload hashes via client: write request: context deadline exceeded

When I issued "systemctl restart neofs-storage.service" to node2 the problem had been sorted out.

Expected Behavior

Node2 should process command while node1 was trying to start.

Current Behavior

Object hash fails on node2 while node1 is unavailable.

Possible Solution

run systemctl restart neofs-storage.service

Steps to Reproduce (for bugs)

  1. node1 create container: neofs-cli --rpc-endpoint node1.neofs:8080 --wallet wallet.json container create --name test --policy "REP 2 IN X CBF 1 SELECT 2 FROM * AS X" --basic-acl public-read-write --await
  2. node1: allocate object1: dd if=/dev/urandom of=object1 bs=1M count=1
  3. node1: put object1:
    neofs-cli --rpc-endpoint node1.neofs:8080 -w wallet.json object put --file object1 --cid 5oFBZsSdAvRKUsL2FknAnNqjyFj6UjvXW8QfWQeEtG3z --no-progress
  4. node1: shards list: neofs-cli --endpoint localhost:8091 -w /etc/neofs/storage/wallet.json control shards list | grep Shard && curl -s localhost:6672 | rg neofs_node_object_counter | sed 1,2d
  5. I was working on validating the following issue , and I performed to node1: systemctl restart neofs-storage.service to double check that shards list and neofs_node_object_counter display the same shard ids
  6. And got the problem that node1 was trying to start: rpc error: read payload hashes via client: write request: context deadline exceeded
  7. node2: created container and put 10 bytes object: 
    neofs-cli --rpc-endpoint node2.neofs:8080 --wallet wallet.json container create --name rep1 --policy "REP 1 " --basic-acl public                  -read-write   --await
neofs-cli --rpc-endpoint node2.neofs:8080 -w wallet.json object put --file object1 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSj
  8. node2: object hash: 
    root@buky:/etc/neofs/storage# neofs-cli object hash -r node2.neofs:8080 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSjrPsqv --oid FqWtXTxQx83bqVNr54nR1AZPh3Mew                  mEmH1TJYbv8eCWh --range 0:11 -w wallet.json
Enter password >
rpc error: read payload hashes via client: write request: context deadline exceeded

Your Environment

Linux buky 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux

NeoFS Storage node Version: v0.33.0-6-g4c63be66 GoVersion: go1.18.4

Attached logs

vkarak1 commented 2 years ago

Also would like to add that when I issued "systemctl restart neofs-storage.service" to node2 it took about 1.5 min. Also before the error message in step#8 displayed the command was running about 15 seconds.