search

nspcc-dev / neofs-node

NeoFS is a decentralized distributed object storage integrated with the Neo blockchain
https://fs.neo.org
GNU General Public License v3.0
32 stars 38 forks source link

Failure to delete an object #2971

Open roman-khimov opened 4 weeks ago

roman-khimov commented 4 weeks ago

Expected Behavior

No errors.

Current Behavior

Oct 18 09:22:07 metis2 neofs-node[2993338]: 2024-10-18T09:22:07.680Z        warn        metabase/delete.go:112        failed to delete object        {"shard_id": "YMGgmv7yj3y5s7K3Xr94kN", "addr": "7eyrRq3wALGwAEwuwSKfH3DcMFVU7jLPhqWmAnmyM7wm/3rvsak6c3eCfM6J33obcHU5G1uzZUaBqnvXjJKBD4FN2", "error": "proto: cannot parse invalid wire-format data"}

Possible Solution

Let objects die in any event, even if they're malformed (which is strange, but that's what it is).

Steps to Reproduce (for bugs)

Unknown

Context

Mainnet.

Regression

Unknown.

Your Environment

carpawell commented 3 weeks ago 
2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/DsX2krdGGPE1pgR1PPrUXF1tEzPdJQHMZqQA8CfqjpG6
2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/3yGt6PwGjqFq94DsgJfuKC88Bo5ZRQeVLQswXz2MASJN

two objects cannot be dropped from metis2 but from a single container. WC has been cleaned

carpawell commented 2 weeks ago

Some more problematic objects.

"2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/2hUsdvs7tdxnXK9hLMd1BYgcVrSezS56XtmoNW1KN75k",
"2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/RDvnKLMNUVHZUZHka4qH5a7AkZGdPH2cFzN6tyy8zXU",
"2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/Wg5R8vaNcYuGk9GUtkxgFGymoG8G8iJnwXTamqtZZos",
"2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/yiiBB9Dm6gDwav3aNu7fXZXL3rkr42hZs5XZ2G19Ujh",
carpawell commented 2 weeks ago

So I have cleaned manually bucket with object headers for this (2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2) container and also cleared WC that could not update metabase with storage ID for objects of containers that do not exists. Now node should not do strange useless things with disks and logs are cleared (all the nodes only clear their metabases since they can now continue their GC cycle).

roman-khimov commented 1 week ago

In fact, it gets contents from the metabase, so #2978 is also relevant here as the one leading to #2988.

carpawell commented 1 day ago

@roman-khimov, also pretty sure it is https://github.com/nspcc-dev/neofs-node/issues/2978#issuecomment-2476998637.