Replicator can't fetch local object

[roman-khimov]

Expected Behavior

No errors.

Current Behavior

Apr 28 08:34:14 titan4 neofs-node[709]: 2024-04-28T08:34:14.639Z        error        replicator/process.go:37        could not get object from local storage        {"component": "Object Replicator", "object": "7dppsTsDszYwN5P9yH7G7ubEbrxRAAVC27SZ356QRkvY/4tvV8gaSiCvT5CgVr35vV4EeFz9WDtgqyak1tFRsQBLH", "error": "status: code = 2049 message = object not found"}
Apr 28 08:37:47 titan4 neofs-node[709]: 2024-04-28T08:37:47.609Z        error        replicator/process.go:37        could not get object from local storage        {"component": "Object Replicator", "object": "EgDAUwMadH55dZznrsentnHJFfqnE2gPUAW7zDmDvYuE/2C2oEuAzNLA1zH8u1DKTR5494vQtPgmJ7Wysuih3Eqe8", "error": "status: code = 2049 message = object not found"}
Apr 28 08:40:32 titan4 neofs-node[709]: 2024-04-28T08:40:32.906Z        error        replicator/process.go:37        could not get object from local storage        {"component": "Object Replicator", "object": "D6AJH9UwLiEG2hh8JHaCt798WePzCYDob7TYEWfCE11M/3aFs6YJBXnGcqPXL3AgWuLYJvUGp1FPF8p4iBdHn6uTj", "error": "status: code = 2049 message = object not found"}
Apr 28 08:40:40 titan4 neofs-node[709]: 2024-04-28T08:40:40.373Z        error        replicator/process.go:37        could not get object from local storage        {"component": "Object Replicator", "object": "DivTTmXeugVPyV7hv5u7RN9Y2tMvqnnyD6mrmoSmAT1g/AQtrN3dgCTfNkMAgTsfYg8ueBhnoFXn7o68HQwWxDaY3", "error": "status: code = 2049 message = object not found"}

Possible Solution

Unknown. Metabase was resynchronized some time ago, so it's unexpected.

Context

Testnet.

Your Environment

• Version used: 0.41.1

[roman-khimov]

Expected Behavior

No errors.

Current Behavior

Jun 21 06:16:36 metis1 neofs-node[2831]: 2024-06-21T06:16:36.455Z        error        replicator/process.go:37        could not get object from local storage        {"component": "Object Replicator", "object": "CeeroywT8ppGE4HGjhpzocJkdb2yu3wD5qCGFTjkw1Cc/Ec7b5vFH9a5MHjTMT5epfcKBUjPkkS7sQPaiuMJ5FD2H", "error": "status: code = 2049 message = object not found"}
Jun 21 06:22:28 metis1 neofs-node[2831]: 2024-06-21T06:22:28.658Z        error        replicator/process.go:37        could not get object from local storage        {"component": "Object Replicator", "object": "CeeroywT8ppGE4HGjhpzocJkdb2yu3wD5qCGFTjkw1Cc/5ZfQQf2pH9RBm4CNQDFRQSf27cUeeTTMrTnAnG2omn6D", "error": "status: code = 2049 message = object not found"}

Possible Solution

Fix the damn thing.

Seems like metabase and real storage got out of sync.

Steps to Reproduce (for bugs)

Unknown.

Context

Happens on mainnet.

Regression

Doesn't look like.

Your Environment

• Version used: 0.42.1
• Server setup and configuration: mainnet
• Operating System and version (uname -a): Debian stable

[carpawell]

About mainnet object only

Such objects look like garbage that is left in metabases by a mistake or a bug. Unfortunately, we neither have backups for the interested period, nor logs so it is impossible now to understand when objects become a problem. The main thing about them is they both are placed in metis1's metabase only and have X/Y storage ID (blobovnicza tree). I have checked the code and it should not be this way for the objects that are resynced after we migrate them from blobovniczas to a peapod. But also, such a record should not be left after metabase resynchronization. @roman-khimov, was it done after migration at least once? Also see: https://github.com/nspcc-dev/neofs-node/issues/2887, https://github.com/nspcc-dev/neofs-node/issues/2886

P.S.: I have also used the tool from the https://github.com/nspcc-dev/neofs-node/pull/2884 in the testnet storages and found so many objects with strange blobovnicza IDs that are not found in peapod too. So maybe we need to resync all our storages to reset them.

[roman-khimov]

control drop-objects has fixed the problem.