Can't remove complex object during test

The "Object can be deleted from any node using s3gate wallet with bearer token for complex object-single replica for all nodes placement rule" test fails. See fail_tests.zip (unzip and run allure serve validate_failed) Logs: eb8e4736378eebb5.zip Deleting an object always occurs from the same node (from s01.neofs.devenv).

Expected Behavior

Complex object must always be deleted

Current Behavior

Tests consistently fail to delete one of the test complex objects.

Failed: Command: neofs-cli --config ./TemporaryDir/password-s3.yml object delete --rpc-endpoint 's01.neofs.devenv:8080' --wallet '../neofs-dev-env/services/s3_gate/wallet.json' --cid '41tVWBvQVTLGQPHBmXySHsJkcc6X17i39bMuJe9wYhAJ' --oid '3FTTkuUyTMUWE1js4P9e1vMHgLfGMiUzSw3NuTwyWvz4' --bearer '/home/varg/work/neofs-testcases/TemporaryDir/0205e85a-8ac5-4bfa-9e9a-5c027f4cbeca'
Error:
return code: 1
output: rpc error: remove object via client: status: code = 2049 message = object not found

The localhost-s01-log.txt log shows that there are no tombstone structure messages for the 3FTTkuUyTMUWE1js4P9e1vMHgLfGMiUzSw3NuTwyWvz4 object.

Possible Solution

Steps to Reproduce (for bugs)

pytest --alluredir validate_failed pytest_tests/testsuites/object/test_object_api_bearer.py -k test_delete_object_with_s3_wallet_bearer

Context

I'm trying to get consistently green tests.

Regression

Yes

Your Environment

# Basenet settings
LOCAL_DOMAIN=neofs.devenv
IPV4_PREFIX=192.168.100
CA_CERTS_TRUSTED_STORE=/etc/ssl/certs

# Bastion image
BASTION_VERSION=10
BASTION_IMAGE=debian

# NeoGo privnet
#CHAIN_PATH="/path/to/devenv.dump.gz"
CHAIN_URL="https://github.com/nspcc-dev/neofs-contract/releases/download/v0.16.0/devenv_mainchain_notary_disabled.gz"
NEOGO_VERSION=0.101.0
NEOGO_IMAGE=nspccdev/neo-go

# NeoFS InnerRing nodes
IR_VERSION=0.35.0
IR_IMAGE=nspccdev/neofs-ir

# NeoFS Storage nodes
NODE_VERSION=0.35.0
NODE_IMAGE=nspccdev/neofs-storage

# NATS Server
NATS_VERSION=2.7.2
NATS_IMAGE=nats

# HTTP Gate
HTTP_GW_VERSION=0.26.0
HTTP_GW_IMAGE=nspccdev/neofs-http-gw

# REST Gate
REST_GW_VERSION=0.5.0 
REST_GW_IMAGE=nspccdev/neofs-rest-gw

# S3 Gate
S3_GW_VERSION=0.26.1
S3_GW_IMAGE=nspccdev/neofs-s3-gw

# Coredns
#COREDNS_VERSION=v019
COREDNS_VERSION=v016
COREDNS_IMAGE=nspccdev/coredns

# NeoFS LOCODE database
LOCODE_DB_URL=https://github.com/nspcc-dev/neofs-locode-db/releases/download/v0.3.0/locode_db.gz
#LOCODE_DB_PATH=/path/to/locode_db

# NeoFS CLI binary
NEOFS_CLI_URL=https://github.com/nspcc-dev/neofs-node/releases/download/v0.35.0/neofs-cli-amd64.tar.gz
#NEOFS_CLI_PATH=/path/to/neofs-cli-binary

# NeoFS ADM tool binary
NEOFS_ADM_VERSION=v0.35.0
NEOFS_ADM_URL=https://github.com/nspcc-dev/neofs-node/releases/download/${NEOFS_ADM_VERSION}/neofs-adm-amd64.tar.gz
#NEOFS_ADM_PATH=/path/to/neofs-adm-binary

# Compiled NeoFS Smart Contracts
NEOFS_CONTRACTS_VERSION=v0.16.0
NEOFS_CONTRACTS_URL=https://github.com/nspcc-dev/neofs-contract/releases/download/${NEOFS_CONTRACTS_VERSION}/neofs-contract-${NEOFS_CONTRACTS_VERSION}.tar.gz
#NEOFS_CONTRACTS_PATH=/path/to/unpacked/neofs-contracts-dir

Linux burzum 5.19.0-31-generic #32-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 20 15:20:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

cthulhu-rider / neofs-node