[BUG] Cannot register snapshot repository on search node because api request is executed on cluster manager

[drewmiranda-gl]

Describe the bug

Consider the following:

An opensearch cluster with 2 nodes:

1. pve-opsrch1 : Cluster Manager, Data (this is the only active node that ingests data and serves search requests)
2. pve-opsrch3 : Search (only for searchable snapshots)

When attempting to register a snapshot repository on the second node, which is ONLY the search role, OpenSearch throws an error saying it cannot register the repository because path.repo is empty, even though it is indeed configured. The OpenSearch log indicates opensearch is executing this rest api request against the cluster manager, despite the cluster manager not having a search role.

[2024-05-14T14:39:22,030][WARN ][r.suppressed             ] [pve-opsrch3] path: /_snapshot/graylog-warm-tier, params: {pretty=, repository=graylog-warm-tier}
org.opensearch.transport.RemoteTransportException: [pve-opsrch1][192.168.0.161:9300][cluster:admin/repository/put]

Related component

Search:Searchable Snapshots

To Reproduce

Recreate an opensearch clsuter as described above.

Attempt to register a snapshot repo by sending a rest api req to the node with the search role:

curl -XPUT "http://localhost:9200/_snapshot/graylog-warm-tier?pretty" -H 'Content-Type: application/json' -d' { "type" : "fs", "settings" : { "chunk_size" : null, "location" : "/opt/opensearch/snapshots", "max_restore_bytes_per_sec" : "40m", "readonly" : "false", "compress" : "false", "max_snapshot_bytes_per_sec" : "40m" } }'

Observe the error

[2024-05-14T14:39:22,030][WARN ][r.suppressed             ] [pve-opsrch3] path: /_snapshot/graylog-warm-tier, params: {pretty=, repository=graylog-warm-tier}
org.opensearch.transport.RemoteTransportException: [pve-opsrch1][192.168.0.161:9300][cluster:admin/repository/put]
Caused by: org.opensearch.repositories.RepositoryException: [graylog-warm-tier] failed to create repository
[...]
Caused by: org.opensearch.repositories.RepositoryException: [graylog-warm-tier] location [/opt/opensearch/snapshots] doesn't match any of the locations specified by path.repo because this setting is empty

Expected behavior

Snapshot repository is registered on the node the api req is executed against.

Additional Details

Plugins Default, out of box

Host/Environment (please complete the following information):

• OS: Ubuntu Server 22.04 LTS
• Version: OpenSearch 2.12

[drewmiranda-gl]

This may be more so a documentation issue, still testing :(

[drewmiranda-gl]

Closing for now as this seems to be working as expected.