[BUG] Create repository is failing in Shared File system in multi node setup and the same is working if we config as single node

[ssrslm]

Describe the bug Create repository is failing in multi node setup and the same is working if we configure as single node setup

To Reproduce Steps to reproduce the behavior:

1. Use below docker-compose.yml to config 2 node setup with dashboard
2. Navigate OpenSearch Dashboard with default admin credential
3. Create repository for Snapshot feature, it;s failing

docker-compose.yml

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - path.repo=/usr/share/opensearch/snapshots
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - opensearch-snapshot-data:/usr/share/opensearch/snapshots
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - path.repo=/usr/share/opensearch/snapshots
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - opensearch-snapshot-data:/usr/share/opensearch/snapshots
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:
  opensearch-snapshot-data:

networks:
  opensearch-net:

Expected behavior Register repository should be functional.

Plugins All default plugins are enabled

Screenshots

Error {“error”:{“root_cause”:[{“type”:“repository_exception”,“reason”:“[testrepo] cannot create blob store”}],“type”:“repository_verification_exception”,“reason”:“[testrepo] path is not accessible on cluster-manager node”,“caused_by”:{“type”:“repository_exception”,“reason”:“[testrepo] cannot create blob store”,“caused_by”:{“type”:“access_denied_exception”,“reason”:“/usr/share/opensearch/snapshots/test_repo_location”}}},“status”:500}

Host/Environment (please complete the following information):

• OS: Windows 10, Docker Desktop
• Version 2.5.x (latest)

Additional context Need help to resolve through local docker setup or how to resolve it through OpenSearch Operator

[anasalkouz]

@ssrslm, could you try to fix the access_denied_exception by making sure the application has the right access to the directory?

[andrross]

@ssrslm You need to ensure that the opensearch user has permission to access /usr/share/opensearch/snapshots. If it does not, then you can modify permissions using chown.