opensearch-project / opensearch-py

Python Client for OpenSearch
https://opensearch.org/docs/latest/clients/python/
Apache License 2.0
359 stars 179 forks source link

[CCI][GUIDE] Added Guide for Snapshot API(snapshot.md) #486

Closed roma2023 closed 3 months ago

roma2023 commented 1 year ago

Description

Added Snapshot Actions Guide (guides/snapshot.md) with code examples

Issues Resolved

issue #406 resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov[bot] commented 1 year ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 72.12%. Comparing base (7b0b58d) to head (dc0cf68). Report is 75 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #486 +/- ## ======================================= Coverage 72.12% 72.12% ======================================= Files 89 89 Lines 7939 7939 ======================================= Hits 5726 5726 Misses 2213 2213 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

dblock commented 1 year ago

@roma2023 Want to finish this?

roma2023 commented 1 year ago

Sure, but I got stuck with this issue: How can I run the docker with the OpenSearch server write permission? Here is more about the details: https://github.com/opensearch-project/opensearch-py/pull/486#discussion_r1328003558. Would appreciate your guidance)

dblock commented 12 months ago

@roma2023 Do you need more help? Let's finish this one?

roma2023 commented 12 months ago

I am trying to use shared file system as a snapshot in samples/snapshot/snapshot_sample.py I followed the steps specified here (https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/#shared-file-system) and added path.repo: ["/mnt/snapshots"] to opensearch.yml , then when running the code, I get the following error:

$ poetry run python snapshot/snapshot_sample.py
Traceback (most recent call last):
  File "C:\Users\HP\Desktop\AmazonOpenSearchService\Opensearch-py-ml\opensearch-py\samples\snapshot\snapshot_sample.py", line 39, in <module>
    response = client.snapshot.create_repository(repository = repository_name, body = repo_body)
  File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\client\utils.py", line 179, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\client\snapshot.py", line 185, in create_repository
    return self.transport.perform_request(
  File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\transport.py", line 409, in perform_request
    raise e
  File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\transport.py", line 370, in perform_request
    status, headers_response, data = connection.perform_request(
  File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\connection\http_urllib3.py", line 266, in perform_request
    self._raise_error(
  File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\connection\base.py", line 301, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.TransportError: TransportError(500, 'repository_verification_exception', 'failed to create blob container')

It seems to me that the OpenSearch process does not have read and write permissions to /mnt/snapshots directory.

I do not know how to solve this issue. If someone could pick this up from here that would be amazing.

dblock commented 12 months ago

Yes, you will need to chmod it inside the container. Here's what I did.

  1. Created a new Dockerfile with the following.
    
    FROM opensearchproject/opensearch:2.11.0

ARG OPENSEARCH_HOME=/usr/share/opensearch ARG UID=1000 ARG GID=1000

RUN echo 'path.repo: ["/usr/share/opensearch/backups"]' >> $OPENSEARCH_HOME/config/opensearch.yml RUN mkdir -p $OPENSEARCH_HOME/backups RUN chown -Rv $UID:$GID $OPENSEARCH_HOME/backups

2. `docker buildx build -t opensearch-snapshot-restore .`
3. `docker run -d -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearch-snapshot-restore`
4.  Change `temp_repo_location = "/usr/share/opensearch/backups"` in your code.

Code ran:

```bash
$ poetry run python snapshot/snapshot_sample.py 
/usr/share/opensearch/backups
{'acknowledged': True}
{'accepted': True}
{'snapshots': [{'snapshot': 'my_snapshot', 'uuid': 'Q4_hA4jATWmsb6ex38qRsQ', 'version_id': 136327827, 'version': '2.11.0', 'remote_store_index_shallow_copy': False, 'indices': ['test-snapshot'], 'data_streams': [], 'include_global_state': True, 'state': 'IN_PROGRESS', 'start_time': '2023-11-21T20:28:44.288Z', 'start_time_in_millis': 1700598524288, 'end_time': '1970-01-01T00:00:00.000Z', 'end_time_in_millis': 0, 'duration_in_millis': 0, 'failures': [], 'shards': {'total': 0, 'failed': 0, 'successful': 0}}]}

LMK if this doesn't unblock you.

dblock commented 11 months ago

@roma2023 want to finish this?

roma2023 commented 11 months ago

@dblock Yes, I will finish this soon.

roma2023 commented 11 months ago

Apologize for the delay

saimedhi commented 8 months ago

Hello @roma2023, could you please take care of the failing DCO and make the requested changes? Thank you for your patience.