search

NASA-PDS / harvest

Standalone Harvest client application providing the functionality for capturing and indexing product metadata into the PDS Registry system (https://github.com/nasa-pds/registry).
https://nasa-pds.github.io/registry
Other
4 stars 3 forks source link

Access forbidden during nominal pipeline execution of harvest on Mars2020 archive #124

Closed jordanpadams closed 1 year ago

jordanpadams commented 1 year ago

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

I’ve been running Harvest on our Mars 2020 holdings. It was going swimmingly, but today I saw that it entered a loop. It would log that it’s processing the same ~50 products, and then an error would spit out after a batch saying the following:

Message = index [registry] blocked by: [FORBIDDEN/8/index write (api)]

🕵️ Expected behavior

I expected to harvest the data, as usual

📜 To Reproduce

See attached log.

logs.txt

🖥 Environment Info

Linux environment

📚 Version of Software Used

v3.7.6

🩺 Test Data / Additional context

Should be executed against IMG's Registry

🦄 Related requirements

No response

⚙️ Engineering Details

No response

alexdunnjpl commented 1 year ago

@jordanpadams this is almost certainly the result of usage of the ES disk allocation reaching a cutoff threshold.

When exactly were these errors encountered, and for which node (i.e. which OpenSearch domain)?

Nvm, found it (img-prod)

image

alexdunnjpl commented 1 year ago

Closing as not-a-bug, further discussion in #125