search

NASA-PDS / registry-sweepers

Scripts that run regularly on the registry database, to clean and consolidate information
Apache License 2.0
0 stars 1 forks source link

46 bulk flush tuning #48

Closed alexdunnjpl closed 11 months ago

alexdunnjpl commented 11 months ago

🗒️ Summary

Previously, the _bulk API updates were flushed based on number of updates. This has been replaced with a memory threshold of 20MB instead.

I also renamed the scroll variable because it's misleading to anyone not already familiar with exactly how that works.

⚙️ Test Data and/or Report

Tested at 5MB, 10MB, 15MB, 20MB, 40MB. Negligible performance increase observed above 20MB in current configuration (prod AWS Opensearch, from local dev machine).

♻️ Related Issues

fixes #46

sonatype-lift[bot] commented 11 months ago

Sonatype Lift is retiring

Sonatype Lift will be retiring on Sep 12, 2023, with its analysis stopping on Aug 12, 2023. We understand that this news may come as a disappointment, and Sonatype is committed to helping you transition off it seamlessly. If you’d like to retain your data, please export your issues from the web console. We are extremely grateful and thank you for your support over the years.

📖 Read about the impacts and timeline