search

NASA-PDS / registry

PDS Registry provides service and software application necessary for tracking, searching, auditing, locating, and maintaining artifacts within the system. These artifacts can range from data files and label files, schemas, dictionary definitions for objects and elements, services, etc.
https://nasa-pds.github.io/registry
Apache License 2.0
3 stars 2 forks source link

Reindex MCP Prod ATM registry index with corrected mappings #339

Open alexdunnjpl opened 1 month ago

alexdunnjpl commented 1 month ago

💡 Description

The reindexing sweeper picked up some properties which aren't described by the DD and aren't desirable to fix (or at least, aren't desirable to map as keyword).

These will need to be removed from the mapping, and a new index created with that mapping, migrated from the existing index.

@sjoshi-jpl to discuss with @alexdunnjpl upon his return

⚔️ Parent Epic / Related Tickets

No response

alexdunnjpl commented 3 weeks ago

@jordanpadams per @sjoshi-jpl , the logstash instance was nuked in the past 24hrs, and redeploying/reconfiguring will be a significant (1wk) amount of work.

Given that fact, it seems like the best option available to us may be to manually (i.e. python script) paginate/insert into a new index, run whatever confirmation checks we deem necessary, then delete/rename.

My only concern is ensuring consistency/completeness of the copied data (as was a factor in the pubcloud->mcp migration)

Any thoughts on that?

tloubrieu-jpl commented 3 weeks ago

We will use logstash for that work.

alexdunnjpl commented 3 weeks ago

@alexdunnjpl and @sjoshi-jpl to collaborate on this task

tloubrieu-jpl commented 2 weeks ago

@sjoshi-jpl is preparing the re-indexation with logstash.

alexdunnjpl commented 5 days ago

Status: logstash does not support AOSS as an input so logstash migration not possible. @alexdunnjpl to develop reusable tool for this purpose