Closed nick-j-roberts closed 8 months ago
Things to consider:
I just changed the tropical storm data for the meilisearch document as having a list of objects associated with each document would make it difficult to use in filters, which is what I would think the purpose of this data is. It is easy to revert to a list of objects, but would need clarification on how this would be interacted with on meilisearch
I have renamed the plugin from ms_update to doc_rank to reflect that the utility of the plugin is not solely geared towards meilisearch consumption
As a side note, the docs from the Trinity SST runs are missing the watershed name. This causes issues with path name creation in the documents, but I can add an override to pull the watershed name from provided plugin parameters instead of the document to take care of this.
Example of missing data from s3 docs for Trinity (s3://tempest/watersheds/trinity/trinity-transpo-area-v01/72h/docs/19790201.json):
"metadata": {
...
"watershed_name": "",
...
}
As a side note, the docs from the Trinity SST runs are missing the watershed name. This causes issues with path name creation in the documents, but I can add an override to pull the watershed name from provided plugin parameters instead of the document to take care of this.
Example of missing data from s3 docs for Trinity (s3://tempest/watersheds/trinity/trinity-transpo-area-v01/72h/docs/19790201.json):
"metadata": { ... "watershed_name": "", ... }
The overwrite thing is done, but may not be necessary for Trinity as @ahdietrich is rerunning
Format of ranked document was updated to minimize information being repeated from s3 docs and meilisearch upload function was revised to consume and construct meilisearch formatted documents from these ranked documents
Closes #36