Dewberry / stormcloud

Creates datasets for hydrologic modeling in the cloud.
MIT License
3 stars 1 forks source link

Feature/ms update #35

Closed nick-j-roberts closed 8 months ago

nick-j-roberts commented 9 months ago
nick-j-roberts commented 9 months ago

Things to consider:

nick-j-roberts commented 9 months ago

I just changed the tropical storm data for the meilisearch document as having a list of objects associated with each document would make it difficult to use in filters, which is what I would think the purpose of this data is. It is easy to revert to a list of objects, but would need clarification on how this would be interacted with on meilisearch

nick-j-roberts commented 9 months ago

I have renamed the plugin from ms_update to doc_rank to reflect that the utility of the plugin is not solely geared towards meilisearch consumption

nick-j-roberts commented 9 months ago

As a side note, the docs from the Trinity SST runs are missing the watershed name. This causes issues with path name creation in the documents, but I can add an override to pull the watershed name from provided plugin parameters instead of the document to take care of this.

Example of missing data from s3 docs for Trinity (s3://tempest/watersheds/trinity/trinity-transpo-area-v01/72h/docs/19790201.json):

"metadata": {
    ...
    "watershed_name": "",
    ...
  }
nick-j-roberts commented 9 months ago

As a side note, the docs from the Trinity SST runs are missing the watershed name. This causes issues with path name creation in the documents, but I can add an override to pull the watershed name from provided plugin parameters instead of the document to take care of this.

Example of missing data from s3 docs for Trinity (s3://tempest/watersheds/trinity/trinity-transpo-area-v01/72h/docs/19790201.json):

"metadata": {
    ...
    "watershed_name": "",
    ...
  }

The overwrite thing is done, but may not be necessary for Trinity as @ahdietrich is rerunning

nick-j-roberts commented 9 months ago

Format of ranked document was updated to minimize information being repeated from s3 docs and meilisearch upload function was revised to consume and construct meilisearch formatted documents from these ranked documents

nick-j-roberts commented 9 months ago

Closes #36