search

CDLUC3 / dmsp_aws_prototype

Sceptre CloudFormation templates for DMPHub v2
MIT License
1 stars 0 forks source link

Investigate use of elastic search #33

Closed briri closed 4 months ago

briri commented 1 year ago

Investigate the use of Elastic Search for the new React based dashboards.

These dashboards will need to pull and display DMP metadata from the DMPHub's DynamoDB as well as the DMPTool's RDS DB. Elastic Search would be a performant way to query both systems (and also prevent DynamoDB table scans which can be costly)

The Elastic Search results should include a URL to view/edit the DMP.

briri commented 1 year ago

Interesting article on open search indexing https://opensearch.org/blog/fine-dining-for-indices/

This course may be useful as we prepare to work with opensearch. https://www.linkedin.com/learning-login/share?account=76816210&forceAccount=false&red[…]are_ent_url%26shareId%3Dr9P%252BADJBTDC98gn4O8DZYQ%253D%253D

briri commented 10 months ago

It seems like DynamoDB Streams with a Lambda that handles the OpenSearch indexing would be the best approach for this. As data changes, the OpenSearch index gets updated.

See this article for reference: https://aws.plainenglish.io/using-aws-opensearch-with-dynamodb-and-lambda-fd036add8c88

briri commented 6 months ago

Started work on this and was able to create an OpenSearch Cluster to host the DMPTool's search functionality. See the opensearch branch but could not figure out how to get the IAM/Cognito roles to work.

I tried connecting the Lambda directly (without user auth) and OpenSearch threw errors. So I created a Cognito user (verified that the user could login and access the OpenSearch Dashboard) and assumed their role within the Lambda but OpenSearch started throwing a different auth error

briri commented 4 months ago

Got an OpenSearch cluster up and running. Updated and tested that the Lambda function that gets triggered by the DynamoStream will update the OpenSearch index as a DMP ID is created/updated/deleted.

Added a helper script to the dmpsp_aws_prototype repo that can be run to rebuild the index based on the current contents of the DynamoTable.