Format of records on S3 and Client-side Routes and How to get Metadata for a Search!

numo-labs / aws-lambda-helper

:lollipop: Collection of helper methods for lambda

GNU General Public License v3.0

22 stars 2 forks source link

Format of records on S3 and Client-side Routes and How to get Metadata for a Search! #54

Open nelsonic opened 8 years ago

nelsonic commented 8 years ago

This needs to be consistent across all our modules.

For each search that is performed on the client we need to send the following fields through SNS:

userId - unique userId ("fingerprint") generated on the client
connectionId - WebSocket connection Id
bucketId - id of the precise search that was executed (returned by Search Request Handler)
itemId - the id of the result tile (article/package) returned for a given search

we will store all tiles on S3 as .json files in the format: /userId/connectionId/bucketId/itemId.json and include a schema.json and meta for each provider inside each bucket.

Tile Provider

Should store a ne_tiles.json file with summary/stats for the given search.

Package Provider

The package provider should store a ne_packages.json

Fields for a meta file

items - an array of the ids of the records.
timems - the time it took from the start of the lambda execution till end
errors - an array of the error objects if any

Kumjami commented 8 years ago

So the front-end URL links should be like this: www.isearch.com/#/hotel/:userId/:bucketId/:itemid www.isearch.com/#/article/:userId/:bucketId/:itemid www.isearch.com/#/search/:userId/:bucketId/

nelsonic commented 8 years ago

@Kumjami that looks good. Jimmy just had the idea of sending the url back to the client inside the items Array so that it does not need to be constructed dynamically. so each item would contain:

{ "url": "/:userId/:connectionId/:bucketId/:itemId" }

This will make it easier to shorten the URLs once we have a URL shortening service hooked up.

Kumjami commented 8 years ago

@nelsonic that would be great

nelsonic commented 8 years ago

@jruts I have a related question. what if someone wants to share the url for the particular search and all the results is that a use-case we are considering right now?

nelsonic commented 8 years ago

@jruts this afternoon we discussed the possibility of storing the records on S3 in the format: ci/bucketId/itemId.json so that the urls could be less cumbersome. However you also suggested that we would store a reference to this file in the more verbose format: ci/userId/connectionId/bucketId/itemId.json and simply link to the other file with the helper doing the lookup transparently.

Can you please confirm my understanding of this so I can implement it? thanks! 👍

jruts commented 8 years ago

Your understanding is correct.

nelsonic commented 8 years ago

So, for now, we are storing the Packages / Articles in S3 using the format: ci/bucketId/itemId.json e.g: https://numo-search-results.s3.amazonaws.com/ci/ffe95600-27d8-11e6-9b2f-133803a346f1/PMIPAGA.json

We need to update the aws-lambda-helper to also store the userId and connectionId so that we can group the buckets by user (device) and by session... this is not essential for shipping the (i.e not a "Priority for Release") so we can leave it for when we need it for "Tracy".