[x] Decently tested function to filter dreams by many criteria (i.e. ensure we are able to write a good amount of model specs.) -- using full text search (notes in #5)
[x] Endpoint for the above (with querystring captures.)
[x] Some numbers/idea of how expensive queries may get (insert a few thousand rows locally/externally, see how it does.) -- looks like we need to keep an eye on dreamed_at -- added a simple btree index, but still seeing sequential scans in my small data set.
[x] Stats endpoints (will have to be raw sql, but make sure to use sampling to keep it "constant")
[x] Use migrations (see notes in #10) for fulltext and JSON indexes, plus others (maybe generate a dataset to explore postgres query plans a bit.)
[x] (optional) if the stats endpoints are expensive, look into caching.
Make a decision on pagination, if the use case is linear, we can dispense with offset and maybe take advantage of the before and after ranges that already exist -- to use as a "keyset" pagination: https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/
Ensure we have not only a default limit, but also a max limit: would suck for abuse to happen! Perhaps the easiest thing is to return a lastSeenId and simply say: "if you include lastSeenId" in your request, I'll give you the N records after that. This also means the response should be something like {lastSeen: XXX, dreams: [a,b,c]} (and return an empty list with a null last seen when reaching the end.) If we want to get mad fancy, there's always this library: https://github.com/chordify/haskell-servant-pagination -- though I'm not sure how much of response headers an ionic app can comfortably parse? Maybe we can keep the response simple and return the Next-Range header?
- Get dreams for everyone
- Always non-private dreams
- Include optional ability to pass dream property (lucid, nightmare, recurring, emotion, keyword, etc) OR user property (country, zodiac sign, gender you identify as), OR date range OR most recent X dreams
- Use cases: setting the color of the sphere, viewing most recent dreams, getting the most recent dreams dream property
- Get statistics we’re pulling and usually calculating on the front end… (Luis: maybe these should be cached for a while?)
- Most common words / topics - day / week / month / year
- Lucid percent
- Nightmare percent
- Total dreams
- Most common emotion
- Most common emotions - day / week / month / year
- Search public dreams
(See notes for querying in #6 )
dreamed_at-- added a simple btree index, but still seeing sequential scans in my small data set.Notes on indices:
Notes on pagination:
Rangeheader: https://devcenter.heroku.com/articles/platform-api-reference#rangesbeforeandafterranges that already exist -- to use as a "keyset" pagination: https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/limit, but also a max limit: would suck for abuse to happen! Perhaps the easiest thing is to return alastSeenIdand simply say: "if you includelastSeenId" in your request, I'll give you the N records after that. This also means the response should be something like{lastSeen: XXX, dreams: [a,b,c]}(and return an empty list with anulllast seen when reaching the end.) If we want to get mad fancy, there's always this library: https://github.com/chordify/haskell-servant-pagination -- though I'm not sure how much of response headers an ionic app can comfortably parse? Maybe we can keep the response simple and return theNext-Rangeheader?Notes from Tina: