dali-lab / pine-beetle-backend

Visualizing data and predicting outbreaks of Southern Pine Beetles across twenty US states. Built in the DALI Lab.
https://spbpredict.com
0 stars 0 forks source link

implement aggregation pipeline for extracting only a few fields #134

Closed jeffzyliu closed 1 year ago

jeffzyliu commented 1 year ago

Description

fields to keep: year: state: [location] where location is 'county' or 'rangerDistrict' ([location] will actually be the right syntax somewhere for this) spbPer2Weeks: probSpotsGT50: spotst0: (or the ln version) (don't keep _id we dont need it here)

steps to doing this:

  1. go in src/utils/aggregate.js, look at this code https://github.com/dali-lab/pine-beetle-backend/blob/c16aaad6d5bc0f501ce9f4c5a568875b38bff84c/src/utils/aggregate.js#L76-L117
  2. copy that code and name it something else. we don't need the $group object, get rid of it. inside project, we don't need year: '_id'.
  3. keep the $sort and $match clauses untouched
  4. project should be fieldName: 1 for each of those fields to keep above. keep the _id: 0 as well.
  5. finally, add the import to src/utils/index.js, and then add that to the export too in that file

this gives us a way to extract only those fields from a data slice while saving CPU time and memory.

Type of Issue