add smoothing functions

alee commented 4 years ago

[ ] Trailing running average of N years
[ ] Centered running average of N years
[ ] Spline fit to timeseries
[ ] Kernel density smoothing (lower priority)

Issues:

moving averages may have incomplete windows of data, response should always have a slot for alerts / messages from the server that could be displayed (e.g., @tako2428 suggests an alert to the user saying "Portions of this window have missing data")

cpritcha commented 3 years ago

Trailing Average (N years)

Request

{
  "dataset": "paleocar",
  "variable": "paleocar_precipitation",
  "studyArea": {
    "type": "Point",
    "coordinates": [100, 100]
  },
  "timeseries": {
    "yearRange": [5, 10],
    "method": "trailingAverage",
    "width": 7
  },
  "postTransforms": [
    {
      "method": "z-score"
    }
  ]
}

The postTransforms key is optional and includes an array of transforms to apply. The one in this example takes the moving average values and computes z-scores using the mean and standard deviation computed from each cell for each selected year in the study area.

Good Response (200)

{
  "yearRange": [8, 10],
  "timeseries": [103, 105, 106.4]
}

Note that the length of the array is shortened to deal with the absence of enough previous years to compute the 7 year trailing average.

Bad Response (400)


{
  "label": "Study area is too large",
  "detailed": "Selected study of 5.3km2 is larger than max area of 5km2",
}

cpritcha commented 3 years ago

Running Average (N years)

Request

{
  "dataset": "paleocar",
  "variable": "paleocar_precipitation",
  "studyArea": {
    "type": "Point",
    "coordinates": [100, 100]
  },
  "timeseries": {
    "yearRange": [5, 10],
    "method": "runningAverage",
    "width": 7
  }
}

Good Response (200)

{
  "yearRange": [8, 10],
  "timeseries": [103, 105, 106.4]
}

Note that the length of the array is shortened to deal with the absence of enough previous years to compute the 7 year running average.

Bad Response (400)


{
  "label": "Study area is too large",
  "detailed": "Selected study of 5.3km2 is larger than max area of 5km2",
}

cpritcha commented 3 years ago

Polynomial Spline

Request

{
  "dataset": "paleocar",
  "variable": "paleocar_precipitation",
  "studyArea": {
    "type": "Point",
    "coordinates": [100, 100]
  },
  "timeseries": {
    "yearRange": [5, 10],
    "method": "smoothingSpline",
    "selection": {
      "method": "generalizedCrossValidation"
    }
  }
}

Good Response (200)

{
  "yearRange": [5, 10],
  "timeseries": [101, 102.4, 105.2, 103, 105, 106.4]
}

Bad Response (400)


{
  "label": "Study area is too large",
  "detailed": "Selected study of 5.3km2 is larger than max area of 5km2",
}

cpritcha commented 3 years ago

Kernel Regression

Request

{
  "dataset": "paleocar",
  "variable": "paleocar_precipitation",
  "studyArea": {
    "type": "Point",
    "coordinates": [100, 100]
  },
  "timeseries": {
    "yearRange": [5, 10],
    "method": "kernelRegression",
   "submethod": "localLinear",
   "kernel": "epanechnikov",
    "bandwidthSelection": {
      "method": "asymptoticMeanIntegratedSquaredError"
    }
  }
}

or

{
  "dataset": "paleocar",
  "variable": "paleocar_precipitation",
  "studyArea": {
    "type": "Point",
    "coordinates": [100, 100]
  },
  "timeseries": {
    "yearRange": [5, 10],
    "method": "kernelRegression",
   "submethod": "localLinear",
   "kernel": "epanechnikov",
    "bandwidthSelection": {
      "method": "manual",
      "bandwidth": 2.5
    }
  }
}

Good Response (200)

{
  "yearRange": [5, 10],
  "timeseries": [103, 102.4, 105.2, 103, 105, 106.4]
}

Maybe the selected bandwidth should get returned here too?

Bad Response (400)

{
  "label": "Study area is too large",
  "detailed": "Selected study of 5.3km2 is larger than max area of 5km2",
}

cpritcha commented 3 years ago

Corner Cases

how do we handle missing data?
- @bocinsky return a warning that portions of the window contain missing data

Details

The backend performs a regression on a wider range than the years selected to avoid not having enough data (use data outside the selected years if needed for a moving average etcetera)

cpritcha commented 3 years ago

Here are some sample requests and responses for the time series service. Any thoughts @tmcphillips @bocinsky?

cpritcha commented 3 years ago

z-score transform options

use mean and standard deviation for selected years and study area (don't use entire map)

alee commented 3 years ago

@cpritcha you should put that comment in #34

also, the good response should have optional label/messages to the user e.g. { "message": { "label": ..., "details": ... }

I think you should rename detailed to details or just message

alee commented 3 years ago

centered running average and trailing running averages have been implemented, z-score fixed interval still pending #34

openskope / skopeui