Remove preprocessing from the cloud function

[time-trader]

Summary

Remove preprocessing from gateway cloud function. This means that there is no more:

• Attempt to convert the window samples to a constant time-step
• Searching for missing data and NAN padding
• Conversion of fixed point numbers to physical variables

All of these will be done separately on the stored raw data (in order not to mess with ingress pipeline) each time new a sampling procedure is adopted or a different calibration is done.

Contents (#41)

Enhancements

• Remove pre-processing from ingress-eu cloud function
• Convert timestamps with convert_window_timestamps_to_unix_time method instead of preprocessor

Operations

• Give more time for cloud function builds to complete

[thclark]

Let's spend our next week session looking at how to orchestrate that post-processing. We also need to be careful how to manage NaNs - need to check that they're going into the DB properly.

[time-trader]

Let's spend our next week session looking at how to orchestrate that post-processing. We also need to be careful how to manage NaNs - need to check that they're going into the DB properly.

With preprocessing removed there will be no more NaNs going into Raw data DB.

[thclark]

Let's spend our next week session looking at how to orchestrate that post-processing. We also need to be careful how to manage NaNs - need to check that they're going into the DB properly.

With preprocessing removed there will be no more NaNs going into Raw data DB.

OK great, got the wrong end of the stick there.

[codecov-commenter]

Codecov Report

Merging #41 (a305a49) into main (edf6292) will decrease coverage by 14.37%. The diff coverage is 83.02%.

@@             Coverage Diff             @@
##             main      #41       +/-   ##
===========================================
- Coverage   93.94%   79.57%   -14.38%     
===========================================
  Files          12       13        +1     
  Lines         661      749       +88     
===========================================
- Hits          621      596       -25     
- Misses         40      153      +113     

Impacted Files	Coverage Δ
data_gateway/packet_reader.py	48.06% <61.95%> (-44.02%)	:arrow_down:
data_gateway/routine.py	93.87% <88.88%> (-6.13%)	:arrow_down:
data_gateway/cli.py	89.58% <93.75%> (-1.46%)	:arrow_down:
data_gateway/data_gateway.py	94.62% <94.62%> (ø)
data_gateway/persistence.py	94.53% <95.83%> (-0.82%)	:arrow_down:
data_gateway/__init__.py	100.00% <100.00%> (ø)
data_gateway/dummy_serial/dummy_serial.py	91.42% <0.00%> (-2.86%)	:arrow_down:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9eb0b0a...a305a49. Read the comment docs.