Add multi-processing

[cortadocodes]

Summary

Convert data-gateway to run on multiple processes and make it more efficient to avoid serial port buffer overflows.

Note: the test coverage has appeared to drop because the test runner isn't taking into account code run in other processes - this will increase back to current levels once the runner is configured to do this.

Contents (#30)

New features

• Run packet reading and packet parsing in different processes, using a process-safe queue to send packets from the reader to the parser

Enhancements

• Adjust log levels for cleaner output
• Continue running if an unknown packet is read from serial port
• Allow specific time after which to stop gateway if when no new data received
• Add process name to logging context
• Use multiprocessing logger throughout data_gateway package
• Log the location(s) that data will be persisted to
• Stop the closing and re-opening of the serial port on buffer overflow
• Log a warning if no routine is provided in non-interactive mode
• Stop the gateway if there's an error in either of the commands threads

Fixes

• Add force_persist method to NoOperationContextManager
• Stop packet reader when a routine finishes
• Ensure packet reader has correct default output directory
• Ensure routines containing a stop command only schedule and run the commands up until and including the stop command
• Ensure routines don't stop the gateway if/when they time out

Refactoring

• Move logic for starting gateway into new DataGateway class
• Rename PacketReader.parse_payload to parse_packets
• Confine notion of session_subdirectory to PacketReader and remove it from the persistence module

Testing

• Use simpler 0 and 1 values for RUN_DEPLOYMENT_TESTS environment variable
• Avoid potentially missing environment variable when skipping test
• Add missing bucket and project arguments in CLI tests
• Remove CredentialsEnvironmentVariableAsFile
• Mock BiqQuery client in tests
• Only save locally in CLI tests
• Remove problematic test (Windows)
• Move non-Cloud Function tests into test_data_gateway subpackage
• Remove use of temporary directories in DataGateway tests to avoid colons in cloud paths on Windows

[codecov-commenter]

Codecov Report

Merging #30 (cd8efc8) into main (edf6292) will decrease coverage by 14.94%. The diff coverage is 81.93%.

@@             Coverage Diff             @@
##             main      #30       +/-   ##
===========================================
- Coverage   93.94%   78.99%   -14.95%     
===========================================
  Files          12       13        +1     
  Lines         661      719       +58     
===========================================
- Hits          621      568       -53     
- Misses         40      151      +111     

Impacted Files	Coverage Δ
data_gateway/packet_reader.py	45.77% <56.33%> (-46.31%)	:arrow_down:
data_gateway/cli.py	87.83% <88.88%> (-3.20%)	:arrow_down:
data_gateway/routine.py	93.87% <88.88%> (-6.13%)	:arrow_down:
data_gateway/data_gateway.py	94.62% <94.62%> (ø)
data_gateway/persistence.py	94.53% <95.83%> (-0.82%)	:arrow_down:
data_gateway/__init__.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update eb1ee44...cd8efc8. Read the comment docs.