Convert packet reader to allow multiple nodes

[thclark]

Contents (#71)

Enhancements

• Add a packet key map property to the configuration
• Cache the packet key map to allow quick usage in the reader
• Update batchers for multiple nodes
• Remove sensor timestamp offset calculation and persistence
• Use node ID when persisting data in PacketReader
• Remove timestamp conversion from cloud function
• Update cloud function to store data for multiple nodes
• Remove sensor coordinates from installation data
• Move sensor coordinates validation into NodeConfiguration
• Update installation data references and field names
• Remove blade ID from installation table
• Remove redundant project name field from microphone data table
• Add stop rutine ability
• Add example configs for aventa tests
• Add bucket name as environment variable
• Add quick routine for debugging
• Make cloud functions robust against empty data

Fixes

• Fix configuration conversions to/from dictionaries
• Include sensor periods in node configuration serialisation
• Get serial port buffer size correctly
• Update non-persistence usages of configuration in PacketReader
• Fix list of default sensor names
• Ensure node IDs are always strings
• Get the node ID from the serial port properly
• Add missing "node_id" subscriptions in PacketReader
• Update to new way of accessing samples per packet for sensors
• Update configuration keys used in cloud function
• Ensure dictionary combination works on python3.8
• Add bye endian to decode
• Make available packet keys a list of ints
• Ensure exceptions are not removed by the finally clause
• Specify local build in docker-compose
• Make default path on stop_routine work
• Not strictly ascending example routine
• Remove dead getBattery command
• Give a default bucket name and make override work

Operations

• Update tests to python 3.9
• Ignore wireguard creds in root dir
• Enable use of balena
• Tweaks to dockerfile
• Enable device variables to override paths for quicker cli use
• Add docker-compose file to get named volume in balena
• Add privilege to the main container to access hardware
• Remove unused data dir from dockerfile
• Remove config file from supervisord generation
• Add debug logger for rerceived packet types
• Add a balena deploy workflow for pushes to main
• Ignore vscode local settings
• Add installation reference to example configs
• Prevent logger from flooding outputs in packet reader
• Add diagnostic to persistence
• Add diagnostic to persistence
• temporary add of old testing script
• Log upload destination prior to attempt
• Add debugging display for use in interactive mode
• Make message more informative
• Add debug statement to window handler
• Allow deployment of gcloud test functions
• Change to Workload Identity Federation
• Give permission for workload identity federation use
• Wire up correct entry points
• Make ingress trigger on file uploads
• Remove incompatible trigger type specificaiton

Dependencies

• Remove cached property to allow use on python 3.7

Refactoring

• Remove duplicated import
• Simplify dictcomps
• Improve readability in PacketReader
• Factor out node config in PacketReader
• Change packet_key things to leading bytes
• Refactor Configuration slightly

Testing

• Restore testing environment to python3.8
• Update configuration usage in test
• Add node ID to queue in tests
• Update data gateway tests to use node IDs
• Rename test constant
• Update CLI test
• Update DatasetMixin.random_window to multi-node format
• Fix upload test
• Simplify test method
• Update CreateInstallation tests
• Use updated Configuration method name
• Update error message test
• Update valid_configuration.json test fixture
• Update info packets test
• Remove directory made by test when finished
• Test DataGateway with two nodes
• Rename test constant

Other

• Update data gateway to use new config
• Start developing examples set for configurations and routines
• Update configuration docstrings
• Update persistence docstrings
• Add extra examples
• Adding new capability to parser
• Warn on missing packet key
• Debugging lost packet type on multi node configuration
• Remove temporarily the sensor stops on clearup
• Additional debug logging of packet types
• Handle base station as node_id=0
• Fix packet key offset
• Work around packets from base station
• Fix up example
• Update node id
• Set id to 2
• Update developer installation instructions
• Tweak the configuration for node 4
• Remove too-verbose logging on packet reciept

[codecov-commenter]

Codecov Report

:exclamation: No coverage uploaded for pull request base (feature/multiple-node-support@3784173). Click here to learn what that means. The diff coverage is n/a.

@@                       Coverage Diff                        @@
##             feature/multiple-node-support      #71   +/-   ##
================================================================
  Coverage                                 ?   79.57%           
================================================================
  Files                                    ?       14           
  Lines                                    ?      847           
  Branches                                 ?        0           
================================================================
  Hits                                     ?      674           
  Misses                                   ?      173           
  Partials                                 ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3784173...bdcd1af. Read the comment docs.