cortadocodes commented 2 years ago

Summary

Add support for thew new measurement campaigns table in BigQuery. This allows data to be associated with a measurement campaign that can be given a label and description. The code has also been cleaned up and some bugs fixed.

Contents (#108)

IMPORTANT: There are 2 breaking changes.

New features

Add measurement campaign data (start time, end time, label, and available sensors) to session table when uploading a window

Enhancements

Replace label field with session_reference in existing tables

Fixes

Raise warning if data gateway configuration not present in metadata

Reversions

💥 BREAKING CHANGE: Remove ability to provide label via CLI

Operations

Use latest conventional commit message checker
Use new Octue GitHub actions in workflows
Pin snok/install-poetry to avoid missing poetry bug
Ensure poetry is available within tox
Use new syntax for setting outputs in GitHub workflow

Dependencies

💥 BREAKING CHANGE: Drop python3.7 support
Update dependencies to patch some security vulnerabilities
Use octue==0.41.0
Add pandas dev dependency
Add missing pyarrow dev dependency
Combine old dev-dependencies group with new dev group
Regenerate requirements files for Raspberry Pi

Refactoring

Remove repetition of method call in BatchingUploader
Factor stopping the gateway out into a method

Testing

Avoid timezone issues with timestamp value of 0
Update MockBigQueryClient
Rename MockBigQueryClient.rows attribute to inserted_rows
Use Mock instead of types.SimpleNamespace
Update deployment test to check for session creation

Upgrade instructions

💥 Remove ability to provide label via CLI

Provide a "label" key mapped to the label in the "measurement_campaign" dictionary of the configuration file instead.

💥 Drop python3.7 support

Update to `python>=3.8`.

codecov-commenter commented 2 years ago

Codecov Report

Patch coverage: 50.00% and project coverage change: -0.76 :warning:

Comparison is base (cf3b26f) 60.87% compared to head (cec5a3d) 60.11%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #108 +/- ## ========================================== - Coverage 60.87% 60.11% -0.76% ========================================== Files 15 15 Lines 1122 1176 +54 ========================================== + Hits 683 707 +24 - Misses 439 469 +30 ``` | [Impacted Files](https://codecov.io/gh/aerosense-ai/data-gateway/pull/108?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai) | Coverage Δ | | |---|---|---| | [data\_gateway/packet\_reader.py](https://codecov.io/gh/aerosense-ai/data-gateway/pull/108?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai#diff-ZGF0YV9nYXRld2F5L3BhY2tldF9yZWFkZXIucHk=) | `44.76% <27.90%> (-2.13%)` | :arrow_down: | | [data\_gateway/data\_gateway.py](https://codecov.io/gh/aerosense-ai/data-gateway/pull/108?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai#diff-ZGF0YV9nYXRld2F5L2RhdGFfZ2F0ZXdheS5weQ==) | `90.00% <68.42%> (+1.11%)` | :arrow_up: | | [data\_gateway/persistence.py](https://codecov.io/gh/aerosense-ai/data-gateway/pull/108?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai#diff-ZGF0YV9nYXRld2F5L3BlcnNpc3RlbmNlLnB5) | `94.20% <87.50%> (-0.50%)` | :arrow_down: | | [data\_gateway/cli.py](https://codecov.io/gh/aerosense-ai/data-gateway/pull/108?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai#diff-ZGF0YV9nYXRld2F5L2NsaS5weQ==) | `91.57% <100.00%> (ø)` | | | [data\_gateway/configuration.py](https://codecov.io/gh/aerosense-ai/data-gateway/pull/108?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai#diff-ZGF0YV9nYXRld2F5L2NvbmZpZ3VyYXRpb24ucHk=) | `94.26% <100.00%> (+0.04%)` | :arrow_up: | | [data\_gateway/serial\_port.py](https://codecov.io/gh/aerosense-ai/data-gateway/pull/108?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai#diff-ZGF0YV9nYXRld2F5L3NlcmlhbF9wb3J0LnB5) | `93.75% <0.00%> (-6.25%)` | :arrow_down: | | [data\_gateway/dummy\_serial/dummy\_serial.py](https://codecov.io/gh/aerosense-ai/data-gateway/pull/108?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai#diff-ZGF0YV9nYXRld2F5L2R1bW15X3NlcmlhbC9kdW1teV9zZXJpYWwucHk=) | `90.00% <0.00%> (-4.29%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aerosense-ai)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

time-trader commented 1 year ago

We should refactor "session" to measurement_campaign" or something like that, to avoid confusion.

cortadocodes commented 1 year ago

Is the sessions table part of this PR relevant anymore as we'll be calculating sessions dynamically using the RawSignal class?