Summary

This pull request:

Makes the SDK fully event-driven by using a single topic to emit/consume events
Majorly refactors the event handler to facilitate asynchronous event retrieval
Adds the ability to get and replay events from a BigQuery store

Contents (#632)

IMPORTANT: There are 4 breaking changes.

New features

💥 BREAKING CHANGE: https://github.com/octue/octue-sdk-python/pull/639
Add get_events function for retrieving events asynchronously from BigQuery
Add EventReplayer class to replay asynchronously-retrieved events
Add Manifest.download method

Enhancements

Get subscription project name from topic by default
Improve asking of asynchronous questions via Child.ask
Return download path from Dataset.download
Include question UUID in delivery acknowledgement log message
Improve handling of invalid events
Add datetime and uuid attributes to all events

Fixes

Await successful publishing of question messages
Fix api_access_endpoint usage in mock_generate_signed_url

Operations

Add test BigQuery dataset, cloud function, and IAM roles to terraform config
Switch to reusable workflows where possible

Dependencies

Add google-cloud-bigquery
Upgrade coolname
Add db-dtypes for converting bigquery rows to dataframes

Refactoring

💥 BREAKING CHANGE: Rename x.received_messages to x.received_events
💥 BREAKING CHANGE: Rename record_messages parameters to record_events
💥 BREAKING CHANGE: Update ChildEmulator to use event* instead of message*
Factor out making minimal dictionary
Factor out creating push subscription
Factor out emitting question event in Service.ask
Factor out event handlers and related logic from OrderedMessageHandler into new AbstractEventHandler
Move validation module into octue.cloud.events subpackage
Rename OrderedMessageHandler to GoogleCloudPubSubEventHandler
Rename "message" to "event" in event handler classes
Rename GooglePubSubHandler to GoogleCloudPubSubHandler

Chores

Update licence year to 2024

Testing

Simplify various tests

Upgrade instructions

Update all services in your services network to this version of octue or later (0.53.0+)
Replace any usages of the received_messages methods with received_events
Replace any usages of the record_messages parameters with record_events
Replace the word message with event in usages of ChildEmulator methods (apart from in the case of monitor_message)

Codecov Report

Attention: Patch coverage is 95.15789% with 23 lines in your changes are missing coverage. Please review.

Project coverage is 94.79%. Comparing base (e9d5d92) to head (14481ac). Report is 21 commits behind head on main.

Files	Patch %	Lines
octue/cloud/events/handler.py	94.32%	8 Missing :warning:
octue/cloud/emulators/child.py	91.66%	3 Missing :warning:
octue/cloud/emulators/_pub_sub.py	86.66%	2 Missing :warning:
octue/cloud/events/counter.py	83.33%	2 Missing :warning:
octue/cloud/pub_sub/bigquery.py	92.30%	2 Missing :warning:
octue/cloud/pub_sub/events.py	97.67%	2 Missing :warning:
octue/cloud/pub_sub/service.py	97.01%	2 Missing :warning:
...cloud/deployment/google/answer_pub_sub_question.py	50.00%	1 Missing :warning:
octue/resources/child.py	75.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #632 +/- ## ========================================== - Coverage 94.86% 94.79% -0.08% ========================================== Files 77 82 +5 Lines 3409 3535 +126 ========================================== + Hits 3234 3351 +117 - Misses 175 184 +9 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

octue / octue-sdk-python

Switch to event-driven infrastructure and improve support for asynchronous questions #632