Enable question chaining

[cortadocodes]

Summary

This release makes major improvements to event handling and question auditing. Some of the main changes are:

• Questions are now automatically associated with their parent question and the question that originated them, however deep they are in a question tree
• Events are ordered by datetime by the event backend, not the SDK
• Better feedback is provided when asking questions in parallel
• You can specify the event store to use
• Log message contexts have been slimmed down without losing any information, and events are replayable with no context (good for smaller screens)
• Various public classes and functions are faster and easier to use
• Question retries have the same question UUID

Contents (#660)

IMPORTANT: There are 6 breaking changes.

New features

Events

• 💥 BREAKING CHANGE: Add parent_question_uuid, originator_question_uuid, originator and retry_count event attributes
• Avoid redelivery of questions by checking the event store on delivery

Event handlers

• Add ability to not include service metadata in logs in even handlers
• Enable EventReplayer to handle question events
• Add RegisteredTemporaryDirectory class, use it when downloading datasets, and add ability to delete them at end of analysis

Enhancements

Resources

• 💥 BREAKING CHANGE: Make datasets recursive by default in Dataset
• Log a warning if a dataset is empty at instantiation

Services

• 💥 BREAKING CHANGE: Remove name argument from Service and provide an SRUID to Child internal service instead of a name
• Improve logging of errors, retries, and threading in Child.ask_multiple
• Order pub/sub messages by datetime using ordering key and remove order event attribute
• Set question UUIDs in advance in Child.ask_multiple

Subscriptions

• Allow existing subscriptions in create_push_subscription
• Give feedback on (un)successful push subscription creation in CLI

Questions and events

• Remove unnecessary sender argument from get_events and make getting the tail of events the default
• Allow retried questions to have the same UUID
• Allow explicit question retries by using retry_count attribute
• Return empty list from get_events if no events for question

Service configuration

• Allow setting of event store table ID and delete_local_files in service configuration
• Use envvar to specify service configuration location by default
• Add overrides option to Runner.from_configuration

Other

• Log warning when PYTHONUNBUFFERED envvar is unset
• Remove "analysis-" from start of question UUIDs in log context

Fixes

• 💥 BREAKING CHANGE: Return question UUID alongside error from Child.ask_multiple for failed questions
• Set analysis ID at start of Runner.run
• Emit correct logs when no diagnostics available with octue get-diagnostics
• Fix deserialisation of events in get_events
• Use (meta-)generation agnostic retry strategy with cloud storage
• Return correct question UUIDs with failed questions from Child.ask_multiple
• Avoid logging that app failed when it didn't when uploading diagnostics
• Allow setting of max_workers when CPU count is indeterminate
• Disable delete_local_files by default

Operations

• Update event handler and its bigquery table

Dependencies

• Loosen Sphinx and other docs package ranges
• Remove unneeded db-dtypes package
• Make google-cloud-bigquery a mandatory dependency
• Upgrade google-cloud-secret-manager

Refactoring

Event handlers

• 💥 BREAKING CHANGE: Remove redundant datetime from delivery ack and heartbeat events
• 💥 BREAKING CHANGE: Rename originator event attribute to parent
• Factor out finalising and cleaning up in Runner
• Move service accounts into separate terraform file
• Cache metadata against datafile/dataset instead of path
• Rename python3.9 dockerfile to reflect its python version

---

Upgrade instructions

• Add recursive=False to Dataset instantiations
• Update all services in your service network to use octue>=0.56.0
• Use version 0.6.1 of the event handler or above and a correspondingly up-to-date BigQuery table.
• Swap the internal_service_name argument for internal_sruid argument to Child.__init__ and provide a valid SRUID
• Instances of Service can no longer be given names. Please give them a valid SRUID instead.
• To get the unraised exception from a failed answer returned by Child.ask_multiple, access the zeroth element e.g. if the third question failed:

  answers = Child.ask_multiple(*questions)
  exception, question_uuid = answers[3]

• Service.received_events, AbstractEventHandler.handled_events, and Child.received_events now include event attributes instead of just the event. These attributes/properties now return a list of dictionaries with the keys {"event", "attributes"}, where what was previously returned is now mapped to the "event" key.
• Stop providing the recipient argument to EventReplayer and GoogleCloudPubSubEventHandler - it's now automatically acquired from each event's attributes
• Stop passing the skip_missing_events_after argument to EventReplayer and GoogleCloudPubSubEventHandler
• Stop using the awaiting_missing_event and time_since_missing_event properties on the event handlers

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 93.35260% with 23 lines in your changes missing coverage. Please review.

Project coverage is 94.56%. Comparing base (330ea82) to head (a21d295). Report is 27 commits behind head on main.

Files	Patch %	Lines
octue/cloud/pub_sub/bigquery.py	75.67%	9 Missing :warning:
octue/cloud/pub_sub/events.py	77.27%	5 Missing :warning:
octue/runner.py	91.66%	3 Missing :warning:
...cloud/deployment/google/answer_pub_sub_question.py	87.50%	1 Missing :warning:
octue/cloud/events/handler.py	95.65%	1 Missing :warning:
octue/cloud/events/replayer.py	88.88%	1 Missing :warning:
octue/configuration.py	87.50%	1 Missing :warning:
octue/mixins/metadata.py	75.00%	1 Missing :warning:
octue/resources/datafile.py	85.71%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #660 +/- ## ========================================== - Coverage 94.62% 94.56% -0.06% ========================================== Files 82 82 Lines 3552 3626 +74 ========================================== + Hits 3361 3429 +68 - Misses 191 197 +6 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.