Make the octue communications API explicit

[thclark]

Feature request

Use Case

We're doing load testing of octue services, and of a worker running django_twined to orchestrate them. So we want to simulate messages coming back to the django_twined app, as if they were coming from running services.

Current state

It's quite difficult to figure out how to make such messages. We're digging deep into the octue library to figure out how to generate realistic events for log_records and other items.

Wider issue

The set of messages that can come back from an octue service is not defined in any one place. This is a big deal, because this clear, opinionated API surrounding inter-service communication probably is our biggest selling point.

Also, it's worth pointing out that it doesn't matter if it's using the octue library or not. Any process that places events onto a pubsub topic in this format is behaving as an octue service. So fundamentally, this is how we create services in languages other than python.

Proposed solution

• [x] Write a JSON schema for an event produced or consumed by the Octue communications API
• [x] Document and describe the schema in a high-level section of the documentation.
• [ ] Add helper functions in utilities to make valid messages correctly serialised for pubsub, eg make_event(type="log_record", msg="name", ...) or a similar API.

[cortadocodes]

Questions:

• [x] We've been allowing anything for input and output values so far but, in practice, have been using objects. Do we want to enforce that they should be objects (or alternatively, objects or arrays)? This would be better practice as it would allow people to update add supplementary data to their intputs/outputs without making breaking changes. Currently, I've gone for allowing objects and arrays only. On the other hand, the input values schema is set in the twine of a service so we may want to allow full customisation of that by letting input and output values be anything in this schema Update: We should leave it to the twine
• [x] How deep do we want to go with the schema on e.g. manifests? At the moment, I'm just requiring them to be objects as the SDK puts them in the correct format, but we could spell out and enforce the exact serialisation we want via the JSON schema Update: I'll put the manifest schema on Strands
• [x] Do we want to include or fully specify in this schema anything that gets validated by twined? e.g. the children field of a question is validated against a children schema inside twined. One thing we could do is factor out the schema within twined into Strands and then reference them in twined but also in this schema Update: I've put the children schema on Strands
• [x] We should validate Pub/Sub message attributes as well as message data. I think we should do this in a separate schema as it's a different object/level being validated. We could alternatively add another level to the schema which would be an object with data and attributes keys but we'd have to transform Pub/Sub messages into the same JSON object format before we could validate them in one go. What do you think @thclark? Update: I decided it makes more sense to keep all of it in one schema

To do in the next service communication breaking change:

• [x] I've noticed for non-required fields, we have a mix of both optional fields and required fields that can be null. I'll make these all fully optional fields that can't be null
• [x] Add a type field to the question message type to make it consistent with the other message types
• [x] Stop double-JSON-encoding output manifests when sending results in Service.answer (use .to_primitive instead of .serialise)
• [x] Update schema with new "required" fields and nulls etc
• [x] Disallow additional properties
• [x] Rename is_question
• [x] Release new version of octue/create-push-subscription
• [ ] Make "message" vs "data" vs "question/answer" nomenclature consistent
• [x] Make enum format consistent? e.g. make all enum values uppercase

[cortadocodes]

Other to-dos:

• [x] Separate Octue's message format from Pub/Sub's message format (we can convert between Octue and Pub/Sub before validating)
• [x] Change "Data for processing" to "Message content" or similar
• [x] Rename data to event