dependency sustainability

[oliver-sanders]

Cylc UI Server has two dependencies from the graphql-python org that are starting to look worrying:

• graphene-tornado
  • 
  • Adds tornado support to Graphene.
  • Bruno maintains the conda-forge feedstock.
  • A v3 beta was started but never released.
  • One bug we have spotted may be related - https://github.com/cylc/cylc-uiserver/issues/328
• graphql-ws
  • 
  • Adds websocket support to GraphQL).
  • Hillary & Bruno maintain the conda-forge feedstock.
  • The project was forked for Graphene 3, but then abandoned.

These projects are fairly small and live in the graphql-python organisation, however, are in need of a little TLC. They are currently stuck on Graphene 2, Graphene has since moved onto version 3 (Nov 2021).

Unfortunately it is relatively unusual to write GraphQL servers in Python (clients less unusual), GraphQL came out the JS ecosystem where the most popular server lives. Consequently there isn't that much interest from the wider community in some of these more niche integrations in the Python ecosystem.

To safeguard the future of these projects they need to make it to Graphene 3, unfortunately this requires an understanding of Graphene that we don't have which makes it difficult for us to help.

What can/should we do and when can we afford the time to do it?

[oliver-sanders]

If the Graphene route becomes difficult there are two other Python GraphQL server implementations which may or may not offer an easier route:

• Ariadne
• Tartiflette

[oliver-sanders]

update: 2023

graphene-tornado is now graphene >=3 compatible, but pending release, pending a maintainer to make the release.

Because graphql-ws is still graphene < 3 this doesn't help us so #388 is blocked with or without this release.

[oliver-sanders]

Moving to Ariadne or Tartiflette is completely possible, however, this would involve swapping our GraphQL "engine" for theirs which could introduce migration pain.

Both Ariadne and Tartiflette are schema-first which means we would need to serialise our Graphene schema, then load it back in again. This should work but it might require a little extra work hooking in our resolvers, etc.

Furthermore, although both support websockets, neither support Tornado so we would still require an integration there bringing us back to the original problem.

So long story short, both are interesting alternatives which we might have considered had they been around at the time, however, migrating is not likely to be a trivial activity.

[edit] correction Ariadne uses graphql-core as its "GraphQL engine" the same thing you get with Graphene, Tartiflette presumably uses something different.

[oliver-sanders]

Update: 2024-08

State of graphene-tornado and graphql-ws

Unfortunately, it does not look like either graphene-tornado or graphql-ws are going to receive any further updates. I have poked, but there doesn't seem to be any appetite for pushing these projects forward. The maintainers are suggesting we move away from these specific integrations and towards another of their projects, graphql-server as a more unified solution. However, graphql-server presently does not support either subscriptions, websockets or tornado, so this isn't a viable solution either. There is a v3 branch with experimental support for subscriptions + websockets on this repo, however, it has not been very active and doesn't look like it is likely to make release. If it was just one missing feature set, we might be able to get involved and thrash it out, but with multiple missing feature sets, it is unlikely that we would be able to make the required progress, especially with no knowledge of the graphql-core library.

Alternative tooling

The good news is that other Python GraphQL servers have advanced reasonably during this period. Notably Strawberry and Ariande appear much more mature solutions, both providing subscription and websocket support. Note, neither has reached a v1 release yet. The smaller Tartiflette project may also be promising.

There is also the graphql-server project mentioned above, however, this does not support either subscriptions or websockets.

None of these options supports Tornado servers, so we will have to accept developing this integration ourselves :(

Note, Strawberry and Ariande appear to have GraphQl-core as a dependency, so may just be frameworks wrapping this library (similar to graphql-server)?

Component parts

There are four components to the GraphQL server-side stack (my naming):

• Schema definition - The tool or framework used to define the GraphQL schema.
• Request execution - The tool that takes a request, calls the required resolvers, and spits out a response.
• Server integration - The library that hooks the "request execution" bit into the required web server framework (Tornado).
• Protocol integration - Related to the "server integration", this bit provides support for issuing subscriptions over websockets.

Migration options

Option	Schema Definition	Request Execution	Server Integration (tornado + subscriptions)	Protocol Integration (websockets)
Current	graphene V2	graphql-core V2	graphene-tornado	graphql-ws
(1)	graphene V3	graphql-core V3	graphql-server V3 (experimental) + bespoke tornado integration	graphql-server V3 (experimental)
(2)	graphene V3	graphql-core V3	bespoke	bespoke
(3)	strawberry	strawberry	bespoke	strawberry [1] ?
(4)	graphene V3	ariande	bespoke	ariande [1] ?
(5)	ariande	ariande	bespoke	ariande [1] ?

[1] Websockets and subscriptions are both supported, however, research would be needed to determine how abstract this implementation is, if it is written into specific server integrations, then it might not be easily reusable in a Tornado context.

All options are going to be highly disruptive, expected pain points:

• We will need to re-develop at least the top layer of our resolvers, possibly the whole resolver stack to work with the new "request execution" tool.
• With (3) and (5) we would need to port away from Graphene for schema definition (easy, but time consuming). There is the potential that this would cause issues with aotf in cylc-ui requiring frontend changes, especially if custom types do not work in the same way (we use custom types to represent more specific strings e.g. "cycle points" or "task ids", aotf uses this context for mutation association).
• The stip-null functionality is currently implemented in a slightly odd way, if we can turn this into a "middleware" it should be more portable.

Migration notes:

• Serialising the schema: It should be possible to serialise our existing Graphene defined schema into an implementation independent format. This would avoid the need to migrate away from Graphene as part of moving to another framework. Unfortunately, the graphql-core v2 code (we are currently using) does not support documentation (e.g. description fields) which are important to us (they carry mutation documentation through to the UI) so we would need to upgrade to graphql-core v3 at the same time as migrating to a new server in order to do this.
• Upgrading to graphql-core v3: The v3 implementation is a complete re-write of the v2 implementation, so this would be similar to migrating to another server framework. Support for graphql-backends was removed at v3 which is important as this is how we shoehorn in the "null-stripping" feature. There is a notable lack of documentation to assist migration.
• Upgrading to graphene v3: This should be relatively straightforward, however, there is a notable lack of documentation to assist migration.

Suggestions

It might be easier to attempt this as a ground-up rewrite :(

Try implementing a simple schema implementing a simple subscription and see how hard it is to expose this through Tornado before continuing.

The popularity of the Strawberry and Ariande frameworks is attractive.