GraphQL API breaking change detection

[novoj]

We have a backward compatibility problems with the team already working on Next.JS problem. I'd like to introduce some process that will warn us before switching to a new version that the GraphQL API schema has been changed by a new evitaDB version / compare schemas on demand when catalog has been reindexed and the publication process built different schema.

Initial idea:

• there be a system endpoint that will be able to compare raw outputs of the web API schemas
• the endpoint will work two ways:
  1. it accepts two catalog names in the very same evitaDB instance
  2. it accepts single catalog name and uploaded ZIP file with the schemas stored in a files with name catalogName.gql, catalogName.rest (or something like that)

The endpoint will compare the schemas and prints out differences in a human readable way (the raw block segments would be enough).

How it'll be used ...

Scenario 1.

I'll update the installation script so that the installation of the new version will look like this:

1. I download and start new docker image of the evitaDB under different container name and start it up with the same data location in read only mode
2. I call endpoints that can generate graphql / rest API schemas and store output to the file
3. I wait for the start and then call verification endpoint on a new container and upload ther the zip with files exported in step 2
4. if no change is detected I'll stop and delete the new container
5. I'll continue with the replacing the old version with the new one

When changes are detected, they'll be printed to the console (or a file) and the new container is deleted, but old is not replaced. The frontend team is safe - still works with the previous version.

Scenario 2.

I'll install a new version of the EdeeCMS server which acts as an evitaDB client and redefines the schemas according to a Java logic. When full reindexation occurs the new contents are indexed into the new catalog and at the end the current catalog is replaced with the newly built one. But before this happens I'll call the verification endpoint to compare previous and new APIs (both catalogs still exists) and the difference will be printed to the indexing job log (if no change occurs the job will result with OK status, if change occurs the job will result with WARNING status).

This will be handy in production as well.

[novoj]

During the discussion with @lukashornych we found out the graphql-java has already basic tools in place:

• graphql.schema.idl.SchemaPrinter for creating IDL (text outputs)
• graphql.schema.diffing.SchemaDiffing for comparison of two schemas (class is marked @Internal though)

If we invest some time into the investigation the compatibility check for GraphQL might be doable. If it works and brings a benefit we might provide the same functionality for REST API as well, but it's not on the table for now.

[novoj]

Similar tool exists also for Open API: https://github.com/OpenAPITools/openapi-diff

[novoj]

GraphQL schema in text form will be provided on GET method for url: https://server:port/gql/catalogName/ (and the same for the GraphQL schema of evita schema and system endpoint)

Open API schema in text form will be provided on GET method for url: https://server:port/rest/catalogName/ (and the same for the REST schema of evita schema and system endpoint) ... this is already done.

[novoj]

New repository created: https://github.com/FgForrest/evitaDB-API-DiffDive

This tool should

• accept two schema files (the source could be combined):
  • file system
  • URL (GET, no authorisation)
  • HTTP PUT (upload)
• in response it produces a MarkDown file describing the changes between the first and second schema
  • report will start with breaking changes first, which are most important.
  • then there will be a section with additions that are not considered breaking changes (or it can be a complete list, combining both breaking and non-breaking changes in an understandable form)

The tool will be distributed as a JAR file via MavenCentral and as a Docker image via DockerHub using evitaDB credentials and sharing the same package.

The tool will be installed on evitadb.io - for example as: https://diffdive.evitadb.io as a docker container. This URL will be used by evitaLab to visualise the report for two schemas listed in it. The tool will also be used in the CI/CD pipeline job in the future (not part of this task).

[novoj]

This is a good example of checking the GraphQL backward compatibility. We can find some inspiration there @lukashornych: https://www.apollographql.com/docs/graphos/delivery/schema-checks/#operation-checks

[lukashornych]

@novoj the GraphQL schema pretty printing endpoint is released.

The DSL schema can be fetched by simply making GET HTTP request to URL where the GraphQL instance is running (the POST method is reserved for queries and mutations). Although this is not how GET method should be used on GraphQL instance, we currently don't want to support GET request for queries, so we used it for DSL schema like we do it with REST API were GET request on root URL of certain catalog API returns OpenAPI specification.

So the DSL schema can be fetched from:

• /gql/system - returns DSL of system GraphQL instance
• /gql/{catalog name} - returns DSL of catalog GraphQL instance for data manipulation
• /gql/{catalog name}/schema - returns DSL of catalog GraphQL instance for evitaDB schema manipulation

[lukashornych]

So after some research of already available tools for diffing of GQL schemas and OpenAPI schemas, I would split the implementation into 2 sections: CLI checking for CI/CD, and evitaLab visualization for developers to AD-HOC check.

CLI

For CLI (CI/CD) env. there are already awesome tools that can be run in Docker, so I wouldn't invent anything ours.

GraphQL

• https://github.com/kamilkisiela/graphql-inspector
• works locally
• can be used as GH Action as well
• returns correct exit codes (1 for breaking change, 0 otherwise)
• prints human-readable overview of changes (not-Markdown)
• run using docker run --rm -v $PWD:/app:ro kamilkisiela/graphql-inspector graphql-inspector diff schema1.graphql schema2.graphql

Example output

Detected the following changes (2) between schemas:

[log] ✖  Field allLocales was removed from object type AdjustedPricePolicy
[log] ✔  Field allLocaless was added to object type AdjustedPricePolicy
[error] Detected 1 breaking change

OpenAPI

• already mentioned above https://github.com/OpenAPITools/openapi-diff
• works locally
• return correct exit codes on breaking change of any change (--fail-on-incompatible or --fail-on-changed)
• returns human-readable overview of changes (not-Markdown)
• run using docker run --rm -t -v $(pwd):/specs:ro openapitools/openapi-diff:latest --fail-on-incompatible /specs/schema1.json /specs/schema2.json

Example output

==========================================================================
==                            API CHANGE LOG                            ==
==========================================================================
                    Web services for catalog `evita`.                     
--------------------------------------------------------------------------
--                            What's Changed                            --
--------------------------------------------------------------------------
- GET    /Category/get
  Parameter:
    - Delete primaryKey in query
--------------------------------------------------------------------------
--                                Result                                --
--------------------------------------------------------------------------
                 API changes broke backward compatibility                 
--------------------------------------------------------------------------

evitaLab visualization

Here it's not so good. The already mentioned https://github.com/kamilkisiela/graphql-inspector can be used easily in browser JS, unfortunately for OpenAPI there is not such library. There is only https://www.npmjs.com/package/openapi-diff which works only in Node.js.

So I suggest that we create only a custom API in lab API for the OpenAPI diffing using the https://github.com/OpenAPITools/openapi-diff. The GraphQL diffing could be also done in the API or we could just use the JS library. Maybe I would try to create uniform lab API for both GQL and OA that would return JSON array with changes so that I don't need to create two separate UIs in evitaLab GUI.

Would do you think? @novoj

[novoj]

I completely agree with your conclusions. Is there any CL tool that would also protect incompatible changes in gRPC API for evitaDB API evolution?!

[lukashornych]

There are some CLI tools for comparing protobuf files (protobuf-format-diff or buf). But it could be difficult to obtain the proto files for comparison from running evitaDB instances. There is Java lib for introspection:

<dependency>
    <groupId>io.grpc</groupId>
    <artifactId>grpc-services</artifactId>
    <version>${grpc.version}</version>
</dependency>

with ProtoReflectionService.newInstance() service. But this service returns some kind of JSON from which we would have to reconstruct the proto files. Unfortunately, I couldn't find any existing tool for this.

Or maybe create our own service that would return the actual proto file from classpath? But there could still be problem with SSL certificate for mTLS I would guess.

[novoj]

I think, we just need something for our internal evitaDB CI/CD pipeline to avoid non-intentional backward incompatible changes in gRPC API. We probably don't need a tool for evitaLab / or a tooling that would our users use.

[lukashornych]

In such case, I think we could come up with some custom GH Action to retrieve proto files from previous release and compare it against the new ones with the mention tools.

[novoj]

This would be great - could we do that as a part of this issue?

[lukashornych]

I'll try to create some prototype then.

[novoj]

Can you @lukashornych summarize what is done and what needs to be done yet? We have to interrupt work on this issue due lack of time, so that when we return back, we have clear instructions where to continue.

[lukashornych]

@novoj I believe the lab API for comparing GQL and OpenAPI schemas is somewhat functional, it just needs some tidying. The CLI tools need to implemented.

Regarding evitaLab GUI, there is rough prototype for rendering GQL changes, but it is missing paging, i18n, file upload functionaility, todo tools, and other small things. GUI for OpenAPI schema is non-existent, but it will hopefully share most of the code with the GQL changes viewer. Also, it will need to be ported the the new internal API of evitaLab after refactoring, but that shouldn't be too time consuming.

The code for the API is in branch 219-api-compatibility-api (probably, broken, had to merge it onto a new dev). The code for evitaLab is in branch evitaLab#117-api-compatibility.

The question is, do we keep it in the custom lab REST API or do we either merge it into the gRPC driver (doesn't seem right) or do we create extension to the main gRPC API i.e. lab gRPC API instead of the lab REST API now that the evitaLab will work with the gRPC-web?