Discussion - Graphql Spec compatability tests

[shobhitchittora]

Hi! As a part of a research around the simplifying graphql spec tests across implementations, I've stumbled upon the specs written here for graphql-ruby. I'd really appreciate if I can get an ideal around how difficult it has been to maintain this test suite, and how often does it go out of parity with the actual graphql spec? Were there cases when mis-understanding the spec led to a issues with the library ? Would having a generic set of compatibility tests help around these issues, if any?

[rmosolgo]

Hi! Thanks for asking. I'd say spec compliance issues are pretty rare. Most of the time, people using GraphQL-Ruby haven't read the spec, and they aren't using any other GraphQL implementations, so any diverging behavior between different implementations isn't a problem.

However, there are some edge cases that pop up from time to time: null input handling, or default values, or fragment spreads, and so on. They're usually addressed on an as-needed basis.

I would have liked to adopt https://github.com/graphql-cats/graphql-cats, but I never had the time or energy.

Does that answer your question?

[swalkinshaw]

From memory there's a few other types of differences between this gem and the graphql-js reference implementation: parsing edge cases and error messages. Though I don't either are in the spec? So I guess it depends on what the spec test would cover exactly (if anything beyond the actual spec).

[rmosolgo]

error messages

That's a huge one -- I wish those were all in sync (that GraphQL-Ruby matched GraphQL-JS), but I can't really prioritize figuring it out, much less doing the work 😩

[pranshuchittora]

As of now, the schema definition can only be done by defining a class that extends GraphQL::Schema in ruby. It does not support defining the schema in a .graphql file (SDL).

Referring to the graphql-cats, it uses the schema defined in SDL format -> sample test. If the project needs to adopt any popular compliance test framework then it must support resolving the SDL.

Any heads up regarding the support of the SDL directly or by using the codegen techniques to convert the schemas defined in SDL to ruby class?

[rmosolgo]

Yes, you can also build a schema with Schema.from_definition, eg:

GraphQL::Schema.from_definition <<-GRAPHQL

type Query {
  # ...
}
# ...

GRAPHQL

[pranshuchittora]

Yes, you can also build a schema with Schema.from_definition, eg:

GraphQL::Schema.from_definition <<-GRAPHQL

That looks interesting.

I am researching to create a bunch of GraphQL spec compliance tests with the help of GrapQL foundation and all library maintainer to test the various GraphQL implementations(e.g. graphql-ruby, graphql-java, etc). To perform such generic testing the only way is to spawn an instance of GraphQL server (graphql-ruby) and test it on various queries & schemas over HTTP. As all the implementations support running over HTTP.

@rmosolgo how does that sound in terms of maintainability and community acceptance. Is there any alternative design that you would like to suggest?

This is an initiative by the GraphQL foundation for the Google Summer of Code 2020. Project idea can be found here

[pranshuchittora]

Hello all, Do check this discussion on the graphql-spec -> https://github.com/graphql/graphql-spec/issues/698

[rmosolgo]

I don't have any other suggestions here. Good luck on the project!