search

anvilco / spectaql

Autogenerate static GraphQL API documentation
https://useanvil.com/docs/api/graphql/reference/
MIT License
1.12k stars 118 forks source link

fix: ensure scalar examples are consistent #976

Open jedevc opened 3 months ago

jedevc commented 3 months ago

Heya! While working with spectaql, I noticed that example outputs produce random results - this can be an issue, since it means that multiple runs of spectaql with the same input can very easily produce different output.

Ideally, this shouldn't be the case, and all output should be as consistent and reproducible as possible.

Maybe this isn't the right way to go about this! :tada: And there's a neater way to do it, maybe a setting that allows for getting reproducible output by inspecting the environment/etc - happy to update the PR to do something else, if this approach isn't desirable.

newhouse commented 3 months ago

HI there @jedevc and thanks for this!

I'm going to have to think about this a bit more as I think there's maybe a different approach. This idea/problem has come up before and I think the solution is more complicated than these changes right here.

jedevc commented 3 months ago

Thanks @newhouse! Feel free to let me know if you have ideas for a different approach, I'm happy to dive in a bit more, I'm currently (ab)using a fork so we can get this setup working, but I'd much rather use upstream as much as possible! :tada:

Maybe an approach where we could take the hash of some canonicalized version of the graphql schema, and then use examples[hash(<schema hash> + <example name>) % len(examples)]? Then you could still have the random examples, but they're now deterministic based on the hash of the schema? (though that is still a bit weird that changing unrelated parts of the schema causes changes in the examples for other parts of the schema).

marknuzz commented 2 months ago

I was disappointed to see the large diffs created by the generator, as it means committing the output to a repository becomes complicated and any 'real' diffs become lost in the noise, among other complications. Would love to see an update from the maintainer.