Operation store sync: full query manifest file

[mroach]

Is your feature request related to a problem? Please describe.

There's nothing broken, but here's my current setup and what I'm looking for.

When we build release Docker images for our frontend app, a build step runs graphql-ruby-client sync to sync queries to the server and generate the mapping of operation names to hashes. This works great.

The final built image has no GraphQL queries or server-executable JS. The image is just nginx, HTML, compiled JS.

The issue is that we have multiple environments aka stages, such as edge, demo, production, qa, etc. This means we have to run the image build and sync process for every environment to get operations to sync to the server even though for any given revision, they're all the same.

This became a more pronounced issue when we wanted to be able to spin-up the full stack locally with the backend services and frontend and use the already-built release images. Everything starts and works, but we have no stored operations, and the built image doesn't have them either.

Describe the solution you'd like

What I think would work is an option for graphql-ruby-client sync to generate a separate manifest/artifact file that includes all the data it sends to the operation store endpoint. This would allow a simple script to use it to hit the operation store endpoint on another server later on and sync the operations.

Essentially what I'm after is a file like the current result of --outfile except it also includes the original query text. I think that's all I'd need to re-sync to another server?

Or maybe the whole POST body could be optionally dumped to a file so I could curl -d @body.raw to another server later on?

{
  "changeset": "2024-11-01",
  "client": "frontend",
  "operations": [
    {
      "name": "MyProfile",
      "alias": "f9aee4f97a07bb2e1edcbb4f3e20deca",
      "body": "query MyProfile { id }"
    }
  ]
}

Describe alternatives you've considered

Our current workarounds are ugly and involve checking-out the git revision associated with the built image and running the sync client inside there and expecting the hashes to match.

[rmosolgo]

Hey, thanks for the detailed writeup. I definitely think we can find something that works better for this case.

Or maybe the whole POST body could be optionally dumped to a file so I could curl -d @body.raw to another server later on?

This definitely caught my ear. What sync does with the data it finds is already somewhat customizeable: https://github.com/rmosolgo/graphql-ruby/blob/dcaaed1cea47394fad61fceadf291ff3cb5f2932/javascript_client/src/sync/index.ts#L70

So, I'm thinking I could add a different function, say, dumpPayload, that writes the would-be POST body to a file instead of sending it to a server. But, one consideration is HTTP Headers: sendPayload currently calculates the necessary headers and includes them in the request. I think you'd need them to curl, but they couldn't go in the request body. Maybe it could dump a fully-formed curl request to a file, so you could run it from there? Or, is there another way that would make those headers readily available to you?

[mroach]

Is the only required header authorization? For our use case, the client secret is different for each environment, so we'd have to calculate it with the environment-specific keys each time it's used. The logic seems simple enough to replicate in any scripting language, even bash and using openssl sha256 -hex -mac HMAC -macopt hexkey:$key (I think?). Having the body dumped to a file would already be a huge help and I think it'd be able to fill in the rest on my own.

[rmosolgo]

Yes, that's it! (It takes custom headers too, but those would be just as easy to script another way.)

[rmosolgo]

👋 I just released graphql-ruby-client v1.14.2 which includes a --dump-payload option. You can pass a filename or, if you leave it blank, it will print the payload to stdout. Please give it a try and let me know what you think!

[mroach]

@rmosolgo It works! Thank you! :rocket: :tada:

In case anyone finds this thread and is looking for a quick way to do this in bash:

set -euo pipefail

bodyfile="$1"
client_name=frontend
client_secret=mysecret
endpoint=http://localhost:7001/graphql/frontend/sync
changeset=$(date +%F)

# --snip-- getopts to override the vars

function sha256hmac() {
  openssl sha256 -hex -mac HMAC -macopt key:$1 | awk '{print $2}'
}

digest=$(sha256hmac "${client_secret}" < "${bodyfile}")

curl "${endpoint}" \
  --silent \
  --show-error \
  --fail \
  --output /dev/null \
  --write-out '[HTTP %{http_code}]' \
  --data-binary @"${bodyfile}" \
  --header "authorization: GraphQL::Pro ${client_name} ${digest}" \
  --header "changeset-version: ${changeset}" \
  --header "content-type: application/json"

[mroach]

@rmosolgo It doesn't seem possible to dump the payload and generate the JS mapping file at once. Is that intentional?

I was thinking I'd be able to run the script as usual and add the --dump-payload option to get the payload dump as a side effect so I can sync it to other servers later. But, it seems that --dump-payload causes the sync to not happen (that's fine!) but also to not generate the JS map file wanted by --outfile.

I'd be fine with a two-step process where:

• Step 1: use --dump-payload and --outfile to generate the payload and JS mapping
• Step 2: use my "replay" script to actually upload the payload dump to the server

But, this doesn't work since --dump-payload and --outfile seem to be mutually exclusive options.

What I'd prefer to avoid is having to run the tool twice: once to actually sync it, once to only generate the payload body. That would result in a situation where the dumped payload is only "probably" what we sent to the server.

It's not a big deal since I can't think of a realistic way that the payloads would change between runs, but I wanted to ask in case it was intentional. :)

[mroach]

The more I think about it, the more I like the two-step process.

Until now for us, the operation sync has been a required step in the Docker image build process since that's the only way to get our JS mapping file to include when building the final JS for the app. It's a bit of an awkward dependency to require the backend operation store to be responding in order to build the frontend, especially when one image would sync with different backends in different environments. It's kind of like requiring database migrations to run while building an app image.

Now with this --dump-payload option I can eliminate that dependency and build Docker images any time without touching a backend, but it requires calling the script twice to generate the two files. Certainly not a big deal though :)

[rmosolgo]

Oops, that was definitely an oversight on my part. I must have missed a spot ... I'll take a look!

[rmosolgo]

I just shipped v1.14.5 with a fix for this so it should properly dump both at once. Please let me know if it gives you any more trouble!

[mroach]

Thank you for the fix here! It's working great now with one call to the utility and we're able to sync the operations to any number of backend servers in different environments now from the built images. Thanks! :rocket: