Build warning from webhook: Invalid event type update

[chrisworman-pela]

I have a Gatsby site that uses gatsby-source-datocms (2.3.0). I am attempting to integrate the site with Gatsby Cloud (GC). I have setup a web hook in Dato CMS that hits a GC URL (to kick-off a preview build) whenever a particular model's record is created, updated, or deleted. Here is a screenshot of the web hook in Dato CMS (with the GC URL redacted):

When I make a change to an applicable record (i.e. structed_page) in Dato CMS, the web hook correctly fires and GC initiates a build. When the build completes I see the following warning in the GC build logs:

Invalid event type update

Also, when the GC build completes, the changes made in Dato CMS to the record do not show up on the web site generated by the GC build.

I have tracked down the following file, which appears to be where the warning is being generated from:

• https://github.com/datocms/gatsby-source-datocms/blob/master/src/hooks/sourceNodes/index.js

Looking closer at this file, it appears that only create and delete operations are supported.

This suggests that the Dato CMS web hook is submitting an update event, but gatsby-source-datocms cannot handle such an event.

Does this make sense? Any help would be greatly appreciated :)

[chrisworman-pela]

Just tried creating a new record (structured_page) in Dato CMS and the GC build output the following warning:

Invalid event type create

And the new record does not show up on the web site generated by GC. Note that I have a local development server running (with gatsby-source-datocms live reloading enabled) and the site updates as expected with no warnings.

When I delete the newly created record in Dato CMS I get the following warning in the GC build logs:

Invalid event type delete

[chrisworman-pela]

Just noticed this is in the raw GC Preview Build logs:

13:31:19 PM: info Received update event for item undefined from DatoCMS

I suspect having "undefined" indicates something is wrong. Looking at the code, this suggests the Dato CMS web hook is not sending entity_id.

[chrisworman-pela]

I think the issue is that the default Dato CMS web hook body does not include entity_id, but gatsby-source-datocms requires it to always be there. Specifically, consider the following line of code from https://github.com/datocms/gatsby-source-datocms/blob/master/src/hooks/sourceNodes/index.js:

const { entity_id, entity_type, event_type } = webhookBody;

Now look at the Dato CMS documentation for web hooks (found here https://www.datocms.com/docs/general-concepts/webhooks#customize-the-http-payload), which shows the following as an example of the default web hook body:

{
  "entity_type": "item",
  "environment": "master",
  "event_type": "update",
  "entity": {
    "id": "293467",
    "type": "item",
    "attributes": {
      "created_at": "2018-05-22T10:13:00.461Z",
      "updated_at": "2018-08-02T10:54:09.616Z",
      "is_valid": true,
      "avatar": {
        "path": "/205/1526984443-irene.png",
        "format": "png",
        "size": 242630,
        "alt": null,
        "title": null,
        "width": 329,
        "height": 286
      },
      "name": "Irene Oppo",
      "gallery": []
    },
    "relationships": {
      "item_type": {
        "data": {
          "id": "1423",
          "type": "item_type"
        }
      },
      "published_version": {
        "data": {
          "id": "716593",
          "type": "item_version"
        }
      },
      "current_version": {
        "data": {
          "id": "716593",
          "type": "item_version"
        }
      }
    }
  }
}

Furthermore, it seems like the default web hook body can't work with the gatsby-source-datocms since if entity_type === 'item' then only publish and unpublish events are supported:

switch (entity_type) {
      case 'item':
        if (event_type === 'publish') {
          const payload = await client.items.all(
            {
              'filter[ids]': [entity_id].join(','),
              version: 'published',
            },
            { deserializeResponse: false, allPages: true },
          );
          if (payload) {
            loader.entitiesRepo.upsertEntities(payload);
          }
        } else if (event_type === 'unpublish') {
          loader.entitiesRepo.destroyEntities('item', [entity_id]);
        } else {
          reporter.warn(`Invalid event type ${event_type}`);
        }
        break;

I'm really confused how the Dato CMS web hook can be used with gatsby-source-datocms.

[matjack1]

Hello @chrisworman-pela

yes, you are right, you should send a custom payload from DatoCMS:

{
  "message": "{{event_type}} event triggered on {{entity_type}}!",
  "entity_id": "{{#entity}}{{id}}{{/entity}}",
  "entity_type": "{{entity_type}}",
  "event_type": "{{event_type}}"
}

Normally the hooks are managed by Gatsby cloud, but we should add some documentation about that! I'll do that as soon as possible, meanwhile let me know if that works for you!

[chrisworman-pela]

@matjack1 Creating a custom HTTP body solves the "missing entity_id" problem, but it does not solve the "no create, delete, or update events supported for entity_type === 'item'" problem. Here are the details:

When I create a custom body as you described above, and then update (resp. create, delete) (but not publish) a record in Dato CMS, the web hook now produces the following HTTP body:

{
  "message": "update event triggered on item!",
  "entity_id": "5740668",
  "entity_type": "item",
  "event_type": "update"
}

Notice that entity_type === 'item' and event_type === 'update'. Now consider the following code in gatsby-source-datocms that handles the web hook:

switch (entity_type) {
      case 'item':
        if (event_type === 'publish') {
          const payload = await client.items.all(
            {
              'filter[ids]': [entity_id].join(','),
              version: 'published',
            },
            { deserializeResponse: false, allPages: true },
          );
          if (payload) {
            loader.entitiesRepo.upsertEntities(payload);
          }
        } else if (event_type === 'unpublish') {
          loader.entitiesRepo.destroyEntities('item', [entity_id]);
        } else {
          reporter.warn(`Invalid event type ${event_type}`);
        }
        break;

This code does not support create, update, delete for entity_type === 'item' and will output an appropriate warning. This is confirmed by the Preview Build logs on Gatsby cloud:

[matjack1]

@chrisworman-pela first of all, this is the settings you should have in your webhook:

it should be set up automatically by Gatsby Cloud, but if you are setting things up manually you might do that.

Then, the code you are looking is about incremental builds, so it should show up in the deploy tabs. In that case we issue also a publish/unpublish when saving/deleting records that are not on the draft/published system, so it should work as expected (works for me right now).

Instead, the update should not trigger a build, but in that case you should be able to see the results in the preview instance of Gatsby cloud.

Does that make sense?

[chrisworman-pela]

@matjack1 I'm starting to get the sense that the Dato CMS and Gatsby Cloud integration does not support the "Publish" workflow in Dato CMS, or at least it does not support how we want to use the Publish workflow. Specifically, we want to have the following flow:

• Create/Update/Delete a Record in Dato CMS -> Preview (trigger GC Preview build) -> Publish in Dato CMS -> Deploy (trigger GC Production build)

In a nutshell, I am trying to use the Gatsby Cloud Preview feature to preview Unpublished records in Dato CMS before the production build is triggered and deployed. I thought this was the entire point of the Publish feature in Dato CMS, but I must be wrong.

Also important to note the previewMode flag supported by gatsby-source-datocms. This allows builds to include, Unpublished Dato CMS records in the build, so the Dato CMS team seems to have anticipated a "Create > Preview > Publish > Deploy" workflow, no?

To be clear, this is the workflow I am trying to support:

1. A user creates (or updates or deletes) a Record in Dato CMS, which triggers a GC Preview build (to a "preview" environment).
2. The user can Preview the change using the Gatsby Cloud Preview Build.
3. If the user approves of how their changes look in the Preview build, then the user can then Publish the change in Dato CMS, which triggers a GC Deploy build (to production).

I'm starting to get the sense that this is not possible using Dato CMS and Gatsby Cloud.

Matteo, can you please confirm (or clarify): Can unpublished Dato CMS records be viewed in the Gatsby Cloud Preview environment?

[matjack1]

@chrisworman-pela yes, unpublished DatoCMS records can be viewed in Gatsby Cloud Preview. It's the whole point, as you said :+1:

Are you checking the preview instance from here:

in Gatsby Cloud?

That one should update every time you save a record, even in draft. That is completely separate from the preview deploys, as those are the "incremental builds" and pulling only published content.

If this is still not working/convincing, please book a call here: https://calendly.com/datocms-support/demo so that we'll figure out things together!

[chrisworman-pela]

@matjack1 Thanks so much for your quick reply. I'm glad to hear that this is possible. I will try your configuration and get back to you. The reason I thought it would not work is because in your screenshot for the Dato CMS web hook configuration, there is no web hook setup to trigger on Record Create, Update, or Delete so I don't understand how newly created/updated/deleted records are being "sent" to Gastby Cloud.

[chrisworman-pela]

@matjack1 Just tried what I think is the Dato CMS configuration you've described above, but GC Preview is NOT updating when I "Save" (but not publish) records in Dato CMS. I'm not surprised by this, since there is no web hook setup that sends "create/update/delete" events for "records". Or perhaps that is not what is supposed to trigger the GC Preview to update? I must be missing something Matteo. If you don't reply today, I'll try to book something in your calendar.

[chrisworman-pela]

@matjack1 Also to confirm your question above, yes I am looking at "Preview Builds" in GC. I am not confusing them with "Deploy Previews".

[chrisworman-pela]

@matjack1 I have found the problem!!!

The issue I was having had nothing to do with web hooks and now I can see why my concerns about no "Record Create/Update/Delete" web hook was way off.

I was missing the disableLiveReload: false option on the gatsby-source-datocms plugin in gatsby-config.js for the Gatsby Cloud Preview environment.

Note that no one at Gatsby Cloud or Dato CMS mentioned this during all my conversations. Furthermore, the web hook configuration had to be incrementally squeezed out of (the very helpful) Matteo in this Github thread.

I really think there needs to be better documentation and/or support for setting up the Dato CMS / Gastby Cloud integration. The solution was soooo simple, but I had to go through all this to figure it out!

[matjack1]

Thank you for the feedback @chrisworman-pela

I didn't think about that, sorry! That is normally the default and that's what we are advertising in the README. But you are right, we are going to improve the documentation around that integration.

Thank you so much for all the feedbacks.