Partial updates only working partially

[CollierCZ]

I have an index I'd like to update only when the underlying files change. Right now, I'm using a modified field that is the last time the file was modified (gitLogLatestDate). This decreases the amount of updates that get done but does not eliminate them(currently: [insert/update: 1503, total: 8573]).

We're planning on adding more files, but if we keep having these updates, we'll run over our limits. Does anyone have any insights into how we could get the number to 0 when the files haven't changed? Thanks.

[Haroenv]

Could you show your app in a repo please? I don't fully understand how you've set it up now :)

[CollierCZ]

In gatsby-config:

 {
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: process.env.GATSBY_ALGOLIA_APP_ID,
        apiKey: process.env.ALGOLIA_WRITE_KEY,
        queries,
        chunkSize: 10000, // default: 1000,
        enablePartialUpdates: true,
        matchFields: ["modified"],
      },
    },

Then queries is defined in a separate (complicated) file, including:

return records.filter(Boolean).map((record) => {
    return {
      ...record,
      hierarchy,
      pageLink: slug,
      modified: parent.fields.gitLogLatestDate,
    }
  })
}

const settings = {
...
}

const queries = [
  {
    query: docsQuery,
    transformer: ({ data }) => data.pages.nodes.flatMap(separateRecords),
    indexName: `gatsby-docs`,
    settings,
  },
]

module.exports = queries

[Haroenv]

Sorry @CollierCZ, but the reason I asked for a specific full reproduction, is because I want to see what values are set to "modified"

[CollierCZ]

It's there. It's a datetime string. It comes from gatsby-transformer-gitinfo

const docsQuery = `{
  pages: allMdx {
    ...
      parent {
        ... on File {
          fields {
            gitLogLatestDate
          }
        }
      }
    }
  }
}`

Example: 2020-04-17 16:38:06 +0200

[CollierCZ]

I found out that the number of records in the Algolia UI is different. This plugin gives me total: 8638, while the UI shows me 7,102. So I might be trying to add duplicate records or something. I'll have to check my code again.

[CollierCZ]

Yeah, seems like that was probably the problem. Sorry for the trouble.

[Haroenv]

no problem! Too bad I wasn't able to help you more detailed, in the future an example that can be run will help