Algolia records deleted on every build/deploy with or without enablePartialUpdates

[wildpow]

If we have Algolia records they will get deleted on a production build on Netlify. My current workaround has been to run a local build to re-add records to the index.

Dependencies:

    "gatsby-plugin-algolia": "^0.14.1",
    "gatsby": "^2.24.91",

gatsby-config

 {
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: process.env.GATSBY_ALGOLIA_APP_ID,
        apiKey: process.env.GATSBY_ALGOLIA_ADMIN,
        queries,
      },
    },

Also tried adding:

     enablePartialUpdates: true,
     matchFields: ["slug", "modified", "description"],

algolia-queries

function pillowToAlgoliaRecord({ shopifyId, handle, title, vendor, ...rest }) {
  return {
    objectID: shopifyId,
    slug: `/accessories/${handle}`,
    title,
    productType: "pillows",
    brand: vendor,
    ...rest,
  };
}
function adjustableToAlgoliaRecord({ id, slug, fullName, brand, ...rest }) {
  return {
    objectID: id,
    slug: `/adjustable/${slug}`,
    title: fullName,
    productType: "adjustable base",
    brand,
    ...rest,
  };
}
function mattressesToAlgoliaRecord({
  id,
  slug,
  name,
  brand,
  subline,
  ...rest
}) {
  return {
    objectID: id,
    slug: `/brands/${brand.urlName}/${slug}`,
    title: name,
    productType: "mattresses",
    brand: brand.displayName,
    subline: subline.name,
    ...rest,
  };
}
const queries = [
  {
    query: mattress,
    transformer: ({ data }) =>
      data.mattresses.nodes.map(mattressesToAlgoliaRecord),
    indexName,
    settings: {
      attributesToSnippet: [`description:20`],
      searchableAttributes: [`brand`, `description`, `productType`, "title"],
    },
  },
  {
    query: adjustables,
    transformer: ({ data }) =>
      data.adjustable.nodes.map(adjustableToAlgoliaRecord),
    indexName,
    settings: {
      attributesToSnippet: [`description:20`],
      searchableAttributes: [`brand`, `description`, `productType`, "title"],
    },
  },
  {
    query: pillows,
    transformer: ({ data }) => data.pillow.nodes.map(pillowToAlgoliaRecord),
    indexName,
    settings: {
      attributesToSnippet: [`description:20`],
      searchableAttributes: [`brand`, `description`, `productType`, "title"],
    },
  },
  {
    query: sheets,
    transformer: ({ data }) => data.sheets.nodes.map(sheetsToAlgoliaRecord),
    indexName,
    settings: {
      attributesToSnippet: [`description:20`],
      searchableAttributes: [`brand`, `description`, `productType`, "title"],
    },
  },
  {
    query: protector,
    transformer: ({ data }) =>
      data.protector.nodes.map(protectorToAlgoliaRecord),
    indexName,
    settings: {
      attributesToSnippet: [`description:20`],
      searchableAttributes: [`brand`, `description`, `productType`, "title"],
    },
  },
];
module.exports = queries;

[Haroenv]

Hmm, this is confusing! Do you have a reproduction on GitHub or somewhere where I can run it fully? The Shopify credentials don't need to be real ones, but I should be able to run it. Does it also fail with out of the box Gatsby plugins?

[wildpow]

This project can be found on Github. The problem is how to reproduce since the data comes from its headless CMS and their Shopify account. Copying some data manually into a new Gatsby repo to reproduce this issue will take me some time, maybe the middle of next month. Also, not sure what you mean by "fail without the box Gatsby plugins."

[Haroenv]

I meant "fail with out of the box Gatsby plugins". If you could get this to misbehave as you describe with just plugins that ship with Gatsby, without requiring cms credentials etc. Thanks!

[wildpow]

Posting this screenshot of our Netlify build just to keep the issue active and maybe could be helpful. I tried removing all our Netlify plug-ins to see if this has something to do with how we are deploying but, still have the same issue. I'm setup up a new project to try and reproduce this and hopefully, I'll have some temporary API keys to go with it.

[Haroenv]

Ah wait, I only just now see that you're doing multiple queries to the same index. #103 is the first PR that works on that use case; do you think you could try it out? cc @prichey

[wildpow]

Can I install this like yarn add gatsby-plugin-algolia@prichey or do I need to do something like this example?

[prichey]

@wildpow You should be able to update your package.json with the following:

"gatsby-plugin-algolia": "https://github.com/prichey/gatsby-plugin-algolia.git"

and run npm install. Let me know if you have issues with this approach

[prichey]

@wildpow Alternatively, you could use patch-package, which would involve modifying the gatsby-node.js file in your node_modules/gatsby-plugin-algolia directory to match the file in my branch

[wildpow]

I'll give it a shot and let you know.

[wildpow]

I ran a local build with no loss of records. @prichey You are a lifesaver.

sidenote: I was watching the Algolia admin panel and it looked like records were duplicated (300 total) then the copies were deleted (back down to 150) in about a 5-second time frame. Not sure if that's normal but, I have not tested enablePartialUpdates yet.

[wildpow]

@prichey Do you have any insight into when this PR will be merged?

[prichey]

@wildpow Glad to hear! I'm not a maintainer of the repo so I have no idea. Hopefully it's close to being ready, especially with everything working for you. I've been running my fork in production for weeks but hoping to get it merged ASAP

[wildpow]

I need to get this in production right now too. Every time a content creator of stakeholder deploys a build I have to run another build to get back our records which is crushing our Netlify build minutes.

Thank you again

[Haroenv]

I've merged that PR now 👍 and it's released as 0.15.0, thanks both of you!