algolia / gatsby-plugin-algolia

A plugin to push to Algolia based on graphQl queries
https://yarn.pm/gatsby-plugin-algolia
Apache License 2.0
177 stars 45 forks source link

Freezing builds on netlify #20

Closed janosh closed 2 years ago

janosh commented 5 years ago

Since adding gatsby-plugin-algolia to some of my projects, they have been suffering from freezing builds on netlify. The build logs below show that the failing step is

8:20:30 PM: Algolia: query 1: moving copied index to main index

This happens on every 4th or 5th deploy, I'd say. Interestingly, so far this has never happened when building locally. So it's very likely related to netlify in some way.

@Haroenv Can you tell from the logs what exactly could be causing this issue and how we might fix it?

8:20:23 PM: Algolia: 3 queries to index
8:20:24 PM: Algolia: query 2: copying existing index
8:20:24 PM: Algolia: query 0: copying existing index
8:20:24 PM: Algolia: query 1: copying existing index
8:20:26 PM: Algolia: query 2: executing query
8:20:26 PM: Algolia: query 2: splitting in 1 jobs
8:20:26 PM: Algolia: query 0: executing query
8:20:26 PM: Algolia: query 0: splitting in 1 jobs
8:20:28 PM: Algolia: query 1: executing query
8:20:28 PM: Algolia: query 1: splitting in 1 jobs
8:20:29 PM: Algolia: query 0: moving copied index to main index
8:20:30 PM: Algolia: query 1: moving copied index to main index
8:49:28 PM: error UNHANDLED EXCEPTION
8:49:29 PM: 
8:49:29 PM:   TypeError: Cannot read property 'id' of undefined
8:49:29 PM:   
8:49:29 PM: Build exceeded maximum allowed runtime
8:49:29 PM:   - nodes.js:21 module.exports
8:49:29 PM:     [repo]/[gatsby]/dist/redux/reducers/nodes.js:21:37
8:49:29 PM:   
8:49:29 PM:   - redux.js:451 combination
8:49:29 PM:     [repo]/[redux]/lib/redux.js:451:29
8:49:29 PM:   
8:49:29 PM:   - redux.js:211 dispatch
8:49:29 PM:     [repo]/[redux]/lib/redux.js:211:22
8:49:29 PM:   
8:49:29 PM:   - index.js:79 action
8:49:29 PM:     [repo]/[gatsby]/dist/redux/index.js:79:91
8:49:29 PM:   
8:49:29 PM:   - redux.js:468 Object.deleteNode
8:49:29 PM:     [repo]/[redux]/lib/redux.js:468:12
8:49:29 PM:   
8:49:29 PM:   - gatsby-node.js:150 emitter.on.action
8:49:29 PM:     [repo]/[gatsby]/dist/internal-plugins/internal-data-bridge/gatsby-node.js:15    0:23
8:49:29 PM:   
8:49:29 PM:   - mitt.js:1 
8:49:29 PM:     [repo]/[mitt]/dist/mitt.js:1:268
8:49:29 PM:   
8:49:29 PM:   - Array.map
8:49:29 PM:   
8:49:29 PM:   - mitt.js:1 Object.emit
8:49:29 PM:     [repo]/[mitt]/dist/mitt.js:1:252
8:49:29 PM:   
8:49:29 PM:   - index.js:102 store.subscribe
8:49:29 PM:     [repo]/[gatsby]/dist/redux/index.js:102:11
8:49:29 PM:   
8:49:29 PM:   - redux.js:220 dispatch
8:49:29 PM:     [repo]/[redux]/lib/redux.js:220:7
8:49:29 PM:   
8:49:29 PM:   - index.js:79 action
8:49:29 PM:     [repo]/[gatsby]/dist/redux/index.js:79:91
8:49:29 PM:   
8:49:29 PM:   - redux.js:468 
8:49:29 PM:     [repo]/[redux]/lib/redux.js:468:12
8:49:29 PM:   
8:49:29 PM:   - api-runner-node.js:48 doubleBoundActionCreators.(anonymous function).args
8:49:29 PM:     [repo]/[gatsby]/dist/utils/api-runner-node.js:48:13
8:49:29 PM:   
8:49:29 PM:   - gatsby-node.js:86 
8:49:29 PM:     [repo]/[gatsby-plugin-page-creator]/gatsby-node.js:86:19
8:49:29 PM:   
8:49:29 PM:   - Map.forEach
8:49:29 PM:   
8:49:29 PM: 
8:49:29 PM: Execution timed out after 15m0s
8:49:29 PM: Error running command: Command did not finish within the time limit
8:49:29 PM: Failing build: Failed to build site
Haroenv commented 5 years ago

there seem to be two things, the maximum time exceeded, as well as an id of undefined. I don't exactly recognise both of those, but can you try to see which code in Gatsby throws that error?

janosh commented 5 years ago

The line that throws the error is

state.delete(action.payload.id);

in gatsby/dist/redux/reducers/nodes.js

"use strict";

module.exports = (state = new Map(), action) => {
  switch (action.type) {
    case `DELETE_CACHE`:
      return new Map();

    case `CREATE_NODE`:
      {
        state.set(action.payload.id, action.payload);
        return state;
      }

    case `ADD_FIELD_TO_NODE`:
    case `ADD_CHILD_NODE_TO_PARENT_NODE`:
      state.set(action.payload.id, action.payload);
      return state;

    case `DELETE_NODE`:
      {
        state.delete(action.payload.id);
        return state;
      }

    case `DELETE_NODES`:
      {
        action.payload.forEach(id => state.delete(id));
        return state;
      }

    default:
      return state;
  }
};
//# sourceMappingURL=nodes.js.map

Somehow gatsby-plugin-algolia seems to be creating actions of type DELETE_NODE without a payload.

Haroenv commented 5 years ago

That's really odd, since the plugin doesn't even create nodes, it only queries the graphql 🤔 I think the error is unrelated

janosh commented 5 years ago

Could it be gatsby-plugin-algolia conflicting with another package? Because these types of errors only started occuring after I started using it and netlify build always fails during Algolia: query [n]: moving copied index to main index.

Haroenv commented 5 years ago

that would seem quite unlikely. What are the other plugins you're using?

janosh commented 5 years ago

Another possibility just occured to me. One difference between my local gatsby builds and those running on netlify is that locally there's at most one build going on at a time.

If I immediately merge a pull request in one of my repos, however, netlify builds the deploy preview and the master branch simultaneously. Could it be that both processes trying to access my Algolia indices simultaneously is causing this problem?

Haroenv commented 5 years ago

That is possibly a source of a bug (you should probably give staging a separate index name), but it would surprise me if that throws an error like the one described, since that code isn't used in the plugin

kuworking commented 5 years ago

I'm seeing the same error, even though in my case it doesn't stop the compilation (in run develop, haven't tried build yet)

So in my case this error has to be unrelated to an "id" property of (in this case) my module.

*Ah, and I cannot see any consecuence of this error in the site

kuworking commented 5 years ago

For me, the error appears when this code is included. So if I eliminate the lines marked with >>>>> I don't see the error with the "id" property (code located in gatsby-node.esm.js)

exports.onCreatePage = ({ page, actions }) => {
  const { createPage, deletePage } = actions
  const oldPage = Object.assign({}, page)
  if (page.path !== `/`) page.path = page.path.replace(/\/$/, ``)
  if (/^.\d{4}\.\d{2}\.\d{2}\./.test(page.path)) page.path = '/' + page.path.split('.').pop()

>>>>>  if (page.path !== oldPage.path) {
>>>>>    deletePage(oldPage)
>>>>>    createPage(page)
>>>>>  }
}

Since this is not related to the plugin, I've submitted a question in gatsby

richhiggins commented 4 years ago

I'm seeing this too, and inconsistently too. for me it hangs like this -

image

never gets past that, I have to cancel and trigger a new deploy which may or may not have a different outcome 😕

janosh commented 4 years ago

@richhiggins For me the issue was definitely due to concurrent builds attempting to access or write to the same resources. It completely disappeared once I disabled netlify builds for feature branches and PRs, i.e. with changes to master the only thing that triggers a build. Have you checked whether that may be the case for you as well?

richhiggins commented 4 years ago

interesting... I wonder if in my case it's because I have 2 queries which the plugin appears to process asynchronously and they are accessing the same index.

dreamer01 commented 4 years ago

I am still facing the same issue, my build is stuck/freeze at Algolia Query Indexing and copying.

Has anybody found any solution or workaround for this?

Screenshot 2020-03-22 at 3 39 18 PM
dreamer01 commented 4 years ago

I am still facing the same issue, my build is stuck/freeze at Algolia Query Indexing and copying.

Has anybody found any solution or workaround for this?

Screenshot 2020-03-22 at 3 39 18 PM

The workaround I found is to clear your indexes from Algolia and then run the build.

richhiggins commented 4 years ago

I seem to have solved this by refactoring to a single algolia query. we also experience the issue on another project, my current theory there is the _tmp index is being written to concurrently causing issues. planning to rename that temp index with a time stamp to test that theory out

dreamer01 commented 4 years ago

I seem to have solved this by refactoring to a single algolia query.

You can refractor it to single query but how were you able to resolve it to multiple indexes, as query options include indexName also

If could help me with this here is how I am using my queries

const queries = [
  {
    query: resourceQuery,
    transformer: ({ data }) => data.gcms.resources.map(resource => resource),
    indexName: `resource`,
  },
  {
    query: blogQuery,
    transformer: ({ data }) => {
      return data.gcms.blogs.map(blog => blog)
    },
    indexName: `blog`,
  },
]
prichey commented 4 years ago

I'm seeing this as well, has anyone figured out a reliable solution?

Haroenv commented 4 years ago

If you fork the plugin and comment out the element where it actually saves objects to Algolia, does it still freeze? Does someone have an open source example which fails?

Haroenv commented 2 years ago

I can't see this being reproduced anymore. If you have this again, please open a new issue with reproduction, thanks!