search

alvis / gatsby-source-notion

A Gatsby source plugin for content management with Notion using the official API.
MIT License
56 stars 6 forks source link

Will latests fixs will be embeded in a release ? #1

Closed emaulandi closed 3 years ago

emaulandi commented 3 years ago

Thank you for this great plugin !

I was wondering if there were any plans of releasing a new version of the npm package that will embed the latest fix that will persist unchanged nodes between two runs (if I understood it correctly) ?

alvis commented 3 years ago

Hi @emaulandi can you elaborate more about the problem you’re facing?

I’m planning to work on it again to support the recently and finally released image/file api. So I’m keen to fix any issues there.

emaulandi commented 3 years ago

My notion database have 48 pages/nodes that I want to access the data to display content and so on.

I think this is covered by the latest fix you commited on the repo ?

Let me know if the explanation is not clear or if you need further elements.

alvis commented 3 years ago

@emaulandi Thanks. It's very clear. I just release 1.0.2 together with a minor fix. Check it out and let us know if it resolves your issue.

emaulandi commented 3 years ago

Thanks a lot for your reactivity ! Just updated the package.

It seems that I run into the same problem where nodes that are still there is the cache but have no changes are not processed. There are present in the oldMap and newMap (node.js) but are not detected neither as new entities or as updated entities (oldMap.get(id) is true and oldEntity.digest !== newEntity.digest is false) and thus this.createNode is never called on them (if I understood node.js correctly, which might not be the case :) )

alvis commented 3 years ago

Let me check with my example repo. I encountered the exact same problem as you and therefore the fix. Let me get back to you by today. ;)

alvis commented 3 years ago

@emaulandi It turns out I got the usage of touchNode wrong. I thought nodes only need to be touched when they were first created, but it isn't. All nodes have to be touched at every run. I have released a newer version to address this issue. Please try and let me know :)

emaulandi commented 3 years ago

It works like a charm ! Thank you so much for this really fast update and help !

emaulandi commented 3 years ago

Sorry, I might not have checked this fully. After few redeploys, I found the same problem. From the plugin infos during build I can see that I have X nodes added and Y nodes processed (touchNode was apply on those Y nodes, Y being the total numbers of nodes). But then still only the X nodes appear in the graphql query allNotionPage { nodes { }}.

From your example repo, does all processed nodes (touched nodes) which where not added are available for query and use ?

alvis commented 3 years ago

It's an interesting found. I managed to reproduce your issue. If you don't have any pages on notion got updated, it works fine. However, if you get just a page updated and you're building with cache, some NotionPage nodes could disappear even they've been touched.

It should be caused by my understanding about touchNode. I tried to call it before or after createNode, but no luck. Some NotionPage nodes still get missing when the source on Notion get updated. Weird.

Get look into it more tomorrow.

alvis commented 3 years ago

@emaulandi It turns out that the bug is introduced unintentionally by including child nodes' content in the hash of a database node. Now I believe it's fixed. You can try the fix via the following and let me know if I can merge and release a newer version. :)

npm install alvis/gatsby-source-notion#pull/2/head
emaulandi commented 3 years ago

Thanks again for the reactivity !
I tested locally and running 2 consecutive deploys on Netlify and I think it works fine ! With no cache, all nodes are created and the following runs I can see [gatsby-source-notion] keeping [X being all] nodes, I have all the nodes in graphql and all nodes are used to build pages an so on. 🙏🙏🙏

alvis commented 3 years ago

@emaulandi Could you try updating some pages or adding new ones? In theory, it should show keeping [x for the number of unchanged nodes] nodes together with added/updated [y] nodes?

emaulandi commented 3 years ago

So I tried running multiple times gatsby develop locally with the following scenario :

Actions : Empty cache Run 1 / Status OK info [gatsby-source-notion] added 50 nodes info [gatsby-source-notion] keeping 0 nodes

Run 2 / Status OK info [gatsby-source-notion] keeping 50 nodes

Actions : Update 1 node in notion Run 3 / Status OK info [gatsby-source-notion] updated 1 nodes info [gatsby-source-notion] keeping 49 nodes

Actions :

Run 3 / Status KO (only two nodes available : the new and updated one) info [gatsby-source-notion] added 1 nodes info [gatsby-source-notion] updated 2 nodes info [gatsby-source-notion] keeping 48 nodes

Run 4 / Status KO (only two nodes available : the new and updated one) info [gatsby-source-notion] keeping 51 nodes

Actions : Delete 2 nodes Run 5 / Status KO There was an error in your GraphQL query: Cannot query field "allNotionPage" on type "Query". info [gatsby-source-notion] updated 1 nodes info [gatsby-source-notion] removed 2 nodes info [gatsby-source-notion] keeping 48 nodes

Actions : gatsby clean Run 6 / Status OK info [gatsby-source-notion] added 49 nodes info [gatsby-source-notion] keeping 0 nodes

alvis commented 3 years ago

Hi @emaulandi thanks so much for helping to test.

Sorry. My silly head. I forgot adding a new page would trigger an update to the database node too. It should be fixed with the latest verion.

npm install alvis/gatsby-source-notion#pull/2/head
emaulandi commented 3 years ago

No worries ! Thank you for you help.

It seems that adding a node is still problematic :

I also tried cleaning the cache after adding 2 dummy nodes, running a first time and then deleting the 2 nodes. In that case, no problem.

alvis commented 3 years ago

Thanks for testing. Just wondering if you've deleted node_modules and package-lock.json and reinstalled everything again before running the test? The info seems to be in the old format.

emaulandi commented 3 years ago

Yes, I did it again but I have the same result.

Regarding infos, I think you've deleted "processed" to be replace by "keeping" ? I can see the computeNodeGraph for example in in nodes_modules/gatsby-source-notion that was added in the last update I think ?

alvis commented 3 years ago

It turns out the root of issue is deeper than I thought. The problem is that whenever a database node need to get updated either by changing its title or adding a page to it, all its children will get deleted. Neither touchNode nor createParentChildLink has any effect at all.

In the official doc, it says

Because all children nodes are derived from their parent, when a parent node is deleted or changed, Gatsby deletes all of the child nodes (and their child nodes, and so on) with the expectation that they’ll be recreated again by transformer plugins. This is done to ensure there are no nodes left over that were derived from older versions of data but shouldn’t exist any longer.

It seems to be long standing problem of Gatsby, see

I'm not expecting any change from Gatsby to solve the issue, so I wrote an update to recreate any missing nodes automatically instead. The latest update can be installed via the following.

npm install alvis/gatsby-source-notion#c97d301
emaulandi commented 3 years ago

Thank you for the detailed explanations. With this update, when adding / deleting a node, all unchanged nodes are preserved.

alvis commented 3 years ago

@emaulandi The fix is now released as 1.0.4. Thanks for helping again :)