Open jshingler opened 9 months ago
Do you mind sharing your code? I have tested with a test page where I created circular links between two pages, but it still works. You also might want to juggle some optional configs defined by Typescript.
Here's my page that acts as the root node: https://lizard-flax-e38.notion.site/Test-bc95525429244d169cad7f6725cb5b34?pvs=4
Running the scraper gives me the following result, without hanging:
{
"nodes": [
{
"id": "bc955254-2924-4d16-9cad-7f6725cb5b34",
"title": "Test",
"type": "page",
"spaceId": "fda5cc70-a0d3-4f12-9b33-2ec0c26ab6ca",
"parentId": "none",
"cc": 2
},
{
"id": "65786275-82a8-4c32-8bd7-11d49a3ff7c6",
"parentId": "bc955254-2924-4d16-9cad-7f6725cb5b34",
"spaceId": "fda5cc70-a0d3-4f12-9b33-2ec0c26ab6ca",
"type": "page",
"title": "A"
},
{
"id": "175b9592-7cf5-4452-be34-da896731654b",
"parentId": "bc955254-2924-4d16-9cad-7f6725cb5b34",
"spaceId": "fda5cc70-a0d3-4f12-9b33-2ec0c26ab6ca",
"type": "page",
"title": "B"
}
],
"links": [
{
"source": "65786275-82a8-4c32-8bd7-11d49a3ff7c6",
"target": "bc955254-2924-4d16-9cad-7f6725cb5b34"
},
{
"source": "65786275-82a8-4c32-8bd7-11d49a3ff7c6",
"target": "175b9592-7cf5-4452-be34-da896731654b"
},
{
"source": "175b9592-7cf5-4452-be34-da896731654b",
"target": "bc955254-2924-4d16-9cad-7f6725cb5b34"
}
],
"errors": []
}
I used the code as is from the info you gave me, .... the page id is the blog. And I used the project from the example scapper and replaced index.js with this code
import { NotionGraph } from "@graphcentral/notion-graph-scraper" import fs from "fs" import { NotionAPI } from "notion-client";
/**
example of how to use @graphcentral/notion-graph-scraper
*/
;(async () => {
const notionApi = new NotionAPI({
activeUser: process.env.NOTION_ACTIVE_USER,
authToken: process.env.NOTION_TOKEN_V2
})
const notionGraph = new NotionGraph({
maxDiscoverableNodes: 10,
maxDiscoverableNodesInOtherSpaces: 10,
verbose: true,
unofficialNotionAPI: notionApi
})
const graph = await notionGraph.buildGraphFromRootNode(
// Some random Japanese blog
'95fcfe03257541c5aaa21dd43bdbc381'
)
console.log(graph.nodes.length)
console.log(graph.links.length)
await new Promise((resolve, reject) => {
fs.writeFile(test0.json
, JSON.stringify(graph), (err) => {
if (err) reject(err)
else resolve(``)
})
});
process.exit(0) })()
@jshingler but you configured process.env.NOTION_ACTIVE_USER
and process.env.NOTION_TOKEN_V2
right?
what is the output when you run it?
and the japanese blog should be accessible without using the custom notionApi argument because it's already public
Yes I Configured the environment variables, .... I am not using the Japanese Blog, ... I just didn't want to expose my private page id
Here is a portion of the output, it just keeps repeating
[12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0 [12/4/2023, 9:44:24 PM] # current requests: 0 / # items in the queue: 0 [12/4/2023, 9:44:24 PM] # total requests sent: 0 [12/4/2023, 9:44:24 PM] # total successful requests: 0
Yeah if it keeps showing zeroes, it shouldn't be about circular links because there's no page to start with. There should be something about not being able to retrieve the information in the first place. Let me try with my own private pages and get back to you
Looking at this again tonight, ... here is the error message that shows up in test0.json.
d.map is not a function. in notion-utils/build/index.js
{"links":[],"nodes":[],"errors":[{"name":"TypeError","message":"d.map is not a function","stack":"TypeError: d.map is not a function\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4072)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)\n at s (file:///Users/jshingler/Archive/Workspaces/graphcentral/notion-scrape-example/node_modules/notion-utils/build/index.js:1:4308)"},{}]}
It works on VERY simple private pages, .... Seems to be React-Notion-X has issues with something on the more complex pages.
Hi, ... Got the Private Pages Info, ... Thanks.
Does the graph navigation handle Circular Links between pages.
When I run the app, ... it just runs and runs. It never finishes.