Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.
notion extractor only retrieves the first page of many blocks, and the subsequent blocks are lost.
According to the introduction of Pagination in the Notion Developers document, when the number of Blocks contained in a Pagination exceeds 100, it is necessary to obtain them in a paginated manner to get the complete content of the Notion Page.
However, the acquisition method in notion_extractor.py can only successfully obtain the first blocks page of the Notion Page (up to 100). It is not difficult to find out from the Notion Developers document that the reason is that when calling https://api.notion.com/v1/blocks/{block_id}/children, the start_cursor of the next page is mistakenly passed as block_id, while in fact start_cursor is passed through the Query Params of the GET request.
In addition, the parameter transmission method of the Query Params of the GET request is also wrong (formal parameter: json -> params).
Fixes # (issue)
Type of Change
Please delete options that are not relevant.
[x] Bug fix (non-breaking change which fixes an issue)
How Has This Been Tested?
Find a longer Notion Page (with more than 100 Blocks) and perform the Sync from Notion operation in Knowledge to verify that the version after this PR can synchronize the complete Notion Page content, while the previous version can only obtain the content of the first 100 Blocks, and the other content is lost.
[ ] TODO
Suggested Checklist:
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[x] My changes generate no new warnings
[x] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
[ ] optional I have made corresponding changes to the documentation
[ ] optional I have added tests that prove my fix is effective or that my feature works
[x] optional New and existing unit tests pass locally with my changes
Description
notion extractoronly retrieves the first page of many blocks, and the subsequent blocks are lost.According to the introduction of Pagination in the Notion Developers document, when the number of Blocks contained in a Pagination exceeds 100, it is necessary to obtain them in a paginated manner to get the complete content of the Notion Page.
However, the acquisition method in
notion_extractor.pycan only successfully obtain the first blocks page of the Notion Page (up to 100). It is not difficult to find out from the Notion Developers document that the reason is that when callinghttps://api.notion.com/v1/blocks/{block_id}/children, the start_cursor of the next page is mistakenly passed asblock_id, while in factstart_cursoris passed through the Query Params of the GET request.In addition, the parameter transmission method of the Query Params of the GET request is also wrong (formal parameter: json -> params).
Fixes # (issue)
Type of Change
Please delete options that are not relevant.
How Has This Been Tested?
Find a longer Notion Page (with more than 100 Blocks) and perform the
Sync from Notionoperation inKnowledgeto verify that the version after this PR can synchronize the complete Notion Page content, while the previous version can only obtain the content of the first 100 Blocks, and the other content is lost.Suggested Checklist:
dev/reformat(backend) andcd web && npx lint-staged(frontend) to appease the lint godsoptionalI have made corresponding changes to the documentationoptionalI have added tests that prove my fix is effective or that my feature worksoptionalNew and existing unit tests pass locally with my changes