use MarkdownElementNodeParser independently

[Bboyjie]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

Hello, I am currently using MarkdownElementNodeParser and have encountered an issue. Although I am using Tongyi LLMs, they do not support it well. When I set llm to None, the system requires an api_key. Could someone please advise on how to use MarkdownElementNodeParser independently without needing an api_key ? Thank you very much for your help!

[dosubot[bot]]

Hey there, @Bboyjie! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this issue together!

You can use the MarkdownElementNodeParser independently without needing an actual LLM or api_key by using a mock LLM. Here's how you can set it up:

from llama_index.legacy.llms import MockLLM
from llama_index.legacy.node_parser.relational.markdown_element import (
    MarkdownElementNodeParser,
)

node_parser = MarkdownElementNodeParser(llm=MockLLM())

By using MockLLM(), you can bypass the requirement for an LLM or api_key, allowing the MarkdownElementNodeParser to function independently ^[1].

To continue talking to Dosu, mention @dosu.

[Bboyjie]

@dosu. Hello,

I am working with Markdown parsers and would like to understand the differences between MarkdownNodeParser and MarkdownElementNodeParser. Specifically, I am curious about the following:

What are the main differences between MarkdownNodeParser and MarkdownElementNodeParser? Is MarkdownElementNodeParser an extension or specialization of MarkdownNodeParser, or are they completely separate implementations? Given that I have Markdown text extracted from PDF documents, which of these parsers should I use, and how should I use them effectively? Any insights or guidance would be greatly appreciated!

Thank you very much for your help!

[dosubot[bot]]

The MarkdownNodeParser and MarkdownElementNodeParser serve different purposes in the LlamaIndex codebase:

1. MarkdownNodeParser:

   • It splits a document into nodes using custom Markdown splitting logic, primarily based on headers and code blocks.
   • It handles metadata by updating it according to the hierarchy of headers and manages code blocks to prevent incorrect splits.
   • It constructs TextNode objects from parsed sections, optionally including metadata, and can display progress for large documents ^[1][2].

2. MarkdownElementNodeParser:

   • It parses a markdown document into different types of nodes, such as text, code blocks, tables, and titles.
   • It specifically processes tables, converting "perfect tables" into pandas DataFrames and handling non-perfect tables as raw text.
   • It merges consecutive text elements into a single text node and provides asynchronous processing capabilities ^[3][4].

Regarding their relationship, there is no information indicating that MarkdownElementNodeParser is an extension or specialization of MarkdownNodeParser. They appear to be separate implementations designed for different parsing needs.

For Markdown text extracted from PDF documents, the choice between these parsers depends on your specific requirements:

• Use MarkdownNodeParser if you need to focus on splitting documents based on headers and managing metadata.
• Use MarkdownElementNodeParser if you need to handle various markdown elements like tables and require more detailed parsing of document structure.

Choose the parser that aligns best with your document structure and parsing needs.

To continue talking to Dosu, mention @dosu.

[Bboyjie]

@dosu. Hello,

I am working on parsing Markdown text that has been extracted from PDF documents with the goal of reconstructing the original document structure. Specifically, I want to split the Markdown text by headings to restore the chapter structure of the PDF. For each section corresponding to a heading, I want to extract different elements, such as tables, and process them accordingly. My aim is to parse the PDF at different levels of granularity and hierarchy.

Given this objective, how can I effectively utilize MarkdownNodeParser and MarkdownElementNodeParser to achieve this? Are there other methods or tools that would be more suitable for processing a list of documents (Documents)?

Any advice or guidance would be greatly appreciated!

Thank you very much for your help!

[dosubot[bot]]

To achieve your goal of reconstructing the original document structure from Markdown text extracted from PDFs, you can effectively utilize both MarkdownNodeParser and MarkdownElementNodeParser as follows:

1. MarkdownNodeParser:

   • Use this parser to split the Markdown text by headings. It maintains the hierarchy of sections by updating metadata with the current header and its level, ensuring that the document's structure is preserved when splitting into nodes ^[1].
   • This is particularly useful for restoring the chapter structure of the PDF, as it can handle nested headers and maintain the hierarchy of sections.

2. MarkdownElementNodeParser:

   • Use this parser to extract and process different elements within each section, such as tables. It provides methods for extracting tables and converting them into structured formats like Pandas DataFrames ^[2].
   • This parser is suitable for handling various Markdown elements and can be used to parse the document at different levels of granularity.

Approach:

• First, use MarkdownNodeParser to split the document into sections based on headings. This will help you organize the text into a hierarchical structure.
• Then, for each section, use MarkdownElementNodeParser to extract and process specific elements like tables, lists, and code blocks.

Additional Tools:

• If you need to process a list of documents, consider using a loop to iterate over each document and apply the parsers sequentially.
• Depending on your specific needs, you might also explore other libraries or tools that specialize in PDF to Markdown conversion or document structure analysis, but the combination of these two parsers should cover most of your requirements for Markdown processing.

This approach allows you to effectively parse and reconstruct the document structure while handling various Markdown elements.

To continue talking to Dosu, mention @dosu.

[Bboyjie]

@dosu. To achieve your goal of reconstructing the original document structure from Markdown text extracted from PDFs, you can effectively utilize both MarkdownNodeParser and MarkdownElementNodeParser，Give me some code examples。

[dosubot[bot]]

To reconstruct the original document structure from Markdown text extracted from PDFs using both MarkdownNodeParser and MarkdownElementNodeParser, you can follow these steps with the provided code examples:

1. Using MarkdownNodeParser to Split by Headings:

from llama_index.core.node_parser.file.markdown import MarkdownNodeParser

# Sample Markdown text
markdown_text = """
# Chapter 1
This is the introduction.

## Section 1.1
Details about section 1.1.

## Section 1.2
Details about section 1.2.

# Chapter 2
This is another chapter.
"""

# Initialize the MarkdownNodeParser
node_parser = MarkdownNodeParser()

# Parse the document into nodes
nodes = node_parser.get_nodes_from_node(markdown_text)

# Print the nodes
for node in nodes:
    print(f"Header: {node.metadata.get('Header_1', 'No Header')}")
    print(f"Content: {node.text}\n")

2. Using MarkdownElementNodeParser to Extract Elements:

from llama_index.legacy.node_parser.relational.markdown_element import MarkdownElementNodeParser

# Sample Markdown text with a table
markdown_text_with_table = """
# Chapter 1
This is the introduction.

| Column 1 | Column 2 |
|----------|----------|
| Value 1  | Value 2  |
| Value 3  | Value 4  |
"""

# Initialize the MarkdownElementNodeParser
element_parser = MarkdownElementNodeParser()

# Extract elements from the document
elements = element_parser.extract_elements(markdown_text_with_table)

# Print the extracted elements
for element in elements:
    if element['type'] == 'table':
        print("Table found:")
        print(element['content'])
    else:
        print(f"Text: {element['content']}\n")

Approach:

• First, use MarkdownNodeParser to split the document into sections based on headings. This will help you organize the text into a hierarchical structure.
• Then, for each section, use MarkdownElementNodeParser to extract and process specific elements like tables, lists, and code blocks.

These examples demonstrate how to use both parsers to handle different aspects of Markdown text processing, allowing you to reconstruct the document structure and extract detailed elements.

To continue talking to Dosu, mention @dosu.