DOM to Semantic Markdown

This library converts HTML DOM to a semantic Markdown format optimized for use with Large Language Models (LLMs). It preserves the semantic structure of web content, extracts essential metadata, and reduces token usage compared to raw HTML, making it easier for LLMs to understand and process information.

Key Features

• Semantic Structure Preservation: Retains the meaning of HTML elements like <header>, <footer>, <nav>, and more.
• Metadata Extraction: Captures important metadata such as title, description, keywords, Open Graph tags, Twitter Card tags, and JSON-LD data.
• Token Efficiency: Optimizes for token usage through URL refification and concise representation of content.
• Main Content Detection: Automatically identifies and extracts the primary content section of a webpage.
• Table Column Tracking: Adds unique identifiers to table columns, improving LLM's ability to correlate data across rows.

Special Feature Examples

Here are examples showcasing the library's special features using the CLI tool:

1. Simple Content Extraction:

npx d2m@latest -u https://xkcd.com

This command fetches https://xkcd.com and converts it to Markdown

2. Table Column Tracking:

npx d2m@latest -u https://softwareyoga.com/latency-numbers-everyone-should-know/ -t -e

This command fetches and converts the main content from to Markdown and adds unique identifiers to table columns, aiding LLMs in understanding table structure.

3. Metadata Extraction (Basic):

npx d2m@latest -u https://xkcd.com -meta basic

This command extracts basic metadata (title, description, keywords) and includes it in the Markdown output.

4. Metadata Extraction (Extended):

npx d2m@latest -u https://xkcd.com -meta extended

This command extracts extended metadata, including Open Graph, Twitter Card tags, and JSON-LD data, and includes it in the Markdown output.

Installation

Using npm

npm install dom-to-semantic-markdown

Using npx (CLI)

> npx d2m@latest -h
Usage: d2m [options]

Convert DOM to Semantic Markdown

Options:
  -V, --version                                      output the version number
  -i, --input <file>                                 Input HTML file
  -o, --output <file>                                Output Markdown file
  -e, --extract-main                                 Extract main content
  -u, --url <url>                                    URL to fetch HTML content from
  -t, --track-table-columns                          Enable table column tracking for improved LLM data correlation
  -meta, --include-meta-data <"basic" | "extended">  Include metadata extracted from the HTML head
  -h, --help                                         display help for command

Usage

Browser

import {convertHtmlToMarkdown} from 'dom-to-semantic-markdown';

const markdown = convertHtmlToMarkdown(document.body);
console.log(markdown);

Node.js

import {convertHtmlToMarkdown} from 'dom-to-semantic-markdown';
import {JSDOM} from 'jsdom';

const html = '<h1>Hello, World!</h1><p>This is a <strong>test</strong>.</p>';
const dom = new JSDOM(html);
const markdown = convertHtmlToMarkdown(html, {overrideDOMParser: new dom.window.DOMParser()});
console.log(markdown);

CLI

d2m -i input.html -o output.md # Convert input.html to output.md
d2m -u https://example.com -o output.md # Fetch and convert a webpage to Markdown
d2m -i input.html -e # Extract main content from input.html
d2m -i input.html -t # Enable table column tracking
d2m -i input.html -meta basic # Include basic metadata
d2m -i input.html -meta extended # Include extended metadata

API

convertHtmlToMarkdown(html: string, options?: ConversionOptions): string

Converts an HTML string to semantic Markdown.

convertElementToMarkdown(element: Element, options?: ConversionOptions): string

Converts an HTML Element to semantic Markdown.

ConversionOptions

• websiteDomain?: string: The domain of the website being converted.
• extractMainContent?: boolean: Whether to extract only the main content of the page.
• refifyUrls?: boolean: Whether to convert URLs to reference-style links.
• debug?: boolean: Enable debug logging.
• overrideDOMParser?: DOMParser: Custom DOMParser for Node.js environments.
• enableTableColumnTracking?: boolean: Adds unique identifiers to table columns.
• overrideElementProcessing?: (element: Element, options: ConversionOptions, indentLevel: number) => SemanticMarkdownAST[] | undefined: Custom processing for HTML elements.
• processUnhandledElement?: (element: Element, options: ConversionOptions, indentLevel: number) => SemanticMarkdownAST[] | undefined: Handler for unknown HTML elements.
• overrideNodeRenderer?: (node: SemanticMarkdownAST, options: ConversionOptions, indentLevel: number) => string | undefined: Custom renderer for AST nodes.
• renderCustomNode?: (node: CustomNode, options: ConversionOptions, indentLevel: number) => string | undefined: Renderer for custom AST nodes.
• includeMetaData?: 'basic' | 'extended': Controls whether to include metadata extracted from the HTML head.
  • 'basic': Includes standard meta tags like title, description, and keywords.
  • 'extended': Includes basic meta tags, Open Graph tags, Twitter Card tags, and JSON-LD data.

Using the Output with LLMs

The semantic Markdown produced by this library is optimized for use with Large Language Models (LLMs). To use it effectively:

1. Extract the Markdown content using the library.
2. Start with a brief instruction or context for the LLM.
3. Wrap the extracted Markdown in triple backticks (```).
4. Follow the Markdown with your question or prompt.

Example:

The following is a semantic Markdown representation of a webpage. Please analyze its content:

```markdown
{paste your extracted markdown here}

{your question, e.g., "What are the main points discussed in this article?"}

This format helps the LLM understand its task and the context of the content, enabling more accurate and relevant responses to your questions.

## Contributing

Contributions are welcome! See the [CONTRIBUTING.md](CONTRIBUTING.md) file for details.

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.