• org-zettel-ref-mode

  +begin_center

  [[file:readme_cn.org][中文版说明]]

  +end_center

** Changlog *** Version 0.4.3 (2024-11-08)

• Optimized:
  • The display method of the overview file window.
    • Added a configuration item to define the width of the overview window: ~org-zettel-ref-overview-width-ratio~, default 0.3
    • Added a configuration item to define the minimum width of the overview window: ~org-zettel-ref-overview-min-width~, default 30 *** Version 0.4.2 (2024-11-08)
• Fixed:
  • The error in org-zettel-ref-db-init #15
  • The issue where the cursor position is lost after executing org-zettel-ref-init
  • The issue where the overview file failed to synchronize correctly due to improper index file retrieval

*** Version 0.4.1 (2024-11-06)

• Optimized convert_to_pdf.py
  • Dropped using OCR to convert PDF

*** Version 0.4 (2024-11-04)

• Attention!
  • If you’ve previously used org-zettel-ref-mode, you need to run ~M-x org-zettel-ref-migrate~ the first time you use the new version to upgrade the data structure in the hash table.
• New Feature: Provides a visual management panel for source files
  • ~org-zettel-ref-list~ (see Basic Usage -> Manage Source Files for details):
    • Visualization: Provides a reference management panel
    • Multi-column list: Displays the current references in a list format, with key columns such as Title, Author, and Keywords
    • Rename: Allows renaming files in the format AUTHOR__TITLE==KEYWORDS.org within the panel
    • Sorting: Click on the column name to sort the list alphabetically
    • Filtering: Filter source file entries by conditions, such as Author, Title, or Keywords. Currently, only one condition can be filtered at a time.
• Upgraded the data structure of the hash table in ~org-zettel-ref-db.el~
• Upgraded ~org-zettel-ref-clean-multiple-targets~
• Fixes:
  • Restored the accidentally deleted custom configuration item ~org-zettel-ref-debug~
• Reminder:
  • Due to the upgrade of the hash table storing the mapping between source files and overview files to version 2.0, the following functions are deprecated:
    • org-zettel-ref-check-and-repair-links, org-zettel-ref-maintenance-menu, org-zettel-ref-refresh-index, org-zettel-ref-rescan-overview-files, org-zettel-ref-status. *** Version 0.3.3 Stable release (2024-09-28)
• Backend optimizations to further enhance code robustness, modularity, and improve plugin stability.
• Fixed an issue in version 0.3.2 where rapid updates to the overview file caused synchronization errors with quick notes and marked text.
• Fixed an issue in version 0.3.2 where the file naming strategy led to frequent re-creation of overview files in Denote mode.
• Fixed an issue in version 0.3.2 where the overview file failed to synchronize correctly due to improper index file retrieval.

After this period of development, the code for org-zettel-ref-mode has finally become modular and robust. No new features will be introduced before version 0.4. Instead, the focus will be on further componentization of the code and providing more customization options.

*** Version 0.3.2 (2024-09-24)

• Improved compatibility with Org-roam v2: Update the record of literature notes in the overview file to the Org-roam database.
• Improved file naming
• Fix a bugs that causes Emacs crash
• Refined code, modularized

*** Version 0.3.1 (2024-09-15)

• Compatible with Emacs 30 and later versions.
• Overview files now have more elegant names, reducing repetitive occurrences of the word "overview."
• Fixed an intermittent (setq, 5) error.
• Removed the dependency on conda.el in org-zettel-ref-mode.el, and the detection of the Python environment is now entirely handled by convert-to-org.py.
  • Automatically sets up a virtual environment via the python venv command and installs required libraries.
    • Note: After updating to this version, running convert-to-org.py will reinstall third-party libraries. If you prefer a clean environment, you may want to manage this manually.
• Improved synchronization mechanism for overview files, preventing multiple overview files from being created for the same source file. Also improved the robustness and stability of this feature.
  • A hash table is now used to map source files to overview files. One great thing is that you don’t need to manually set the hash table file location.
  • The overview file header now includes a new property block: ~#+SOURCE-FILE:~ to confirm the mapping.
  • New commands:
    • ~org-zettel-ref-check-and-repair-links~ - Check and repair links between source files and overview files.
    • ~org-zettel-ref-maintenance-menu~ - Display a menu for org-zettel-ref-mode maintenance operations.
    • ~org-zettel-ref-refresh-index~ - Manually refresh the overview index.
    • ~org-zettel-ref-rescan-overview-files~ - Rescan the overview directory and update the index.
    • ~org-zettel-ref-status~ - Display the current status of org-zettel-ref-mode.

*** Version 0.3 (2024-09-03)

• Improved integration with org-roam:
  • Added conditional loading and error-handling mechanisms for better stability
  • Optimized database operations for increased efficiency
  • Enhanced file handling for greater compatibility
  • Added a database status check feature for easier debugging
• Enhanced support for Conda environments:
  • Provided more flexible Python environment configuration options
  • Improved the initialization and activation process for Conda environments
• Refined logic for filename generation and processing:
  • Added a filename cleanup feature for greater robustness
  • Optimized file naming strategies across different modes
• Optimized overview file synchronization:
  • Implemented selective updates, only refreshing changed sections
  • Improved buffer handling to reduce file I/O operations
  • Enhanced content generation for increased efficiency
• Added debugging features:
  • Included detailed log output for easier troubleshooting
  • Provided more error messages and status check options ** Main Features Invoke the command M-x org-zettel-ref-init to open the "Overview Window," displaying the notes recorded in the original document and the marked text.

1. Each time an overview is generated, a literature note is automatically created and saved to a folder of your choice.
2. Quick note-taking: use M-x org-zettel-ref-add-quick-note to directly input notes.
3. When reviewing literature notes, you can jump directly back to the corresponding location in the original text to reread the context.
4. Offers a method to convert documents in other formats into org format.
5. Provides quick markup functionality to easily add bold, italic, underline, and other formatting to text in the source file.
6. Supports integration with knowledge management tools like org-roam and denote.
7. Flexible file association mechanisms that support multiple knowledge management modes (Normal, Denote, Org-roam).
8. Directly call external Python scripts from within Emacs to convert various document formats into org files. ** Demo As shown, the left window displays the original text, while the right window displays the overview.

[[file:demo/org-zettel-ref-mode-demo.png]]

** Applicable Scenarios org-zettel-ref-mode is only effective when org-mode is activated:

1. Directly targeting org files
2. Other user-defined text formats processed in org-mode, such as md, txt, etc. In these cases, the functionality of the major mode for those formats may be affected.

However, I generally convert materials directly into org format for saving, so the second scenario is rare.

** Value: A Reading Method That Balances Breadth and Depth

TL;DR Version:

• Simply saving, excerpting, or copying materials is not enough; information needs to be processed to be transformed into useful knowledge.
• The Zettelkasten method emphasizes summarizing/reviewing in your own words and establishing connections, providing multiple opportunities for information processing. However, many introductions overlook Luhmann's method of handling a large volume of literature notes.
• Literature notes are an efficient and in-depth method that records key points and inspirations, facilitating quick review and deep reading, while also helping distinguish between existing and new information.

Full Version:

As a longtime note-taking enthusiast and writer, I've gradually realized some "counterintuitive" insights:

• Simply saving is almost useless.
• Simply excerpting is almost useless.
• Simply copying is almost useless.

The reason is that merely transporting material only increases the amount of information without reprocessing it. Remember the classic hierarchy? Data -> Information -> Knowledge -> Wisdom.

The Zettelkasten method always emphasizes summarizing in your own words, frequently reviewing past notes, and increasing the connections between notes. From a methodological standpoint, it offers at least 4-7 opportunities for information processing.

Even so, the literature and videos introducing the Zettelkasten method often get caught up in the craze of double-linking, falling into the trap of merely saving data—essentially ignoring the method Niklas Luhmann used to handle a massive amount of literature notes.

Let me share a number: among the more than 90,000 index cards Luhmann left behind, over 10,000 were literature notes.

Luhmann's astounding productivity came from an exaggerated amount of data processing, and behind that was his efficiency in handling this data—achieved through the creation of literature notes.

Luhmann had a habit of taking literature notes while reading. His books or materials had no underlining, no margin notes, and were incredibly clean, almost as if they hadn't been read. Each literature note was essentially an index of the material. He only excerpted the original text from the book when absolutely necessary.

However, after understanding how researchers create literature notes, I discovered that Luhmann's literature notes are almost identical to standard research literature notes. They are also annotated in one's own words, while recording the specific location of inspiration in the paper, for future in-depth reading.

In other words, this method of taking literature notes balances efficiency and depth.

When it's unnecessary to deeply understand a material, literature notes can record key points (not the important content, but the insights useful to oneself). When a deep understanding is needed, the literature notes can quickly point to the corresponding context for in-depth reading and thinking, without wasting time re-reading from the beginning.

Besides balancing efficiency and depth, literature notes also have the advantage of easily distinguishing between existing and new information. Concepts or key points that have been annotated similarly before are existing information, and it is unnecessary to annotate them again when encountered in another material. Conversely, concepts, data, or ideas that have not been encountered before are worth annotating and recording their sources, making the discovery of new knowledge easier.

** Installation *** Installation Steps

1. Download the org-zettel-ref-mode.el file.
2. Place the file in your Emacs load path (e.g., ~/.emacs.d/lisp/).
3. Add the following to your Emacs configuration file (such as ~/.emacs or ~/.emacs.d/init.el):

Example Configuration 1:

+BEGIN_SRC emacs-lisp

(add-to-list 'load-path "~/.emacs.d/lisp/") (require 'org-zettel-ref-mode)

+END_SRC

Example Configuration 2:

+BEGIN_SRC emacs-lisp

(use-package org-zettel-ref-mode :ensure t ; If installed via package manager :init (setq org-zettel-ref-overview-directory "~/Documents/org-overviews/") :config (setq org-zettel-ref-quick-markup-key "C-c m"))

+END_SRC

Example Configuration 3 (Doom Emacs users):

In ~/.doom.d/packages.el:

+BEGIN_SRC emacs-lisp

(package! org-zettel-ref-mode :recipe (:host github :repo "username/org-zettel-ref-mode"))

+END_SRC

In ~/.doom.d/config.el:

+BEGIN_SRC emacs-lisp

(use-package! org-zettel-ref-mode :config (setq org-zettel-ref-overview-directory "~/Documents/org-overviews/"))

+END_SRC

4. Restart Emacs or evaluate the newly added code.

** Basic Usage

*** Activating the Mode In any org-mode buffer, run: M-x org-zettel-ref-init

*** Clean Up <<>> in Source Files

Since the core functionality of adding notes involves adding <<>> target links in the original text, many materials converted to org format come with a lot of <<>> text.

Before annotating or marking text in the org file for the first time, you can use org-zettel-ref-clean-targets to clean up the format and ensure the quick note feature works correctly.

*** Adding Quick Notes

1. Place the cursor where you want to add a note
2. M-x org-zettel-ref-add-quick-note
3. Enter the note name and content

*** Quick Markup

1. Select the text in the source file
2. M-x org-zettel-ref-quick-markup
3. Choose the markup style you prefer

*** Sync Overview Files Automatic sync by default: Automatically runs when saving the source file. Manual sync: M-x org-zettel-ref-sync-overview

*** Manage Source Files

1. Launch Panel

[[file:demo/org-zettel-ref-list.gif]]

~M-x org-zettel-ref-list~

Reminder: The following commands are all executed within the panel interface.

2. Rename Source File ("r")

[[file:demo/org-zettel-ref-list-rename-file.gif]]

~M-x org-zettel-ref-list-rename-file~

Rename according to the fixed format AUTHOR__TITLE==KEYWORDS.org.

3. Edit/Add Keywords ("k")

[[file:demo/org-zettel-ref-list-edit-keywords.gif]]

~M-x org-zettel-ref-list-edit-keywords~

Independently add one or more keywords to the source file.

4. Delete Source File

[[file:demo/org-zettel-ref-list-delete-file.gif]]

Delete a single file ("d") ~M-x org-zettel-ref-list-delete-file~

[[file:demo/org-zettel-ref-list-delete-marked-files.gif]]

Delete multiple files ("D") Press "m" in the list to mark multiple files, then execute ~M-x org-zettel-ref-list-delete-marked-files~

If the marked files are incorrect, press "u" to clear the marked status, and press "U" to clear all marked statuses.

5. Use Filters

[[file:demo/org-zettel-ref-list-filter-by-regexp.gif]]

Simple Filter ("/ r"): Use Author, Title, Keywords as filter conditions, only one filter condition can be applied at a time ~M-x org-zettel-ref-filter-by-regexp~

Complex Filter ("/ m"): Multiple filter conditions can be applied using Author, Title, Keywords as conditions

*** ⚠️ Caution

1. Do not casually change the filename of note files. If you do, adding quick notes/markups again in the source file will generate duplicate notes during sync. ** Advanced Features *** File Association Mechanism org-zettel-ref-mode now supports multiple file association mechanisms and no longer fully relies on the "-overview" suffix in filenames:

• Normal Mode: Still uses the "-overview" suffix (for backward compatibility).
• Denote Mode: Follows Denote's naming conventions.
• Org-roam Mode: Follows Org-roam's naming conventions and ID attributes.

If you're upgrading from an older version, your existing "-overview" files will still work. However, for new files, we recommend using the new association mechanisms.

*** Debugging in org-roam Mode The M-x org-zettel-ref-check-roam-db function checks the status of the org-roam database.

*** Overview File Sync Mechanism Optimization

In version 0.3, org-zettel-ref-mode made significant optimizations to the overview file synchronization mechanism:

1. Selective Updates: Only updates the changed sections (Quick Notes and Marked Text) instead of rewriting the entire file.
2. Smart Buffer Handling: Prioritizes using open buffers, reducing file I/O operations.
3. Content Generation Optimization: Removed redundant content generation steps.

These optimizations provide the following benefits:

• Significantly improved sync speed, especially for large overview files.
• Reduced resource usage, including memory and disk I/O.
• Enhanced stability, reducing the risk of data loss.
• Improved user experience with less interface flickering.

For users who frequently use the overview feature, these optimizations will bring noticeable performance improvements. *** Custom Note Saving Modes (Updated 2024-08-29) org-zettel-ref-mode provides three modes: normal, org-roam, and denote, allowing note files to be saved in the corresponding format. For example, after selecting org-roam mode, the saved note files will automatically include an ID, making them easier to retrieve.

Configuration Method:

=(setq org-zettel-ref-mode-type 'normal) ; Options: 'normal, 'denote, 'org-roam)=

*** Custom Overview File Location

+BEGIN_SRC emacs-lisp

(setq org-zettel-ref-overview-directory "~/my-notes/overviews/")

+END_SRC

*** Adjusting Auto-Sync Behavior Disable Auto-Sync:

+BEGIN_SRC emacs-lisp

(org-zettel-ref-disable-auto-sync)

+END_SRC

Enable Auto-Sync:

+BEGIN_SRC emacs-lisp

(org-zettel-ref-enable-auto-sync)

+END_SRC

*** Enabling Debug Mode If you encounter issues during use, you can enable debug mode to get more information:

+BEGIN_SRC emacs-lisp

(setq org-zettel-ref-debug t)

+END_SRC

*** Using Scripts to Convert Documents in PDF, ePub, HTML, MD, TXT Formats to Org Files

[[file:demo/pkm-system-diagram.png]]

Script: [[file:convert-to-org.py]]

org-zettel-ref-mode now supports directly calling external Python scripts from within Emacs to convert various document formats into org files.

**** Key Features

1. Multi-format Support:

   • Supports converting PDF, EPUB, HTML, Markdown, and TXT formats to Org format.
   • Can handle both electronic and scanned PDFs, supporting mixed Chinese and English documents.

2. OCR Functionality:

   • Uses OCR technology to process scanned PDFs, supporting Chinese and English recognition.

3. File Management:

   • Automatically checks file size to prevent processing overly large files.
   • After conversion, it can automatically archive the source file.

4. Flexible Configuration:

   • Supports custom paths for temporary files, reference materials, and archives.
   • You can choose to use the system Python, Conda environment, or virtual environment.

**** Usage Instructions

1. Configure Python Environment:

   +BEGIN_SRC emacs-lisp

   (setq org-zettel-ref-python-environment 'conda) ; or 'system, 'venv (setq org-zettel-ref-python-env-name "your-env-name") ; If using Conda or venv

   +END_SRC

2. Set Script Path and Folders:

   +BEGIN_SRC emacs-lisp

   (setq org-zettel-ref-python-file "~/path/to/document_convert_to_org.py") (setq org-zettel-ref-temp-folder "~/Documents/temp_convert/") (setq org-zettel-ref-reference-folder "~/Documents/ref/") (setq org-zettel-ref-archive-folder "/Volumes/Collect/archives/")

   +END_SRC

3. Run Conversion Script: Use the command M-x org-zettel-ref-run-python-script to execute the conversion.

**** ⚠️ Caution

• Ensure that all necessary Python libraries (e.g., PyPDF2, pdf2image, pytesseract, etc.) are installed.
• For scanned PDFs, the conversion process may be slow, and the results may not be as good as for electronic versions.
• It's recommended to use this script primarily for converting electronic PDFs, EPUB, Markdown, and TXT documents.

**** Workflow Recommendations

1. Use a browser extension (e.g., Markdownload) to save web pages as Markdown files.
2. Use org-zettel-ref-mode's Python script to convert Markdown files to Org format.
3. For audio files, you can first convert them to text using Whisper and then use the script to convert them to Org format.

This feature significantly expands the application range of org-zettel-ref-mode, making it a more comprehensive knowledge management tool. **** ⚠️ Caution It is recommended to use this script for converting ePub, markdown, txt, and electronic PDF documents.

It is not recommended to use this script to convert scanned PDFs due to slow conversion speed and suboptimal conversion quality.

** Available Commands

Here are the main commands provided by org-zettel-ref-mode:

• M-x org-zettel-ref-init: Initialize org-zettel-ref-mode, create or open an overview file
• M-x org-zettel-ref-add-quick-note: Add a quick note at the current position
• M-x org-zettel-ref-sync-overview: Manually sync the overview file
• M-x org-zettel-ref-quick-markup: Quickly add markup to selected text
• M-x org-zettel-ref-clean-multiple-targets: Clean up excess targets in the source file
• M-x org-zettel-ref-enable-auto-sync: Enable auto-sync
• M-x org-zettel-ref-disable-auto-sync: Disable auto-sync
• M-x org-zettel-ref-check-roam-db: Check org-roam database status
• M-x org-zettel-ref-run-python-script: Run the specified Python script

** Configurable Variables

Here are the main configurable variables for org-zettel-ref-mode:

• setq org-zettel-ref-overview-directory "~/org-zettel-ref-overviews/": Set the overview file storage directory
• setq org-zettel-ref-mode-type 'normal: Set the mode type (options: 'normal, 'denote, 'org-roam)
• setq org-zettel-ref-include-empty-notes nil: Set whether to include empty quick notes
• setq org-zettel-ref-include-context nil: Set whether to include more context in the overview
• setq org-zettel-ref-quick-markup-key "C-c m": Set the shortcut key for quick markup
• setq org-zettel-ref-python-environment 'system: Set the Python environment type (options: 'system, 'conda, 'venv)
• setq org-zettel-ref-python-env-name nil: Set the Python environment name
• setq org-zettel-ref-python-file "~/path/to/script.py": Set the Python script file path
• setq org-zettel-ref-temp-folder "~/Documents/temp_convert/": Set the temporary folder path
• setq org-zettel-ref-reference-folder "~/Documents/ref/": Set the reference materials folder path
• setq org-zettel-ref-archive-folder "/Volumes/Collect/archives/": Set the archive folder path
• setq org-zettel-ref-debug nil: Set whether to enable debug mode

** FAQ

Q: How do I use org-zettel-ref-mode across multiple projects? A: You can set different overview directories for each project, dynamically changing the value of org-zettel-ref-overview-directory when switching projects using let-bound.

Q: What should I do if the overview file becomes too large? A: Consider splitting the overview file by topic or time period. You can customize the org-zettel-ref-create-or-open-overview-file function to achieve this.

Q: How do I back up my notes? A: Include both the source files and overview files in your version control system (e.g., Git). Additionally, regularly perform file system-level backups.

Q: How can I check the status of the org-roam database? A: You can use the M-x org-zettel-ref-check-roam-db command to check the status of the org-roam database, including version information, number of nodes, etc.

** Troubleshooting

If you encounter issues:

1. Ensure you are using the latest version of org-zettel-ref-mode.
2. Check your Emacs configuration to ensure there are no conflicting settings.
3. Try to reproduce the issue in a clean Emacs configuration (emacs -q).
4. Check the *Messages* buffer for any error messages.
5. If the issue is related to the Python script or Conda environment, check your Python environment configuration.
6. Enable debug mode (set org-zettel-ref-debug to t) to get more detailed log information.

If the issue persists, please submit an issue on the GitHub repository, including a description of the problem, steps to reproduce it, and debug logs.

** Contributions

We welcome community contributions! Here are some ways you can get involved:

• Report bugs or suggest new features.
• Submit patches or pull requests.
• Improve documentation or write tutorials.
• Share your experiences and tips for using org-zettel-ref-mode.

** Acknowledgments

org-zettel-ref-mode was inspired by my friend [[https://github.com/lijigang][@lijigang]]'s [[https://github.com/lijigang/org-marked-text-overview][org-marked-text-overview]]. Due to extensive modifications, I decided to release it separately as org-zettel-ref-mode after discussing it with him.

** Version History

• v0.3 (2024-09-03)
  • Enhanced integration with org-roam
  • Improved Conda environment support
  • Optimized file processing logic
  • Improved overview file synchronization mechanism
  • Added debugging features
  • Integrated external Python script functionality
• v0.2 (2024-08-29)
  • Refined overall workflow, providing automated scripts to handle different formats of electronic documents
  • Improved connectivity with other tools, allowing org-zettel-ref-mode to save note files in denote or org-roam formats through custom configuration
  • Provided quick markup functionality, allowing users to quickly add bold, italic, underline, and other formatting to highlighted text in the source file using org-zettel-quick-markup
• v0.1 (2024-8-21): Initial release
  • Implemented basic quick note and markup functions
  • Added auto-sync mechanism
  • Provided custom options

** Future Plans

• ✅ Improve performance, optimizing handling of large files
• ✅ Integrate with other knowledge management packages, such as org-roam and denote
• Support more file formats (possibly)
• ✅ Further optimize Python script integration
• Add more customization options
• Optimize file association mechanisms, reducing reliance on specific filename suffixes

If you like it, please Star.