search

erykjj / jwlmanager

Multi-platform GUI for managing JW Library (.jwlibrary and .jwlplaylist) files: view, delete, edit, merge (via export/import), etc.
MIT License
66 stars 8 forks source link

Feature Request: Selective and Incremental Export for Efficient Synchronization with External Tools (e.g., Obsidian) #188

Open SchneiderSam opened 1 week ago

SchneiderSam commented 1 week ago

Current Workflow

With the newly added ability to export notes in Markdown, I have developed a workflow that involves:

  1. Exporting all JWL notes/folders/files generated by JWL-Manager and importing them into my Obsidian Vault.
  2. This allows the notes to be indexed within Obsidian, and all my mobile devices can synchronize, giving me access to all JWL-Notes within Obsidian.

While this setup is effective, it presents a challenge when managing a large volume of notes.

Problem Statement

With over 9000 notes, it has become increasingly difficult and time-consuming to keep my notes synchronized between JWL and Obsidian. Although I understand that real-time synchronization may not be feasible in the near term (which would be ideal), a faster and more streamlined solution could greatly improve this workflow.

I had previously synchronized my Obsidian-Notes via iCloud, but because of the amount of data I had to switch to Obsidian Sync. It runs faster there. Obsidian reaches its limits with large amounts of data - at least with standard cloud solutions (e.g., iCloud).

I am currently completely satisfied that I use my JWL as the basis for my JWL note manipulation - so changes to the notes I make in JWL are really only made there. Otherwise I'll get confused at some point. I would only use Obsidian as a view tool for my JWL notes.

Proposed Solution

  1. Selective Export of Recently Modified Notes:
    Add an option to export only the notes that have been added or modified within a defined timeframe, such as the last X days or weeks. This would reduce the volume of exported data and allow for a more focused and manageable update process.

  2. Incremental Synchronization Capabilities:
    Enable JWL-Manager to synchronize newly created or modified notes with an existing dataset, rather than re-exporting the entire library. This would significantly reduce the time and effort required for re-indexing and re-syncing notes in Obsidian.

Current Limitation

Presently, the export is organized into folders, which is logical; however, the re-indexing and synchronization of such a large dataset becomes cumbersome and time-intensive as it grows.

Related

erykjj commented 1 week ago

Hi Sam,

First of all, synchronization of files between devices is an unrelated topic. I use syncthing for all my devices and it works great, even with a large number of files.

erykjj commented 1 week ago
  1. Selective Export of Recently Modified Notes: Add an option to export only the notes that have been added or modified within a defined timeframe, such as the last X days or weeks.

Sorting notes by modification time is a realistic/workable solution. I will add that in to my todo list. Thank you for the idea.

rather than re-exporting the entire library

You can already select a subset

erykjj commented 1 week ago

I am currently completely satisfied that I use my JWL as the basis for my JWL note manipulation - so changes to the notes I make in JWL are really only made there.

My workflow is the opposite: I edit all my study notes in a markdown file (one file for each Bible book) on my laptop; I find it much easier to copy/paste, etc.

I have a simple script to convert that to an import file. When imported into JWLManager, all the previous notes are automatically removed and replaced with the new file.

SchneiderSam commented 1 week ago

I have a simple script to convert that to an import file. When imported into JWLManager, all the previous notes are automatically removed and replaced with the new file.

@erykjj Ok respect. But how do you do that when you are in the meeting or on ministry. And you take notes. Do you also do that in Obsidian? Only on your laptop? never on a mobile device?

I currently only make notes in my ipad and occasionally synchronize with my other devices via JWL-backup-files. I have a lot of respect/fear that something will be messed...

erykjj commented 1 week ago

Everyone has their preferred way to work. I find it... limiting to write content on a mobile device. I do add notes on my tablet/phone as well, but my Bible study notes are all in markdown on my laptop first and foremost - less likely to vanish :-).

Anyway, your time-based export/import is a very interesting idea which I will pursue further. Does Obsidian have a way to update a note's property? So, say you modify a note on Obsidian. Can it set the 'modified' property to the modification time? If not, then JWLManager would have to look at the file modification time and compare with the modification time for the note in the archive database. If newer, import; otherwise, ignore. It would still have to process all the notes one by one.

Other thoughts/ideas?

SchneiderSam commented 1 week ago

Other thoughts/ideas?

Good questions.

My manual (and probably beginner behavior) would look like this:

Step 1: JWL-Manager exports all notes that have been changed in the last x-days (e.g., 30 days)

Step 2: I get folders that only have the updated notes. These folders have the same folder structure as the existing one. image

Step 3: I replace the notes of the last 30 days in the corresponding folders. Theoretically, I could simply drag the new folders into my Explorer. If Windows finds duplicates, I would replace them. All the others would remain. While writing these lines, this thought also occurs to me: you could also work with a script here...

Of course, it would be cool if this could be done automatically and with little effort. Depending on how many folders (notes) I have made changes in, this can be very time-consuming.

To be honest, I can't say how Obisidian handles this. I'm still quite new to it. Do you have a suggestion/idea?

erykjj commented 1 week ago

What is the difficulty if you export all notes and simply over-write the ones in your vault? If they have been updated, they will be over-written; if they have not changed, they will also be over-written, but it won't matter.

SchneiderSam commented 1 week ago

What is the difficulty if you export all notes and simply over-write the ones in your vault?

Not really a problem, but I am concerned that with a large number of notes (my notes - I currently have around 9k notes in JWL), all notes have to be re-indexed. This is very time-consuming for the Obsidian synchronization between different devices of very many notes.

erykjj commented 1 week ago

OK, I think I understand.

So, does Obsidian reindex always? Even if the content is the same? Does it look at the file modification time? If you could find that out, it could help me find a work-around ;-)

SchneiderSam commented 1 week ago

@erykjj yes it does reindexing all notes (this time i have only exported study bible): image

erykjj commented 1 week ago

OK, please research the answer to my other questions. And what triggers this reindexing?

SchneiderSam commented 1 week ago

OK, please research the answer to my other questions. And what triggers this reindexing?

I posted on the Obsidian forum to see if anyone there has insights: https://forum.obsidian.md/t/optimizing-re-indexing-in-obsidian-for-bulk-md-file-imports/90674

I couldn’t find any relevant information in the documentation either…

SchneiderSam commented 1 week ago

@erykjj i got two answer here: https://forum.obsidian.md/t/optimizing-re-indexing-in-obsidian-for-bulk-md-file-imports/90674

Does this help?

erykjj commented 1 week ago

Hi @SchneiderSam That's what I suspected. Thank for confirming. I will see what can be done, though it may be a while.

SchneiderSam commented 1 week ago

Hi @SchneiderSam

That's what I suspected. Thank for confirming. I will see what can be done, though it may be a while.

Thanks for working on this

erykjj commented 4 days ago

Hi @SchneiderSam Would you be able to do some testing for me?

SchneiderSam commented 4 days ago

Hi @SchneiderSam

Would you be able to do some testing for me?

Sure 😁

SchneiderSam commented 4 days ago

Would you be able to do some testing for me?

Just to me sure: this is not this version right? https://github.com/erykjj/jwlmanager/releases/tag/v5.1.2

I have noticed some changes to the master, but I don't know how best to run it on my windows computer. will you create a new version or should I somehow figure out how to get the master to run?

erykjj commented 4 days ago

Thank you. I will build a test release ;-)

erykjj commented 4 days ago

@SchneiderSam Please try v6.0.0-beta1

The UI (light/dark color mode) is still very much a WIP. What I need to know is how the markdown note export is behaving. It should only over-write notes that have been modified since the modification time in the markdown file.

SchneiderSam commented 2 days ago

@erykjj ok looks very good.

What i ve tested:

Step 1: I created a backup of my JWL notes and reformatted them to .md using JWL Manager. The first time, Obsidian had to re-index all the notes again.

Step 2: I made a change to just one note in JWL and then repeated step 1. This time, not all notes were re-indexed, but only the changed one. So I would say: It works wonderfully...

Wow thank you!!!!

SchneiderSam commented 2 days ago

synchronization of files between devices is an unrelated topic. I use syncthing for all my devices and it works great

Now that this function has been so wonderfully implemented by you, the next thought occurs to me: How does he do that?

I imagine the following scenario:

  1. i edit my notes in Obsidian and push them to my iPad/iPhone/laptop (maybe similar to Github). Could the JWL Manager documentation be extended for this? I would be happy to formulate this properly if you want to share your workaround. I only need keywords and I would try to document it step by step from the perspective of a “noob”. I'm sure there are others who have similar questions/problems.

2 I can well imagine importing an existing note in Obsidian to JWL: I edit an existing note in Obsidian and push it to my devices. I still need to figure out how to do this step, but once I've got it down, I'll be able to understand it. See step 1.

  1. how is it when you are in a meeting and they come up with interesting notes on a Bible verse? Do you write it down in JWL or in Obsidian? It would be easier to go JWL -> Obsidian than the other way around - right? How do you then proceed so that all the notes are synchronized with each other?

  2. (idea) An Obsidian plugin that automatically implements steps 1+2 with just one synchronization button would be really cool. Just as an idea...

erykjj commented 1 day ago

v6.0.0-beta2 is out. Please continue testing ;-)

erykjj commented 1 day ago

@SchneiderSam I will reply to your other message later :-)

erykjj commented 1 day ago

4. (idea) An Obsidian plugin that automatically implements steps 1+2 with just one synchronization button would be really cool. Just as an idea...

Thank you for the suggestion, but I will not be working on Obsidian (or any other) plugins. Outside of the scope of this application.

erykjj commented 1 day ago

3. how is it when you are in a meeting and they come up with interesting notes on a Bible verse? Do you write it down in JWL or in Obsidian? It would be easier to go JWL -> Obsidian than the other way around - right? How do you then proceed so that all the notes are synchronized with each other?

@SchneiderSam , it's best you contact me by email, please.

erykjj commented 1 day ago
  1. i edit my notes in Obsidian and push them to my iPad/iPhone/laptop (maybe similar to Github). Could the JWL Manager documentation be extended for this? I would be happy to formulate this properly if you want to share your workaround. I only need keywords and I would try to document it step by step from the perspective of a “noob”. I'm sure there are others who have similar questions/problems.

I'm not sure what needs to be added to the documentation? JWLManager is there to manage .jwlibrary backup archives. Everyone can use it as it best suits their workflow :-)

SchneiderSam commented 1 day ago

v6.0.0-beta2 is out. Please continue testing ;-)

@erykjj still works as espectect. What i've tested:

Step 1: I created new notes in JWL today.

Step 2: I have made a JWL backup.

Step 3: With the JWL-backup from step 2, created an md export with beta-2

Step 4: Only the notes that I actually added were indexed in Obsidian.

Brilliant. It works as expected.

erykjj commented 1 day ago

Thank you for your suggestions, testing and reports, @SchneiderSam !!