View Random Note

[spaceclottey]

One of the best features of using SuperMemo for note taking is that there's peace in knowing no note will ever be permenantly forgotten as it will come back to you in time.

In Obsidian this is can be somewhat replicated using the 'Random Note' feature. Having this in Foam would be great. If this could include weighting for notes that haven't been accessed in a longer time, that would be even better

[riccardoferretti]

this feels like an interesting feature, would you be interested in looking into it? I expect the basic version to be pretty straightforward. Maybe down the line we could also support weighting notes so that the least connected notes are given a bit more weight (as those are the notes most likely to be lost/forgotten)

[rajlego]

We haven't forgotten about this, I'm writing a proposal to explain the pieces needed (should be able to post in next few days. I don't think Space is interested in coding and I'm not a coder (though I can read code) but if anyone else would be interested in collaborating on it, I'd be happy to give input/design ideas.

[riccardoferretti]

Please do share @rajlego, there have been other people interested in the feature, someone might take it

[MCluck90]

I'd be happy to take it on once I know the full scope! I already tested out a local solution for the most basic version

[rajlego]

It wasn't written so incrementally so please forgive any lack of coherence: I recommend reading the proposal here, on our site but just to make this as visible as possible I'll also post it here (though images won't work). Whatever questions you have, ask away (no matter how silly or basic they seem).

Incremental Writing

(below is all for proposal of implementation in Foam)

How does incremental writing work?

In SuperMemo, incremental writing is based on two facets: priorities and intervals. Imagine that I have this mindmap which I wish to convert into a wiki: I can construct a simple writing queue: and finish writing each post before moving on. This functions but has the downside of being in highly arbitrary order. Writing in order of last added or in random order is not efficient. SuperMemo handles this with priorities: (0 is highest and 100 is lowest priority)

Priorities

SM priorities are assigned by me and are at my discretion. SM then uses its sort algorithm which doesn't sort by pure priority (to prevent prioritiy bias). Using priorities, we can then go through list in less arbitrary order and get some added efficiency.

Intervals

The problem with a priority queue by itself is that each operation is all or nothing; you have to finish writing top priority thing before moving onto 2. To fix this, in SuperMemo we use intervals: (this is from my SuperMemo collection, to show actual intervals in action)

I'll get into the algorithm for intervals slightly later but in essence, when you see highest priority note, you can write as much as you want and know that interval will make it so that you can see it again later, you can move on to next one. (I'll call this [incrementability])

In this way, you can go through your queue and allocate as much time and attention as you'd like till you get bored and would like to move on. There are also strong benefits from:

• memory consolidation: when you write about something today and a few days later, in between your brain moves some of that to long-term memory and hones it; when you see it next you'll be able to think and write about it much better than your initial go.
• if you have 7,000 notes, a priority queue by itself is not feasible. You'll never get to the bottom of it or even have the chance to see and possibly reprioritize other parts. With intervals, since there's space between reviews, there's more space for other material. In a year, I could feasibly get through and incrementally write 7,000 notes without going insane; I'm just juggling my attention over time.

Foam Implementation

How would all this work in foam? First off, you would want priority and interval metadata for each foam note. I don't think this would be hard. At start of each day (or scheduled time), you'd want a daemon to make queue of:

-notes with intervals due today

-sort order of queue by priority (more about sorting algorithm SM uses here)

You'd want to press a button and then see the first entry. On seeing it, you want to be able to:

-reprioritize (in SM you press alt+p which brings up this dialougue)

-modify interval manually (generally interval change would be based on algorithm outlined later but sometimes you want to be able to manually touch it. in SM, this is ctrl+j which leads to this dialougue. you can either click calendar day you want to see it on or give it a number)

-write (obviously)

When you're done doing whatever you want to do with the note, you should be able to press something to move onto the next element in the priority queue. When you press next, the IW plugin should also generate next interval for finished element.

Algorithm

I vaguely recall there being more documentation on the algorithm supermemo uses for topics but I can't find it. The thrust of it is that SuperMemo generates new intervals by a-factor x interval = new interval. A-factor, from what I can tell, is influenced by priorities. For high priority material (0% is highest in SM), a-factor should be higher to modulate to more frequent review and the opposite for low priority material. I can probably ask Woz for more details on this though even something basic should be better than status quo.

Postpone

If you don't get through entire queue (very likely) you need some means of having queue handle that for overdone material. SuperMemo uses a feature called auto-postpone (documented here). It works well enough that I hardly know how it practically works since I never have to think about it. It doesn't seem that complicated and in simplest case you could just have things that you don't get to stay in queue and get sorted by priority against everything else.

Other Desired Features

I write on foam with my friend Space. It would be strongly ideal if we could do incremental writing seperately and his priorities and intervals didn't mess with mine. Probably possible with seperated metadata? It does seem like using IW could make it feasible to have multiple people incrementally contribute to refinement of some set of knowledge (which is what we're already doing, just more arbitrarily and at whim of what seems neatest to us in the moment).

Further Work

I'm more than happy to call or talk much further with anyone interested in implementing this. I've used SM ~2.5 years and taught it for 1, and for around 1 year also used IW (though I stopped when I had complex things I wanted to write about but couldn't figure out how structure them in SM without it being a little tedious. Still find it good enough for one-shot isolated notes).
I'm on the Foam discord now as RajT [work]#9202 so maybe we could make a group there to collaborate?

Other Ideas

You can look at your notes as a map/graph and then IW as a random traversal to incrementally shore up and improve parts of the graph. I wonder if you could use weights/links to decide where to go to maximize value (e.g. if you want to explore right now, you could weight towards orphans that don't have many connections or if you want to consolidate you could weight towards nodes that are well connected)

Further Reading

For other explanations of incremental writing, I recommend these posts:

Incremental Writing: No More Writer's Block

The Magic Behind Incremental Writing: Spacing and Interleaving

Incremental Writing in SuperMemo

Incremental Writing Disambiguation Page on SuperMemo.guru

[riccardoferretti]

Thanks @rajlego for the detailed write-up. I am not very familiar with SuperMemo, so it's hard to evaluate the functionality per se. This specific workflow feel better suited to a plugin built on top of Foam core. We don't expose the API yet, but we could to enable an extension built on the core model (that is the longer term plan, we haven't done it yet to be able to iterate without worries on the model).

I believe a simpler version of this could be part of core Foam, I guess something like that would include:

• random or weighted selection
• filters on notes to include (e.g. via globs)

Thoughts?

[rajlego]

Had a great discussion with Riccardo, from what he said you'd have 3 layers: I'm not entirely sure about actual code but it generally seems fairly feasible if anyone wants to collaborate. @MCluck90 are you still interested/have stuff you'd want me to clarify I'd also be interested in trying your prototype regardless of how far the implementation is (since by comparison only other option right now is just random notes which is not ideal)

[MCluck90]

My small prototype only handles the most basic case: random note selection without any prioritization. @riccardoferretti @rajlego Did the two of you come to the conclusion that this sort of prioritized randomization should be a separate plugin or included as part of core? My impression is that this sort of Anki-like feature would be better suited as a separate plugin

[riccardoferretti]

@MCluck90 the more advanced prioritization solution that @rajlego is talking about will be a separate plugin. Foam core just needs a basic implementation as that will be fine for most users.

[riccardoferretti]

Closing this as the requirements for Foam have been satisfied for now. For future iterations on opening random notes we can create new issues.