[Feature Request] Options to share context between instances

[noncom]

Currently each code block of Numerals has a separate context. Using values from one block is not possible in another one. It would be great if that was possible. For example by annotating a code block with some special comment like:

Somewhere in block A:
# Context: Base calculations

Somewhere in block B:
# Context: Derived values stage 1
# Import-Context: Base calculations, Some other context, One more context

or with a different syntax:

Somewhere in block A:
context calculations-phase-1
a = 1

Somewhere in block B:
context calculations-phase-2
c = 2

Somewhere in block C:
import calculations-phase-1 as ph1 // imported as object ph1
import calculations-phase-2  // merged into the current context for direct access. could also be "inherit" or smth else
b = ph1.a + 2 + c

I know that there are many nuances to such a feature, like transient dependencies sharing, import failure handling, and so on, so I'm not sure how easy that would be.

With this feature it would be possible not to work with one huge code block when you have a lot of stuff, but break it apart into several, and have a comfortable view of the previous ones already rendered and their intermediate results, while only editing the current one you're working with.

This could also help later to share context with inline blocks so that a text that has such an inline block would not only be able to show what was immediately calculated there, but take values from a greater calculation body, happening elsewhere. For that a variable could probably be prefixed with global or export:

export a = 321
global b = 654

Note: These are just a few examples of the concept, the shown syntax is just pseudo-syntax, and any way of implementing the possibility of sharing the variables would be good.

[gtg922r]

Thanks for such a thorough feature request!

This is definitely something that I want to incorporate. The biggest initial problem is figuring out how to re-process every block so that the results are deterministic and not dependent on the order of editing blocks.

[noncom]

Thanks! That would be super awesome! :)

As for the implementation, I think that one possible solution is to maintain the dependency graph. I'll share what I think though my idea is pretty standard.

So you build the graph from the very start, when a context starts refering another one. So the graph is a tree. Circular dependencies should be forbidden since they make no sense in an application like this. They should show an error similar to the syntax error and stop the evaluation.

On every change see if the graph needs to be updated, to keep it current who refers who. And then, when something is changes in one member, the change propagates down the tree to the children. Since there would be no circular dependencies, it would be rather trivial.

The cost of updating everything down the line would not be really different from updatng a single huge code block.

Optimizations could be implemented:

• If the update caused no difference in one of the blocks it's propagating through, it stops propagating down that chain.

• If a block should be updated during a propagation, it checks if it has more ways of dependency from one of the blocks that were updated during the current propagation. And if these ways were not yet propagated. If there are such ways, then it prevents updating and waits for when the last path is being followed through.

  • So let's say there's A -> B -> C -> D -> E -> F and B -> X -> Y -> Z -> D
  • From B it has 2 ways to follow.
  • And D has 2 ways of dependency from block B
  • So let's say you change A and it starts on the first one and it goes like:
  • A -> B [remember to start for the other child, X] -> C -> D [!stop, D has another incoming link]
  • So then it continues form the memory of B, it goes the other route:
  • (B) -> X -> Y -> Z -> D [good, now it's the last incoming path, so recalculate D with info from C and Z, and go next] -> E -> F
• And if in the previous example the interpretation of the second chain stops on any of X -> Y -> Z because there's no change, then it must remember that it's waiting to continue on D and it was the last incoming link, so it returns to that.

I might have missed something, but that's the general outline of how I imagine it currently.

[gtg922r]

I have added a relatively simplistic implementation in Numerals 1.4 (Beta). Please help test it out by following the instructions on the README for beta testing with BRAT!

Prefacing any variable with $ will indicate that the variable should be a page "global", and will be available to any block on that page.

Please read the beta release notes for instructions on how to use and warnings.

[UsefullPartBrad]

How does Mathpad do it? Seems like this would be an interesting thing to steal but also improve. Dataview inline fields use the last instance. It Could be an interesting additional feature to log each variable as a dataview field if dataview is installed. That lets there be persistence and the ability to reference across notes not only contexts. Obviously recalc only happens when the note is loaded... but... I have uses for this.

Further abstraction could be to (with the $ sign notation) allow inline blocks like both dataview and mathpad. (mathpad is no longer being developed and might be available for takeover)

[sbliven]

I switched to the beta just to test this issue. So far it works great.

What's the reason not to include all variables in the global context? Are you worried about performance issues? I mostly have small notes, so for me it would be more useful to just re-evaluate all math blocks every time one of them changes, with all blocks sharing all variables. If you're worried about backwards compatibility, perhaps this behavior would be enabled in the frontmatter. I'd rather avoid having to use $ for all my variables (I have PTSD from perl).

I like @UsefullPartBrad's suggestion of dataview integration. I used dataviewjs for calculations for a long time before switching to Numerals. The mathjs syntax is much cleaner, but cross-note references would be awesome.

[gtg922r]

Closing with public release of Numerals 1.5

@sbliven - right now its a bit of a sub-optimal implementation. It relies on all code blocks in a page having been rendered in order if you set the same global in multiple math blocks. So if you aren't careful its possible to get non-deterministic behavior. Which isn't great. I couldn't come up with an easy to implement solution to that problem.

So two reasons:

1. I didn't want the default behavior to be one that could lead to confusion and bugs
2. (Most importantly) I didn't want to break backwards compatibility. Theoretically this could have unexpectedly changed the value that was displayed which seems unacceptable.

[gtg922r]

Oh by the way with "Result Insertion" you get dataview integration.

using the syntaxt @[value], the value will be inserted after :: - which results in inline variable in dataview. So typing @[meaning_of_life] would get modified, in the raw text of the note to become: @[meaning_of_life::42]