Feature Request: Watch for updates to the underlying .jl file

[MasonProtter]

One thing that would be nice is if I have a Pluto notebook, Foo.jl, and then if I opened Foo.jl in my editor, changed and saved something in the editor, I would want the Pluto notebook to automatically update.

[fonsp]

It's not part of my overall goal of making computing more accessible, but it is something that I (as an experienced programmer) would like to experiment with.

I am also reluctant, I feel like it is a lot of work to get this right, because there is no obvious bijection between Julia files and Pluto notebooks. See my comments here about why this is difficult: https://github.com/fonsp/Pluto.jl/issues/131#issuecomment-636764311 . One thing that I would like to emphasize is that Pluto is a project to design an IDE, this is what I find fascinating to work on!

[shashi]

One thing that was annoying to me is if I have a notebook open, and do a git pull which applies a change cleanly to the notebook, I still see the old notebook until I restart Pluto (refreshing the page also doesn't change it.)

[antoine-levitt]

It's not part of my overall goal of making computing more accessible, but it is something that I (as an experienced programmer) would like to experiment with.

I feel there is a bit of a false dichotomy here - that experienced programmers want this kind of features, but novice users should not interact with text any other ways than through a plain text box. Inputting text is a serious business, one to which entire projects have been dedicated, and which is not easy to replicate in a project such as Pluto (and indeed Jupyter as well). Anybody serious about computing (in the broadest sense) should spend at least some time learning the tools of the trade (any good editor, as long as it's not a plain text box), and hybrid solutions (eg a notebook that allows users to use a text editor) are a productivity boost for any user. Attaching Pluto notebooks to a particular file and having two-way synchronization seems to be a very good way to achieve that. I understand it's not a priority though.

[fonsp]

Thanks for your input! I did not look at it that way :)

[fonsp]

PRs welcome as an experiment, but it's not a priority, and it will not be implemented if it gets too complicated.

[pbouffard]

Could this be extended to include watching external files on which the notebook is dependent? A lot of workflows involve generating a file with some external tools that is the primary input for a notebook, and/or use files as intermediate steps. Not being able to watch these "breaks" reactivity in a sense.

It looks like a basic version of this could be implemented using Clock from PlutoUI.jl, but AFAICT that would trigger downstream cell updates whether or not the file contents have changed. Actually more generally can Pluto prevent this kind of needless downstream updates and/or is there a way to implement such a thing? Even if it wasn't automatic--it would be nice to be able in particular to prevent longer running cells from recomputing, or at least to be able to mark certain cells for manual recomputation. Wow, this comment has gone really OT 🙃.

[fonsp]

Good suggestion! But could you move it to #115?

[Moelf]

can we make an intermediate step where we can manually refresh a notebook?

Pluto already has a way of tracking if the cell has been changed, can we manually ask pluto to:

1. update cell text
2. check each cell if changed
3. do normal pluto thing?

this should pave the way for auto watching&update

if someone can point me to relevant code I'm willing to take a dive

[fonsp]

That would be a good intermediate step for the experiment! But the end result should definitely be automatic. Some pointers (but I'm afraid that this experiment still needs a general understanding of how pluto works which is an adventure that you need to take some time for):

You can deserialize (load) a notebook file into a Notebook struct using this function: https://github.com/fonsp/Pluto.jl/blob/928bf8e7f327975873545f1d810a29ef3bf07c12/src/notebook/Notebook.jl#L138

Then you compare that to the current Notebook. How you get it depends on where your code will live, but it is stored in the current ServerSession. You can also run your code in an @async while true as an experiment here: https://github.com/fonsp/Pluto.jl/blob/928bf8e7f327975873545f1d810a29ef3bf07c12/src/webserver/SessionActions.jl#L48

[jerlich]

I just want to 2nd this. Maybe I'm doing something wrong, but when I load my Pluto notebook (with Plots, DataFrames, MixedModels....) it takes a while. it would be really nice to be able to do a git pull and have Pluto figures out which code cells changed and react to just those changes.

[fonsp]

https://user-images.githubusercontent.com/6933510/112561129-e1742300-8dd4-11eb-98cf-952ce4e1b7b6.mov

[MasonProtter]

Amazing @fonsp! Is there a public branch where one can play around with this? Is it just https://github.com/fonsp/Pluto.jl/pull/1008?

[antoine-levitt]

I really like the concept of stateless notebooks, but there's no way I'm leaving emacs to write code, so I'm super excited for this, can't wait to try it out!

[fonsp]

no way I'm leaving emacs to write code, so I'm super excited for this, can't wait to try it out!

I should say that this use case is more involved than just auto-refreshing Pluto when the file changes, because editing the notebook file can lead to invalid notebook files. The file format is not designed to be human-written (and I am not personally interested in making that). This is the concern that I wanted to express in https://github.com/fonsp/Pluto.jl/issues/240#issuecomment-667714716

The primary reason to implement this feature is git https://github.com/fonsp/Pluto.jl/issues/240#issuecomment-687208506, and we are experimenting in using it for #1045 and https://github.com/JuliaPluto/PlutoSliderServer.jl

[antoine-levitt]

Sure, but being able to do interactive things in Pluto but edit larger cells (like function definitions) in emacs would already be nice.

[holomorphism]

I've been trying PR #1029 for a while to edit notebook files directly in my editor, and this feature is extremely useful. Thank you very much!

In my environment (Ubuntu 18.04 & vim), it's working fine so far.

As a tip for Vim users, to get the same feel as VS Code in Fons' video, you may want to enable the autoread feature in vim. https://stackoverflow.com/a/18866818

[GuanglinDu]

@fonsp A similar problem: Pluto cannot update the file with new changes edited with another Pluto.

Scenario: I run Pluto in the cloud and locally on my desktop or laptop PC at the same time, and work on both sides as needed. I use Syncthing to sync all the files between devices. I only edit the files with Pluto from the browser. However, Pluto cannot update the changes from either side, even if I refresh the browser or re-open the .jl file. I checked the .jl files with textual editors and found that they're in sync all the time.

Workaround: Edit only one-sidedly. Chances are that I'll have to work outside of my office network with my laptop running Pluto locally. Once our office network is available, Syncthing will automatically update the files on the server side. However, the changes cannot show up in Pluto. Any suggestions?

[fonsp]

@GuanglinDu did you try the new feature? You need to enable it with Pluto.run(auto_reload_from_file=true)

[GuanglinDu]

@fonsp Many thanks. That flag does the magic! julia> import Pluto; Pluto.run(;host="0.0.0.0", port=1234, launch_browser=false, require_secret_for_open_links=false, require_secret_for_access=false, auto_reload_from_file=true)