Feedback for the Esbonio VSCode extension `v0.92.1`

alcarney commented 1 year ago

I've just pushed a pre-release version of the Esbonio VSCode extension - v0.90.1, the first draft of what ultimately should become v1.0. It would be greatly appreciated if people can try it out and offer some feedback.

What to expect

The extension now includes a bundled copy of what will become v1.0 of the language server - it should no longer be necessary to install esbonio directly into your documentation's Python environment!
The core of the language server has been re-written from the ground up with the aim of supporting multi-root workspaces / multiple sphinx projects from the start.
The preview pane in the VSCode extension itself has been overhauled and should now support synchronized scrolling between itself and the rst source document.
Bugs and rough edges!

What not to expect

Being a rewrite and a very early pre-release, none of the language features (e.g. completions, diagnostics, etc.) from the previous version have been ported across - yet.
Python 3.7 support, as it is now end of life.

What to test

Does the language server even start? This is my first attempt at distributing the language server in this way, I wouldn't be surprised if there are some edge cases
Can you configure your Python environment as required?
Can you get the server to start Sphinx with the correct configuration for your project?
Does it work with your multi-root / multi-project setups correctly?
Does the sync scrolling work?
Does the preview update correctly on save?

The focus of this pre-release is purely on getting the fundamentals right, distribution, multi-root support, Sphinx configuration, the new preview implementation. Once it's clear that this new approach is going to work out, it should be possible to port across the missing features... relatively quickly :sweat_smile:

davbeek commented 1 year ago

This is the VSCode output for esbonio set to debug. I only changed my username to xxxxx:

``` [client] Python extension is available [client] Python extension is active [client] Using environment /Users/xxxxx/miniconda3/bin/python: /Users/xxxxx/miniconda3/bin/python [client] Server start command: /Users/xxxxx/miniconda3/bin/python -S -m esbonio.server [client] LanguageClientOptions: { "documentSelector": [ { "scheme": "file", "language": "restructuredtext" }, { "scheme": "file", "language": "python" } ], "initializationOptions": { "server": { "logLevel": "debug", "logFilter": [], "showDeprecationWarnings": false, "completion": { "preferredInsertBehavior": "replace" } } }, "outputChannel": { "name": "Esbonio" }, "middleware": { "workspace": {} } } [client] Starting Language Server [esbonio] Loaded extension 'esbonio.server.features.sphinx_manager' [esbonio] Loaded extension 'esbonio.server.features.preview_manager' [esbonio] Language client: Visual Studio Code 1.79.2 [esbonio.SphinxManager] Opened document 'file:///Users/xxxxx/1-repos/website/main/index.rst' [client] workspace/configuration: { "items": [ { "scopeUri": "file:///Users/xxxxx/1-repos/website/main/index.rst", "section": "esbonio.sphinx" } ] } [client] Python extension is available [client] Python extension is active [client] Using environment /Users/xxxxx/miniconda3/bin/python: /Users/xxxxx/miniconda3/bin/python [esbonio.SphinxManager] User config: SphinxConfig(enable_dev_tools=False, enable_sync_scrolling=True, python_command=['/Users/xxxxx/miniconda3/bin/python'], build_command=[], cwd='', python_path=[]) [esbonio.SphinxManager] Cwd: /Users/xxxxx/1-repos/website/main [esbonio.SphinxManager] Trying path: /Users/xxxxx/1-repos/website/main [esbonio.SphinxManager] Sphinx agent env: { "PYTHONPATH": "/Users/xxxxx/.vscode/extensions/swyddfa.esbonio-0.90.1/bundled/libs/esbonio" } [esbonio.SphinxManager] Starting sphinx agent: /Users/xxxxx/miniconda3/bin/python -m sphinx_agent [esbonio.SphinxManager] Starting sphinx: sphinx-build -M dirhtml /Users/xxxxx/1-repos/website/main /Users/xxxxx/Library/Caches/esbonio/422b08b53fe0c924f6622ade34ee83d3 Running Sphinx v5.3.0 [client] Previewing: file:///Users/xxxxx/1-repos/website/main/index.rst [esbonio.PreviewManager] Preview file called file:///Users/xxxxx/1-repos/website/main/index.rst [esbonio.PreviewManager] '/Users/xxxxx/1-repos/website/main/index.rst' -> '/index.rst' -> '/' [esbonio.PreviewManager] Preview running on port: 53861 [esbonio.PreviewManager] Websockets running on port: 53862 [client] Result: {"uri":"http://localhost:53861/?ws=53862"} [esbonio.PreviewServer] "GET /?ws=53862?r=1 HTTP/1.1" 200 - ```

No html files were generated. This is the preview panel output:

After switching back to the esbonio normal release version, everything worked OK. It was a single root setup, since I no longer work with multi-root setups.

alcarney commented 1 year ago

Thank you for taking the time to try it! :)

No html files were generated.

Ah, something I forgot to mention in the original write up - the only build trigger currently implemented is saving a file. If you make some changes and save, once the build finishes the preview should hopefully work then 🤞

I no longer work with multi-root setups.

Good to know. While multi-root will be a headline feature, the architecture should eventually be an improvement even for single root projects. Since the Sphinx app has been offloaded to another process, work can be done in parallel and the main language server is free to do other things during builds etc.

davbeek commented 1 year ago

Indeed, after saving and opening the html preview it works, and scrolling either the editor or preview causes the preview or editor to scroll sync perfectly. Nice!

One thing that does not work, when clicking on another rst file of the same website, the preview panel does not update, it stays at the same preview of the old rst file. Editing and saving does not make any difference. The debug output stops at this line after the new rst file is opened:

[esbonio.SphinxManager] Opened document 'file:///Users/xxxxx/1-repos/website/main/assignments/final-assignment/index.rst'

After closing and reopening the preview window, the correct preview appears:

[esbonio.SphinxManager] Opened document 'file:///Users/xxxxx/1-repos/website/main/assignments/final-assignment/index.rst'
[client] Previewing: file:///Users/xxxxx/1-repos/website/main/assignments/final-assignment/index.rst
[esbonio.PreviewManager] Preview file called file:///Users/xxxxx/1-repos/website/main/assignments/final-assignment/index.rst
[esbonio.PreviewManager] '/Users/xxxxx/1-repos/website/main/assignments/final-assignment/index.rst' -> '/assignments/final-assignment/index.rst' -> '/assignments/final-assignment/'
[esbonio.PreviewManager] Preview running on port: 55501
[esbonio.PreviewManager] Websockets running on port: 55502
[client] Result: {"uri":"http://localhost:55501/assignments/final-assignment/?ws=55502"}
[esbonio.PreviewServer] "GET /assignments/final-assignment/?ws=55502?r=4 HTTP/1.1" 200 -
[esbonio.PreviewServer] "GET /_static/pygments.css HTTP/1.1" 304 -
[esbonio.PreviewServer] "GET /_static/documentation_options.js HTTP/1.1" 304 -
[esbonio.PreviewServer] "GET /_static/webview.js HTTP/1.1" 304 -
[client] viewInControl cooldown ended.
[client] viewInControl cooldown ended.

davbeek commented 1 year ago

I also noticed that editing and saving does not trigger scroll sync. The only way I found to force scroll sync is scroll a tiny bit, then immediately, editor and preview go to the same synced position.

davbeek commented 1 year ago

Multi-root also appears to work: I opened two projects in the same VSCode workspace and I could open a previewer on a document from the first project. When I then close the preview and open a document from the second project, edit and save it, and then open the previewer again, I see the correct preview of that document.

alcarney commented 1 year ago

Awesome, thank you for the feedback! :)

p-rogalski commented 1 year ago

I'm on Windows and got the following traceback when trying to save or open the preview pane:

Output: Esbonio

```shell [esbonio.SphinxManager] Saved document 'file:///f%3A/python-pdm/docs/index.rst' [client] workspace/configuration: { "items": [ { "scopeUri": "file:///f%3A/python-pdm/docs/index.rst", "section": "esbonio.sphinx" } ] } [client] Python extension is available [client] Python extension is active [client] Using environment F:\PYTHON-PDM\.VENV\SCRIPTS\PYTHON.EXE: f:\python-pdm\.venv\Scripts\python.exe [esbonio.SphinxManager] User config: SphinxConfig(enable_dev_tools=False, enable_sync_scrolling=True, python_command=['f:\\python-pdm\\.venv\\Scripts\\python.exe'], build_command=['sphinx-build -b html docs/ docs/_build'], cwd='', python_path=[]) [esbonio.SphinxManager] Cwd: f:\python-pdm [esbonio.SphinxManager] Sphinx agent env: { "PYTHONPATH": "c:\\Users\\me\\.vscode\\extensions\\swyddfa.esbonio-0.90.1\\bundled\\libs\\esbonio" } [esbonio.SphinxManager] Starting sphinx agent: f:\python-pdm\.venv\Scripts\python.exe -m sphinx_agent [esbonio.SphinxManager] Starting sphinx: sphinx-build -b html docs/ docs/_build [esbonio.SphinxManager] Process exited with code: 1 [esbonio.SphinxManager] Stderr: Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "c:\Users\me\.vscode\extensions\swyddfa.esbonio-0.90.1\bundled\libs\esbonio\sphinx_agent\__main__.py", line 1, in import asyncio File "C:\Users\me\AppData\Local\Programs\Python\Python311-32\Lib\asyncio\__init__.py", line 42, in from .windows_events import * File "C:\Users\me\AppData\Local\Programs\Python\Python311-32\Lib\asyncio\windows_events.py", line 8, in import _overlapped OSError: [WinError 10106] Der angeforderte Dienstanbieter konnte nicht geladen oder initialisiert werden ```

Notes:

I had to manually specify the build command sphinx-build -b html docs/ docs/_build
I replaced my username with me
Log Level set to debug

alcarney commented 1 year ago

I'm on Windows and got the following traceback

Interesting that you got a traceback! I had a quick play around with this on Windows the other day and couldn’t even get the Sphinx process to launch - there was some cryptic error about not being able to generate random numbers.

I was eventually able to fix it by making the sure the server passes through the SYSTEMROOT env variable - there’s a chance that will also fix the issue you’re seeing? 🤞

I also encountered a bunch of issues caused by Window’s use of \ in paths - I hope to push another release soon^TM with some fixes for the issues we’re seeing in this thread.

Thank you for the feedback! 🙂

alcarney commented 11 months ago

I hope to push another release soon^TM

So, soon^TM is further off than I anticipated, but I've been using v0.90.1 a fair amount and have identified a bunch of little details that probably should be tackled before it's worth pushing another release - let me know if you think I've missed anything

On the plus side, the new preview approach is proving to be much more flexible! It should be possible to support previews in any editor - not just VSCode. If you are interested, clicking the image below will take you to a short clip showing that previews can even work over your LAN - here I'm using an iPad to preview documentation being built by a raspberry pi on my local network!

Previews

[x] The preview window does not stay closed, if the next document VSCode focuses happens to be an rst document.
[x] The preview window should give some kind of loading indication so that there is some feedback after clicking the "open preview" button. The first preview on a fresh build can take a while!
[x] handle the case where the user opens a preview halfway down a page.
[x] Add show-markers url parameter to have webview.js highlight all the .linemarker elements in the page
[ ] Navigating in the preview window should be reflected back into editor... somehow
[ ] ~move didChangeTextEditor logic into the preview server~
[x] the preview pane does not immediately recover from being hidden by another editor tab. Requires switching to another document and back to give it a "kick"
[x] investigate webview persistence and serialisation
[x] preview pane should (be destroyed?) and recreate itself when the server restarts
[x] move editor in control/view in control logic into the preview server
[x] Scrolling to the very top (and possibly bottom?) of the page should be handled as a special case.
[x] Previews should trigger a build if there has not been a build already
[x] The PreviewManager does not gracefully handle the case where the ws connection disappears
[X] On windows, the language server is not able to correctly transform the path for an rst file into a valid url for the previewer to use.

Sphinx Agent

[x] When starting the server with multiple rst files open, the server will receive multiple textDocument/didOpen messages in quick succession. Too quickly too properly initialize a sphinx agent! This leads the server to believing that there are no agents yet for the given file(s) and spawning multiple agents simulataneoulsy.
[x] :bangbang: The server can enter an infinite loop where it endlessly spawns new sphinx agent processes! The good news is that is only appears to be am issue when using an OS like Fedora Silverblue and providing an explicit buildCommand
[ ] Implement sphinx.env a dict of environment variables to add to the sphinx agent process.
[x] Sphinx agent should be producing diagnostics
[x] Agent should accept a content overrides when triggering a build so we can handle live preivews.
[x] Agent should return a mapping from source files it knows about -> output files on each build
[ ] The line number transform should include the source filename into the HTML. This should allow sync scrolling to work for included files.
[ ] Sphinx build failures are not being handled correctly.
[ ] Bring back sphinx.configOverrides as an easy way to override aspects of the project's config.
[X] Implement sphinx.envPassthrough a list of env variables to pass through to the sphinx agent process
[X] The SYSTEMROOT environment variable should be passed through to the Sphinx agent on windows.

Misc

[ ] ~Duplicated log messages following server/sphinx agent restarts~
[X] On windows, the language server does not realise it has already created a client so it creates a new one everytime a new file is opened.

davbeek commented 11 months ago

Thanks for the update and for all the time you put into perfecting this beautiful tool. A lot of work that still needs to be done!

alcarney commented 9 months ago

I've just pushed the next pre-release version of the Esbonio VSCode extension - v0.91.0 It would be greatly appreciated if people can try it out and offer some feedback.

What to expect

Improvements to the preview pane
- Sync scrolling should now snap to the top of the page
- An initial sync should happen when opening the preview for the first time (Note: I'm aware this doesn't work 100% reliably yet)
- Builds should no longer need to be manually triggered in order to generate a preview
- Preview pane should survive being hidden and server restarts
- Preview pane should give some feedback during loading
Diagnostics
- Issues from Sphinx (and now also!) docutils should be reported as diagnostics...
- ...and be correctly cleared when they are resolved
- I'm aware the docutils based diagnostics are annoying for non-reStructuredText content (e.g. code-blocks), I hope to have a fix for that in the next pre-release version.
Live previews!!
- Following a short break from typing (~2s IIRC) the server should automatically trigger a rebuild and refresh the preview automatically (if it's open)
- Even if no preview is open, diagnostics (and eventually everything else) should feel much more responsive as it will be all updating in almost-but-not-quite-real-time^TM
Document symbols
- so the Outline view should work again
Bug fixes, new bugs and some rough edges!

What not to expect

Being an early pre-release, many of the language features (e.g. completions, goto X) from the previous version haven't been ported across - yet.
Live previews on Windows - I think case insensitive file paths are to blame (see #633)
Python 3.7 support, as it is now end of life.

What to test

Does the live preview work?
How does the delayed build trigger feel? Too fast? Too slow?
What is the resource usage like?
Can you configure your Python environment as required?
Does it work with your multi-root / multi-project setups correctly?
Can you get the server to start Sphinx with the correct configuration for your project?

davbeek commented 9 months ago

This works:

Live preview in single root setup works perfectly, also when opening and closing files and when switching focus from one file to another.
Syncing works both ways.
As you mentioned initial sync does not always work: for many files it works, but for one (big) file it did not work.
Upon saving, the preview updates shortly after. The delay is perfect for me.
When loading, there is a moving horizontal bar at the bottom, which gives nice feedback.
When restarting the server via Shift-Command-P > Esbonio: Restart Language Server, the server restarts properly and reloads the preview.
Multi-root workspaces work as they should. Changing focus from a file in one root folder to a file in another root folder works. The only thing is that when you add a second root folder to a single workspace, with an open previewer, so that you change from a single root workspace to a multi-root workspace, the preview stays at the old view. You need to close and reopen the preview, to get the multi-root workspace to work properly. Starting VSCode in a multi-root workspace works perfectly for Esbonio.

This does not work (anymore): Instead, clicking "{}" now shows several unresponsive items:

Note that changing focus in the editor or previewer leads to perfect syncing, but changing focus to a different part of the editor via clicking in the OUTLINE in VSCode, does not sync the previewer: it stays unchanged.

I got the same results in Windows as on my Mac, but I did not yet test multi-root workspaces in Windows. Edit: I just tested multi-root workspaces in Windows and they work almost the same as on the Mac. The only difference that I found, is that from time to time, in Windows clicking from one file to another in VSCode (also changing between different root folders), sometimes causes the previewer to hang: at the bottom of the preview window there was the message Loading, and the preview window went almost black. Changing focus to another file and then going back to the file that hanged always instantly solved the problem. On the Mac, it always worked reliably and correctly. Screenshot from Windows:

I do have one single root workspace that does not work anymore, but I need to look into this further. Could be a problem on my part. Edit: was an error in conf.py. After solving this, everything worked as mentioned in the above.

alcarney commented 9 months ago

Thanks for the quick feedback! Glad to hear that things appear to be (mostly!) working

The only thing is that when you add a second root folder to a single workspace, with an open previewer, so that you change from a single root workspace to a multi-root workspace, the preview stays at the old view. You need to close and reopen the preview, to get the multi-root workspace to work properly.

Note that changing focus in the editor or previewer leads to perfect syncing, but changing focus to a different part of the editor via clicking in the OUTLINE in VSCode, does not sync the previewer: it stays unchanged.

Ooh interesting, hadn't considered these scenarios.

This [Language Status Items] does not work (anymore):

I think these might need a rethink in general...

It probably makes sense for old Restart Server option - which was shown next to the Sphinx version to only restart the underlying Sphinx process - since there can be more than one now
Would you also want a more global option that does restart the entire server? Or is doing that via just the command palette enough?

Edit: was an error in conf.py. After solving this, everything worked as mentioned in the above.

Sounds like there should at least be some more feedback from the server when things go wrong!

davbeek commented 9 months ago

I think these might need a rethink in general...

Yes.

Would you also want a more global option that does restart the entire server? Or is doing that via just the command palette enough?

I think the old Restart Server option, which just restarts the underlying Sphinx process, should be enough in most cases.

Sounds like there should at least be some more feedback from the server when things go wrong!

Actually there were very many lines of error reporting with resulting exceptions in the Esbonio OUTPUT tab, which did not help me. The problem was an incorrect old sphinx-contrib extension in conf.py, which was nicely reported by a single line in the PROBLEMS tab. I missed it at first, because I mistakenly blamed Esbonio (should have had more confidence in v0.91.0 😊). Once I noticed the PROBLEMS tab, fixing it was straightforward.

davbeek commented 9 months ago

After using v0.90.0 a bit more for my day to day work, I now understand the "What to test" issue: How does the delayed build trigger feel? Too fast? Too slow?. Esbonio now updates the preview a few seconds after the editor text has changed. I initially thought that Esbonio updated the preview after I had saved the file. In general, I am fine with the new behavior and the time delay before updating. However, in one case, it is inconvenient.

The problem is that Esbonio disregards tabsets (.. tab-set::) and tab-items (.. tab-item) as defined in the sphinx_design extension. See figure below for an example of a tabset:

When a tabset is selected in the preview window (in the figure, the second tabset is selected), and a change is made in the text and Esbonio updates the preview after the short delay, the preview always switches back to the first tabset (in the figure, the "1DH Distributing Handling" tabset).

The main problem here is that Esbonio is not aware of tabsets. Ideally, the preview should sync to the tabset that contains the edit cursor. I don't know if this is possible. If not, then I would prefer Esbonio to only update the preview when saving. In general, I am not sure what I prefer (the current behavior, or updating the preview when the editor has been saved), because I have only just started using the new functionality.

davbeek commented 9 months ago

While working, I noticed that the automatic update of the preview after about two seconds of inactivity (without the need to save the file) is actually very nice and useful.

However, I ran into another problem. Consider the large file from the figure below.

The file has a section "The Distributing Station", which is selected in the TOC bar on the left. I can then scroll to the next section of the file for editing, for example "The Handling Station". The preview syncs while I scroll the editor, but the TOC stays fixed at "The Distributing Station".

When I make a change in the editor in the "Handling Station" section, the preview is updated and jumps back to the start of the "Distributing Station" section, where I was before scrolling to the next section. Obviously, I would like the preview to stay at the new editing postion, but apparently, the position in the TOC has priority.

Scrolling the editor a bit does make the preview sync back to the editing position, but after each edit, the preview jumps back to the position of the TOC.

The problem occurs on all levels of (sub)headings:

On the level of different subsections and subheadings: after scrolling to another heading, and editing, the preview jumps back to the active (sub)heading in the TOC,
Within each (sub)section. When a (sub)section is long, and you need to scroll down while editing, after each edit, the preview jumps back the the beginning of the section, which is the postion where you end up in the preview by clicking in the TOC on that heading.

Edit: I cannot reliably replicate this behavior in other files, also due to lack of time to properly investigate this, so for the time being, consider the above just as findings that hold for that specific case, but that need not be generally applicable.

alcarney commented 9 months ago

Esbonio now updates the preview a few seconds after the editor text has changed. I initially thought that Esbonio updated the preview after I had saved the file.

It now does both :)

Ideally, the preview should sync to the tabset that contains the edit cursor. I don't know if this is possible.

Technically possible... but unlikely to be implemented anytime soon. It would be nice to get to a point where extensions like sphinx_design could teach esbonio about how they work, but that's a discussion for another day!

I expect the easiest option is to provide a few config values to control how the preview refreshes/incremental builds are triggered.

Scrolling the editor a bit does make the preview sync back to the editing position, but after each edit, the preview jumps back to the position of the TOC.

This is only an issue after clicking on an entry in the TOC right? If so, I think I know what is happening.

The preview pane is essentially a web browser tab so it has an internal url of the document that is being shown. When clicking on an item in the TOC I suspect that url is being updated to something like http://localhost:1234/manual.html#the-distributing-station

A preview refresh is the equivalent of hitting F5 - which causes the page to scroll back to the heading referenced in the url.... :thinking:

I'm not sure it would be wise to try and teach the preview about TOCs, since the details are likely to differ across sphinx versions and themes... but perhaps having the preview drop the #the-distributing-station part of the url after scrolling would be enough to fix this - I'll have to have a play around.

alcarney commented 9 months ago

Quick question when it comes to previewing documents that .. include:: other files.

Currently when scrolling the preview, the editor will "bounce" to the top of the current file and back when scrolling. This is due to the preview only having access to line numbers - and not the source file they belong to.

Injecting the source file information into the preview should be easy enough, but I'm not sure what should be done once that data is available and I'm wondering if you have any thoughts?

Should it

pause scrolling the editor until the preview clears the content that was .. included:: from the other file(s)
or have the editor cycle through each included:: file and main source file as the preview enters the corresponding sections

Both are just as easy to implement - but I'm worried the second option could be jarring, especially if you happen to scroll quickly?

davbeek commented 9 months ago

Currently when scrolling the preview, the editor will "bounce" to the top of the current file and back when scrolling.

I don't understand this. I don't see any "bouncing" to the top. When I open a file, the preview starts at the top. When I then scroll the editor up and down, the preview follows by scrolling from line to line in discrete steps.

I think that the first option (pausing) could be confusing.

I have a clear preference for the second option. This would be a great help in visualizing the structure of the file and included files. If the user would experience it as jarring, then possibly he/she has too many small included files. In such a case, the user can change the file structure by reducing the number of included files. This is exactly what I have experienced when a Teacher Assistent created too many, too small included files, which was changed later on by removing most of the included files.

davbeek commented 9 months ago

A preview refresh is the equivalent of hitting F5 - which causes the page to scroll back to the heading referenced in the url.... 🤔

The strange thing is that I have just tried to replicate the behavior (see https://github.com/swyddfa/esbonio/issues/609#issuecomment-1721219040) by testing many, including very large, files. But all the other files that I have tested behave as they should. So the problem appears to occur only in this one single file, see https://github.com/swyddfa/esbonio/issues/609#issuecomment-1721219040. It is a file that has been generated from Latex source, but I don't see any strange rst code, so I have no explanation for this.

davbeek commented 9 months ago

I think I found the culprit. When I remove all references to :math: from the document, the preview does not jump back to the selected TOC heading after each edit, so that the preview behaves as it should.

In conf.py, I have:

mathjax_path = (
    "https://cdn.jsdelivr.net/npm/mathjax@2/MathJax.js?config=TeX-AMS-MML_HTMLorMML"
)

As extension, I use: sphinx.ext.mathjax.

Note that with :math: code in the document, updating the preview occurs in two steps: first the preview is updated with the change, as it should, but immediately after that, a second update occurs that scrolls back the preview to the selected TOC element.

davbeek commented 9 months ago

Would you also want a more global option that does restart the entire server? Or is doing that via just the command palette enough?

I just noticed that the "Esbonio: Restart Language Server" does not appear to first remove the cache, which is now at one of the subfolders at ~/Library/Caches/esbonio/. I believe that the cache used to be in a subfolder _build of the project. Since the cache is now more difficult to reach, it would be nice to have a command to clear the cache or to restart the server after first clearing the cache. I believe something like this has been asked before as a feature in some other issue.

alcarney commented 9 months ago

I don't see any "bouncing" to the top.

So far I've only seen the bouncing in files that contain an .. include:: directive.

I think I found the culprit.

Thank you for taking the time to track it down! Why the use of MathJax would cause that though I have no idea - will have to investigate :thinking:

I just noticed that the "Esbonio: Restart Language Server" does not appear to first remove the cache, which is now at one of the subfolders at ~/Library/Caches/esbonio/

That (the cache location) should have always been the case.... at least by default. In the previous version there was a large array of options to configure the Sphinx application that Esbonio creates - I suspect you had set esbonio.sphinx.buildDir to use the _build subfolder of the project?

One of the breaking changes in the new version is that all those options have been removed in favour of inferring them all directly from a sphinx-build command. The server will try to guess a sane default placing the build dir in a cache folder so as not to interfere with the project.

But if you prefer you can set esbonio.sphinx.buildCommand to something like ["sphinx-build", "-M", "html", "docs", "docs/_build"] to bring the cache back into _build.

it would be nice to have a command to clear the cache or to restart the server after first clearing the cache. I believe something like this has been asked before as a feature in some other issue.

I agree :)

davbeek commented 9 months ago

So far I've only seen the bouncing in files that contain an .. include:: directive.

I probably missed those.

I suspect you had set esbonio.sphinx.buildDir to use the _build subfolder of the project?

Yes indeed and I see now that the esbonio.sphinx options have all been greyed out, so that they are no longer active. Thanks for clearing this up.

But if you prefer you can set esbonio.sphinx.buildCommand to something like ["sphinx-build", "-M", "html", "docs", "docs/_build"] to bring the cache back into _build.

Will do that, thanks.

davbeek commented 9 months ago

Another small issue: when you click on an internal link in the previewer, so that the previewer changes its view to another rst document, clicking the original rst file in the editor does not make the previewer change back to that original rst file.

Scrolling either the editor or previewer then also does not reactivate syncing. The only way to get the editor and previewer back in sync is by activating another rst document in the previewer.

alcarney commented 9 months ago

Ah yes... the previewer needs to be told how to connect to the server - currently this is done by injecting some information into the URL for the current page. When you click on an internal link, this information is lost and the previewer loses its connection to the server.

Shouldn't be too bad to fix :crossed_fingers:

louking commented 8 months ago

Just went to to 0.91.0 to see if things get better as I was having problems with even the earlier release (my directory has spaces and apostrophes which was confusing things)

I have sphinx in a virtualenv, and I think the esbonio process must be starting without activating the venv.

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "c:\Users\lking\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\__main__.py", line 5, in <module>
    from esbonio.lsp.sphinx import DEFAULT_MODULES
  File "c:\Users\lking\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\lsp\__init__.py", line 52, in <module>
    from .rst import CompletionContext
  File "c:\Users\lking\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\lsp\rst\__init__.py", line 45, in <module>
    from .io import read_initial_doctree
  File "c:\Users\lking\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\lsp\rst\io.py", line 22, in <module>
    from sphinx.environment import default_settings
ModuleNotFoundError: No module named 'sphinx'
[Error - 12:26:50 PM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Error - 12:26:50 PM] The Esbonio Language Server server crashed 1 times in the last 3 minutes. The server will not be restarted. See the output for more information.
[Error - 12:26:50 PM] Esbonio Language Server client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097

alcarney commented 8 months ago

Oh that's interesting, for some reason the extension is trying to start the wrong server. :thinking:

Quick bit of context, the extension is currently shipping with 2 versions of the language server

esbonio 0.x is designed to be launched from within the same virtual environment as your project
esboino 1.x (in development) is designed to be launched separately and attach itself to your project's environment once it has started.

For some reason the extension is trying to launch the original 0.x version.

Can you set the esbonio.server.startupModule option to esbonio.server - that should force the extension to start the correct server.

Assuming that you also have the right virtual environment activated in the official Python extension, the server should then be able to attach itself to the right environment :crossed_fingers:

louking commented 8 months ago

Can you set the esbonio.server.startupModule option to esbonio.server - that should force the extension to start the correct server.

I did this in settings.json and restarted, now don't see the error I reported. I do however see the following, which I saw before as well.

then when I clicked No

Do I need to uninstall reStructuredText (LeXtudio Inc) v189.3.0?

alcarney commented 8 months ago

Do I need to uninstall reStructuredText (LeXtudio Inc) v189.3.0?

At the very least I would disable it in the current workspace while you experiment with Esbonio

louking commented 8 months ago

Yup, now the preview works. It's not displaying my graphviz chart, but that's not displaying on readthedocs.io (I just realized), so I'm sure it's not an esbonio problem. Thanks!

louking commented 8 months ago

FWIW, I've fixed the graphviz problem on readthedocs. This was fixed with https://github.com/louking/tm-csv-connector/commit/b6701fed0056a33c14287f76a4c91e9b917bcaa4 (just the bits on lines 17-18 were enough, I'm sure).

Still esbonio displays the raw graphviz code (no chart) in the preview for https://github.com/louking/tm-csv-connector/blob/3c920236d9a6b3e307151b4c5bad0d8ef2351036/web/docs/source/design.rst?plain=1#L53-L100. Interestingly github also displays the raw code.

See https://tm-csv-connector.readthedocs.io/en/latest/design.html#design for reference

alcarney commented 8 months ago

Still esbonio displays the raw graphviz code (no chart) in the preview

Interesting... did the rest of the preview render ok? Would you mind sharing a screenshot? I would've expected you to see the HTML version of the page complete with the read the docs theme etc.

louking commented 8 months ago

Yes the rest of the preview rendered ok, including use of readthedocs theme. Here are a couple of screenshots. The design.zip file includes the html file my local build produced, which renders like https://tm-csv-connector.readthedocs.io/en/latest/design.html

design.zip

alcarney commented 8 months ago

Awesome thanks for that!

It looks like the source code is set as the alt text for the chart's <img> tag which is not loading/is missing. I assume if you build your docs with sphinx manually and load the page in your browser the chart loads?

I'm wondering if esbonio is missing a build phase or something... :thinking:

louking commented 8 months ago

Yes, the local build's html file I included loads properly. Not sure if it's important, but there are spaces and apostrophes in my path, as you can see from this screenshot.

looks like the _images folder is collecting a bunch of stuff

alcarney commented 8 months ago

Not sure if it's important, but there are spaces and apostrophes in my path, as you can see from this screenshot.

I would be surprised if it mattered since the preview does the equivalent of python -m http.server in your project's build directory - but I'll keep it in mind.

looks like the _images folder is collecting a bunch of stuff

Hmm, at least it looks like esbonio is producing the image file - will have to investigate! Thanks for all the details you've provided! :smile:

louking commented 8 months ago

Not sure it's clear that esbonio produced any image file -- I have been running ./make html pretty often for my testing, especially before I installed the latest esbonio. I wasn't able to get the preview to work before this, or I wasn't looking at it. I could give details but the short explanation is that the version which required esbonio to be installed in the venv would throw errors because of the spaces in my directory name.

louking commented 6 months ago

FWIW, around this time a folder %SystemDrive% started appearing in my vscode file tree, possibly after I used this for a preview. I assume this is trying but failing to translate to C: on windows.

Subfolders look like

alcarney commented 6 months ago

Interesting.... I'll have a look but I'm not sure if that's from anything esbonio is doing directly as those filenames don't mean much to me - unless you have a Sphinx extension that's trying to create them?

Is SystemDrive an environment variable on Windows? Perhaps the language server is accidentally stripping it from the environment when spinning up the sphinx process... :thinking:

louking commented 6 months ago

Yes an environment variable: https://learn.microsoft.com/en-us/windows/deployment/usmt/usmt-recognized-environment-variables

I'm not certain this is related to Sphinx or esbonio. But I noted it started appearing around Oct 15, which fits the timeframe of installing the new esbonio, according to my change log.

2023-10-13 12:19 vscode install Esbonio extension pre-release [v0.91.0]

When I get time, I'll try to reproduce this and see if it's clear.

rpfeifer-tcr commented 6 months ago

Hi, thanks for your work.

I have issue https://github.com/swyddfa/esbonio/issues/699 with 0.91.0. If theres is any further info that would be usefull, I'm glad to help.

alcarney commented 5 months ago

New year, new pre-prelease! v0.92.1 should now be available. (There is now also a standalone pre-release of the esbonio Python package on PyPi (1.0.0b1) if anyone is interested.) It would be greatly appreciated if people can try it out and offer some feedback.

What to expect

Workspace symbol support!
- Using Ctrl+T/Cmd+T you should be able to search for section and titles and directive usages from across your project (assuming the first Sphinx build has completed)
Directive completions
- Completion of directive names has returned, both the insert and replace values for esbonio.server.completion.preferredInsertBehavior have been re-implemented.
- If you use the .. default-domain:: directive at all, esbonio should now, (finally!) respect it when generated completion suggestions.
- Documentation hovers and completion of arguments/options haven't been ported (yet)
Configuration file support
- You can now save options for esbonio in pyproject.toml files under the [tool.esbonio] namespace, useful for sharing settings with collaborators on a project.
- e.g. to set a common build command for your project.
```
[tool.esbonio.sphinx]
buildCommand = ["sphinx-build", "-M", "dirhtml", "docs", "docs/_build"]
```
- Settings unique to you e.g. Python environment are still best set via VSCode
- There are few known issues with this already (e.g. #711) but I'll be interested to see how it works for you.
The beginnings of MyST support!
- Assuming you have the relevant dependencies installed, builds and previews of MyST source files should mostly "Just Work"^TM
- Document symbols, workspace symbols and diagnostics should also mostly work
- Limited support for completions of directives (::: syntax and nested directives not currently supported)
Faster reboots.
- esbonio now uses a SQLite database to ferry data from the underlying sphinx process back to the main server.
- This means after the first build, 90% of the work is already done! You should find the server returns much quicker on the 2nd, 3rd, 4th reboot etc.
New bugs, old bugs and rough edges!

What not to expect

Improvements to sync scrolling/previews.
- I haven't forgotten the points raised above, but they have taken a back seat while working on this round of updates.

neongreen-sc commented 3 months ago

I've tried the prerelease version (Windows, VSCode 1.86, Python 3.11, Sphinx 7.1.2). Live preview, Ctrl+T, problem reporting all work.

One issue I've noticed is that I have to manually restart the Esbonio server after a fresh VSCode start. Otherwise I get this:

connection handler failed
Traceback (most recent call last):
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\protocol.py", line 963, in transfer_data
    message = await self.read_message()
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\protocol.py", line 1033, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\protocol.py", line 1108, in read_data_frame
    frame = await self.read_frame(max_size)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\protocol.py", line 1165, in read_frame
    frame = await Frame.read(
            ^^^^^^^^^^^^^^^^^
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\framing.py", line 68, in read
    data = await reader(2)
           ^^^^^^^^^^^^^^^
  File "asyncio\streams.py", line 726, in readexactly
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\server.py", line 236, in handler
    await self.ws_handler(self)
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\esbonio\server\features\preview_manager\webview.py", line 80, in connection
    async for message in websocket:
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\protocol.py", line 498, in __aiter__
    yield await self.recv()
          ^^^^^^^^^^^^^^^^^
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\protocol.py", line 568, in recv
    await self.ensure_open()
  File "c:\Users\...\.vscode\extensions\swyddfa.esbonio-0.92.1\bundled\libs\websockets\legacy\protocol.py", line 939, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: no close frame received or sent

alcarney commented 3 months ago

Thanks for giving it a try! This looks like a similar issue to #646, I need to look into how to re-establish a connection once it has been interrupted...

alcarney commented 2 months ago

Closing this issue as it has probably served its purpose. I've gone back through all the messages and opened issues for anything that was not covered elsewhere.

Feel free to open an issue if you think I've missed anything

swyddfa / esbonio

Feedback for the Esbonio VSCode extension `v0.92.1` #609

Notes: