Totto16 commented 2 months ago

Overhaul code highlighting

Description

As described in #271 code highlighting should become optional, so that user need to install an additional package or something similar to make it work.

There are a few options, I'll test all of them and list benefits from them and shortcomings

Overview

We need an option that can do these things:

detect languages, say how "confident" (highlight.js calls it relevance) the guess is, based on content only (no file extension or similar things, or shebangs)
format languages to Pango markup or something similar, that we can convert to it
provide custom styles or use a set of predefined styles or be easy to parse / and or add custom styles to it
support a few languages, not just 1,2

Our options are:

1: Use a command-line highlighter, that comes with many distributions, it can be installed and is then detected by Pano, to use it
2: use a JavaScript package as before, the user needs to install files into a correct location, and then Pano can detect it.
3: make a custom deb / rpm / arch etc. package with Pano and built-in highlighting, that installs the needed package automatically

1. Option

Pros:

the packages are quite robust and have many languages
it is easy to install for the enduser.

Cons:

we need to "shell out" e.g. spawn a shell process, that is more complex and error-prone than other methods

Feature Table

package name	detection	many languages	markdown output	styleable
source-highlight	❌	✔️	⚠️	⚠️
pygments	⚠️	✔️	✔️	✔️

Notes

source-highlight:

It can not detect languages without filename or shebang, it also can't tell you, how confident it was (there is a option to infer the language, but it only works on file extensions / shebangs or similar metadata) It can't output to markdown, but to simple html, which would be styleable in some way You can specify custom css or similar things, but it's not that easy, but maybe doable.

pygments(python3-pygments)

It is good at detecting creating languages (like python) but when I tried JS, or HTML or some other ones it failed ... i didn't test 100 files, but it wasn't that good, the one good thing was, that it's better than highlight.js in many cases and the detected languages all seem to look similar, when highlighted. It has support for many many languages, it is easy styleable and can output pango markdown out of the box.

update: after some modification to the detection logic, it seems really stable and robust when working with more than ~ +- 200 characters. The shortcoming of it is, that it has no real relevance value, but we can emulate that, by using that charaacter length. After that it is really good.

2. Option

Pros:

it is easy to integrate
it has many options
many highlighter have many languages and are quite good

Cons:

If we don't ship them, it's not easy for the end-user to install

3. Option

Pros:

We can pre package everything we need, the cons of the 2. Option or the previous implementation go away
We can select option 1 or 2 as core implementation

Cons:

We would stop publishing on EGO or the package published there is not complete and maybe, depending on the use option (1 or 2) it might be not easy to add the code highlighting to Pano again by hand

Current state

I would focus on option 1, since it's the best choice atm. I tested source-highlight but it's quite bad. So I'll be using pygments(python3-pygments) for now.

TODOs:

[ ] fix all bugs with the new implementation
[ ] fix slider (Gtk4.Scale) Size in Settings
[ ] more testing
[ ] fix search of styles in settings

optional:

[ ] Improve pano theme for pygments

Type of change

[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

Checklist

[x] My code follows the style guidelines of this project
[x] My commits follow the commit standards of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have checked my code and corrected any misspellings

Totto16 commented 2 months ago

@oae Do you have an auto branch protection rule in some way, if the branch isn't named feat/ (is that even possible) Since I am currently unable to push (it fails with the reason, that this is a protected branch ❓) grafik

oae commented 2 months ago

The first option with pygments sounds good to me too

oae commented 2 months ago

@oae Do you have an auto branch protection rule in some way, if the branch isn't named feat/ (is that even possible) Since I am currently unable to push (it fails with the reason, that this is a protected branch ❓)

I have changed some settings now. Can you try again?

Totto16 commented 2 months ago

The first option with pygments sounds good to me too

@oae Do you have an auto branch protection rule in some way, if the branch isn't named feat/ (is that even possible) Since I am currently unable to push (it fails with the reason, that this is a protected branch ❓)

I have changed some settings now. Can you try again?

Same error, strange, it seems, that I can't push to any branch on this repo, but I could earlier this day 🤔

Edit: indeed, it is fixed ❤️

oae commented 2 months ago

I think it should be fixed now

HearseDev commented 1 month ago

Not really experienced with anything gnome, but was wondering if this would be a viable and much cleaner solution. It could be an optional lib install and also has bindings for js. Or it could be required and could be used as a default text view?(think it might be preinstalled with gnome, not too sure) I am just throwing out an idea, never really worked with anything remotely close to this. https://wiki.gnome.org/Projects/GtkSourceView https://gnome.pages.gitlab.gnome.org/gtksourceview/gtksourceview5/class.PrintCompositor.html

Totto16 commented 1 month ago

Not really experienced with anything gnome, but was wondering if this would be a viable and much cleaner solution. It could be an optional lib install and also has bindings for js. Or it could be required and could be used as a default text view?(think it might be preinstalled with gnome, not too sure) I am just throwing out an idea, never really worked with anything remotely close to this. https://wiki.gnome.org/Projects/GtkSourceView https://gnome.pages.gitlab.gnome.org/gtksourceview/gtksourceview5/class.PrintCompositor.html

I think I saw this in my research, but It isn't half as good as pygments in autodetecting languages and it's ballpark is nearly the same as source-highlight.

It has integrated GTK support, but it's not themeable (at least easily) and it has no "real" autodetection feature, you have to create a widget to get the language, which is not suitable for our purpose 😓

PiotrGrobelak commented 2 weeks ago

Hi @oae and @Totto16, what can I pick up to help you with for the full release of GNOME 46?

oae / gnome-shell-pano