Gemini Protocol

[Ygarr]

As it is Gopher support it would be great to add also Gemini(it`s successor and logical developement) gemini://gemini.circumlunar.space/docs/faq.gmi

[textbrowser]

Specifics are available at https://portal.mozz.us/gemini/gemini.circumlunar.space/docs/specification.gmi.

[danrobi11]

+1 Gemini support would be more useful than Gopher now as Gemini is getting more popular. Sry I cant implement that, no coding knowledge here :/

[textbrowser]

It will become stellar when the greasy wheels of corporations grind into it.

[ghost]

+1 Also agree. Gemini is quite nice. Implementing it should not be too hard.

[Maffsie]

I've spent some of the past day or two working on a (frankly embarrassing, but nonetheless) partially-working implementation of Gemini in Dooble, using the previously-implemented Gopher code as a base. It's bad, and on Mac OS it doesn't like self-signed certs, but it kinda works!

My changes are located at https://github.com/Maffsie/dooble/tree/feat_proto_gemini - if I ever get it into a state I don't find embarrassing, I'll submit it as a PR, but I thought it'd be useful to share what I've done so far!

Screenshot:

[textbrowser]

Thank you for the effort. I will review it and adjust it to fit my particular style sometime after you submit your work. :)

[textbrowser]

Why do the back and forward buttons look like they do? Are you using a new style sheet?

[Maffsie]

I think the back and forward buttons looking like that are due to having the tab bar at the bottom of the screen (or at least I didn't notice until I set that preference).. I never thought much of it. I'll try with a fresh profile and the latest release tomorrow to see if it's reproducible, assuming it's a bug.

[Maffsie]

I wiped my profile and opened up the latest released version of Dooble and confirmed; the strange styling on the back and forward buttons appears when the tab bar's location is set to "South". Do you want me to raise this as an issue? :)

[textbrowser]

I wonder if this appearance is caused by your distribution's styling. For my distribution, the buttons appear normal regardless of the tab bar's location.

[Maffsie]

I'm running on Mac OS 12.1 ("Monterey"), I tested with an Alpine Linux + KDE virtual machine I had and the issue doesn't occur there so I suppose it's a Mac-related bug.

[textbrowser]

It does look like Mac. :) Let me see.

[textbrowser]

I see it too. It's like a remnant of the tab's corner widget. East and West are fine.

[Maffsie]

It's a very strange thing, to be sure. I'm not sure what causes it, I tried to look into it while I had Qt Creator open, but the QML tooling is just incomprehensible to me :(

I've made more progress on Gemini support! MIME types are now handled more appropriately, more of the document specification is implemented, etc. Redirects are (in theory) handled. Still to do is:

• Handle 1x user input responses, but I don't have any idea how to prompt for that.. it works differently from Gopher, where the implementation was able to just present it as a form as part of the page, while there is no page in Gemini, only the prompt message.
• Handle 3x redirects properly; currently all redirects are blindly honoured. The implementers guide (https://gemini.circumlunar.space/docs/best-practices.gmi) suggests that the user should be prompted upon any redirect (or perhaps be able to set a preference for whether to follow/prompt/refuse to honour redirects), and cross-protocol redirects should be treated with a high degree of suspicion, and the user should always be warned about redirects that result in an unencrypted load. Permanent Redirects should probably also be cached in-browser, I'm not sure.
• Handle 4x temporary error and 5x permanent error responses; these should present the regular browser error page, with the actual error and any additional information supplied via the Meta field
• Handle 6x client certificate messages; I do not have the energy to even think about client certificates..
• Handle the quote format specifier for text/gemini pages
• Handle server certificates better; the spec suggests trusting the fingerprint of a given server on first sight, and prompting if the fingerprint ever changes (a-la OpenSSH), and I feel like this is a good way of doing it if the certificate is self-signed. If the certificate is not self-signed, I feel like it's worthwhile doing the usual validation against certificate authorities.
• Handle text/markdown documents. In the Gemini Client Torture Suite (gemini://random-projects.net/torture/), there are markdown documents that are currently handled as plaintext. This..might be best done via a new feature request, as the browser independently supporting markdown might be a useful feature?
• Handle mixed text/gemini and image/svg. In the torture suite linked above, there are mixed-type documents. I'm not actually sure how these are intended to be handled, but their existence implies that support for this is expected in an 'advanced' Gemini client.
• Figure out why, on Mac OS, QSslSocket TLS handshake errors -cannot- be ignored, when they can be ignored if they occur during an HTTPS request.

[textbrowser]

I wiped my profile and opened up the latest released version of Dooble and confirmed; the strange styling on the back and forward buttons appears when the tab bar's location is set to "South". Do you want me to raise this as an issue? :)

Added as a reminder.

[textbrowser]

It's a very strange thing, to be sure. I'm not sure what causes it, I tried to look into it while I had Qt Creator open, but the QML tooling is just incomprehensible to me :(

I've made more progress on Gemini support! MIME types are now handled more appropriately, more of the document specification is implemented, etc. Redirects are (in theory) handled. Still to do is:

* Handle 1x user input responses, but I don't have any idea how to prompt for that.. it works differently from Gopher, where the implementation was able to just present it as a form as part of the page, while there is no page in Gemini, only the prompt message.

* Handle 3x redirects properly; currently all redirects are blindly honoured. The implementers guide (https://gemini.circumlunar.space/docs/best-practices.gmi) suggests that the user should be prompted upon any redirect (or perhaps be able to set a preference for whether to follow/prompt/refuse to honour redirects), and cross-protocol redirects should be treated with a high degree of suspicion, and the user should always be warned about redirects that result in an unencrypted load. Permanent Redirects should probably also be cached in-browser, I'm not sure.

* Handle 4x temporary error and 5x permanent error responses; these should present the regular browser error page, with the actual error and any additional information supplied via the Meta field

* Handle 6x client certificate messages; I do not have the energy to even think about client certificates..

* Handle the quote format specifier for text/gemini pages

* Handle server certificates better; the spec suggests trusting the fingerprint of a given server on first sight, and prompting if the fingerprint ever changes (a-la OpenSSH), and I feel like this is a good way of doing it if the certificate is self-signed. If the certificate is not self-signed, I feel like it's worthwhile doing the usual validation against certificate authorities.

* Handle `text/markdown` documents. In the Gemini Client Torture Suite (gemini://random-projects.net/torture/), there are markdown documents that are currently handled as plaintext. This..might be best done via a new feature request, as the browser independently supporting markdown might be a useful feature?

* Handle mixed `text/gemini` and `image/svg`. In the torture suite linked above, there are mixed-type documents. I'm not actually sure how these are intended to be handled, but their existence implies that support for this is expected in an 'advanced' Gemini client.

* Figure out why, on Mac OS, QSslSocket TLS handshake errors -cannot- be ignored, when they can be ignored if they occur during an HTTPS request.

How is the progress?

[textbrowser]

Anything?

[Maffsie]

@textbrowser Sadly I've not had much time or energy to work on this since my last comment :(

[textbrowser]

S'OK. Maybe it'll be added to a better browser. :D

[zzo38]

Other ideas related to this:

• Note that text/gemini files are not limited to Gemini protocol, and can also be used with HTTP(S) and with local files (with the file: scheme).

• Likewise, HTML and other file formats can also be used with Gemini protocol, although the end user may wish to disable some file formats.

• If Markdown is implemented (in any protocol), then you should handle parameters in the MIME type, so that if it is CommonMark then it will work; otherwise, there are variants of Markdown and might be badly handled (Gemini file format does not have this problem, though).

• Some people, including myself, have thought that mandatory TLS is a bad idea (it should be optional). Therefore, I propose to add insecure-gemini: scheme, which is like gemini: but without TLS, and 6x responses are not valid. (It should not be a problem to use the same port number (since it cannot be confused with a TLS message), although I do not know that any server software supports this.)

• There should be some way to identify MIME types in user CSS, so that user CSS can be applied differently by MIME types.

• See issue #153 if you want to be able to display Gempub files.

[Maffsie]

@zzo38 I think it may be beyond scope to define an insecure-gemini: protocol. I'm aware of the discussion around gemini's mandatory TLS, but anything not in the spec should probably be left until after the code can behave like a proper Gemini UA. FWIW I have started picking this back up, but it's been long enough that I don't currently have a working Qt dev environment, so I've been fixing that in the background first.

[zzo38]

Maybe you are right, although I think that it would be a good idea. However, maybe after more discussion, people might agree to use a different name than insecure-gemini: for the non-TLS Gemini protocol.

(Currently I cannot access the newsgroup for Gemini because aioe seems to be down, so I cannot post any further messages at this time.)

[textbrowser]

This is abandoned and incomplete. Closing and thanks and see you whenever.

[zzo38]

@Maffsie further comments:

• To handle 1x responses, my idea is to make a HTML file with <html><isindex><plaintext> and then the prompt text. (This won't hide passwords, but I think that the 11 status code is a bad idea anyways, and I am not the only one who believes that. The Scorpion protocol lacks the 11 status code, for this reason (but has a different way to handle passwords).) (If the <isindex> command and/or <plaintext> command is not implemented, then you should add that command so that it will work.)

• About 3x responses, I think that it is helpful to not blindly honour redirects regardless of the protocol. Setting a limit before it prompts the user, and making some redirects always prompt the user, would be better. This is helpful for HTTP(S) too, but possibly the default limit might be higher for HTTP(S). This limit might be configurable by the user; zero means always prompt for redirects.

• About server certificates, I think this should be configurable by the user.

• About Markdown, if it is implemented then it should apply to all protocols (including local files) and Gemini files also should. However, Markdown is not required in order to correctly implement Gemini. I believe that doing so is acceptable, however.

• About SVG, I think that Gemini isn't supposed to do mixed text/gemini with image/svg in one file, although such a thing is possible.

• About mandatory TLS, there are other protocols such as spartan: and scorpion: which are similar than Gemini in some ways but different in other ways, and neither has mandatory TLS (Spartan has no TLS; Scorpion has TLS only if the scheme is specified as scorpions:). However, note that Spartan defines a new command in the file format even though it is the Gemini file format (however, I do not expect that it should be a problem to implement it in all Gemini files regardless of the protocol, so if Spartan is implemented then so should the extra command in the Gemini file format, I think).

• About Mac OS, I don't know.

• If a table of contents menu is not already implemented, it might be a good idea to implement it, for any formats that it is appropriate to be implemented (HTML, Gemini, Scorpion, Markdown). (In the case of HTML, it would list the <h1>, <h2>, etc)

• Dooble claims to be a "scientific" browser. While this is not relevant for implementing Gemini and Spartan (as far as I know), it is relevant for implementing Scorpion; any conformant Scorpion browser must either implement the database format described in that document (which is normally optional) or not claim to be "scientific".

[Maffsie]

@zzo38 I haven’t personally heard of the Scorpion or Spartan protocols, but I’d expect they’d be implemented in a separate ticket since this one’s just about implementing Gemini support, and my preference is to follow the letter of the protocol’s formal specification as much as possible rather than deviate and potentially violate a user’s expectations - it’s not really the UA’s job to have opinions.

@textbrowser I had intended to continue working on the implementation of Gemini but ran into some kind of fatal issue when compiling - unsure if my Qt toolchain was broken or something. I’ll see if I can do something about that now, but if it’s not something I can figure out myself, are you willing to offer some help in getting it building again?

[zzo38]

@Maffsie You are right; however, Spartan also uses the Gemini file format. So, the implementation of the Gemini file format only needs to be implemented once and then it can be used with any protocol if the MIME type is text/gemini (including local files if the .gmi suffix has been configured as text/gemini). (If a separate branch or fork is made, then you can copy the file to implement Gemini file format to both branches/forks, and should ensure that they both use the same implementation so that they will not conflict if you try to merge both into the main code.)

About the details of Gemini, yes it should follow the specification (and I have mentioned some things about it, in order to perhaps help with such a thing), but even if configurable, the defaults can be; e.g. it can set a default redirect limit of 5 for Gemini, since that is what the specification says. SVG is not mentioned in the Gemini specification at all as far as I can tell, so mixed Gemini/SVG files are not proper according to the specification as far as I know, so is probably not necessary to be implemented.

[textbrowser]

Dooble = two. Gemini = two.

[textbrowser]

I requested more information about jar. Like, how do you want the files presented? jar is a difficult problem:

• a browser has a history and a jar scheme results in an external program and a result in the browser
• how do you cancel a jar?
• what do you do with huge jars?

The original Dooble included a terminal interface. In my view, a jar is two things. It's not like viewing a video because Dooble doesn't need to load the entire video. You can't read a jar unless you take out all of its contents. A 100 GiB jar? What's Dooble supposed to do? It's a nice request but the solution isn't.

[textbrowser]

Gemini. A list of public servers is required. If we can't access servers, the protocol is going to be whatever you think it is. If you think it follows the protocol, OK. But if you can't repeat it, well, it's broken.

[textbrowser]

The implementation has to work like gopher. In Dooble, gopher is like http. History, source view, stop, etc. You don't need to compile Qt. It's not an add-on. The implementation is the browser.

[textbrowser]

I requested more information about jar. Like, how do you want the files presented? jar is a difficult problem:

* a browser has a history and a jar scheme results in an external program and a result in the browser

* how do you cancel a jar?

* what do you do with huge jars?

The original Dooble included a terminal interface. In my view, a jar is two things. It's not like viewing a video because Dooble doesn't need to load the entire video. You can't read a jar unless you take out all of its contents. A 100 GiB jar? What's Dooble supposed to do? It's a nice request but the solution isn't.

Actually if a jar is like a tar, you can view the files without decompressing. You can sneak attack the cookie jar!

[textbrowser]

Yeah, I see the implementation.

1. jar is the scheme.
2. You sneak attack the file: a listing is produced.
3. You place that list in a nice little HTML page with formatting and tags and such.
4. History, source-view, reload, stop work magically.

The remaining huge problem is if you want to get a particular file. So #3 includes links into the jar. If a link is clicked, you have to sneak attack the jar and only extract that file and emit a save-as signal. Probably impossible.

[textbrowser]

Gemini is simple with a test server. It's hopefully a well-defined protocol with good tokens and perhaps a language. If it's like HTML, yikes. That's a browser engine with years and years and decades of work. I read a bit and it looks like gopher.

[textbrowser]

The TLS layer is addressable through Qt and maybe (?) QWebEngine. WebEngine supports other schemes, but I don't know if supports other schemes which require TLS. If it does, then things work magically. You just get text from the server because WebEngine handled the ciphering. If it doesn't then you need to download the data through QNetworkReply. That does the TLS. More work though.

[textbrowser]

The gopher implementation in Dooble is through a QTcpSocket. There is also QSslSocket. I prefer QNetworkReply here because it's a URL you're fetching. I may be convincing myself to replace QTcpSocket with QNetworkReply.

[textbrowser]

@Maffsie You are right; however, Spartan also uses the Gemini file format. So, the implementation of the Gemini file format only needs to be implemented once and then it can be used with any protocol if the MIME type is text/gemini (including local files if the .gmi suffix has been configured as text/gemini). (If a separate branch or fork is made, then you can copy the file to implement Gemini file format to both branches/forks, and should ensure that they both use the same implementation so that they will not conflict if you try to merge both into the main code.)

About the details of Gemini, yes it should follow the specification (and I have mentioned some things about it, in order to perhaps help with such a thing), but even if configurable, the defaults can be; e.g. it can set a default redirect limit of 5 for Gemini, since that is what the specification says. SVG is not mentioned in the Gemini specification at all as far as I can tell, so mixed Gemini/SVG files are not proper according to the specification as far as I know, so is probably not necessary to be implemented.

Yup. You can implement the protocol as a bunch of downloads or a single download. A single download is easier because it's clear that it is. Multiple downloads are how modern browsers work. But since this is gemini, it's supposed to be easier to implement and better on dial-ups, right?

[dryeo]

On 01/13/24 10:16 AM, textbrowser wrote:

Actually if a jar is like a tar, you can view the files without decompressing. You can sneak attack the cookie jar!

Jar files are basically zips with certain parameters. There's a table of contents and you can extract one file if needed.

[dryeo]

On 01/13/24 10:19 AM, textbrowser wrote:

The remaining huge problem is if you want to get a particular file. So

3 https://github.com/textbrowser/dooble/issues/3 includes links into

the jar. If a link is clicked, you have to sneak attack the jar and only extract that file and emit a save-as signal. Probably impossible.

Mozilla does (did?) it fine. Packaging the browser, make package, sees a lot of js, properties, css, xul etc all packaged into a jar or IIRC, a newer container that is similar. Same with extensions, they usually come as jars. The head of a jar here,

Archive: 
Z:/mozprofiles/Mozilla/SeaMonkey/Profiles/2nf8z26q.new/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/chrome.jar
  Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
     4048  Defl:X     1307  68% 01-01-2010 00:00 52cf586e 
calendar/content/calendar/WindowsNTToZoneInfoTZId.properties
    40402  Defl:X     9276  77% 01-01-2010 00:00 ddca6376 
calendar/content/calendar/agenda-listbox.js
    12794  Defl:X     2608  80% 01-01-2010 00:00 12063f46 
calendar/content/calendar/agenda-listbox.xml
     1756  Defl:X      909  48% 01-01-2010 00:00 70001015 
calendar/content/calendar/calApplicationUtils.js

Possible they use zlib

[zzo38]

The jar: scheme can be used with any ZIP file, regardless of its name. This means that it can also be used with Gempub and EPUB files. Also note that it is possible for the implementation to restrict the jar: scheme to local files if you do not want to deal with downloading; being only for local files should be suitable for most uses.

[textbrowser]