Does not work with TB 115

[m707]

I tried the most recent build, but after hitting connect I just get:

An unexpected error occurred

:(.

[chris21k]

Does not work with the TB 115.0 release version.

[creative-coder]

try this, its a pached version and works... https://trella.info/allgemein/thunderbird-sieve-filter-funktioniert-nicht/

[m707]

Not for me in TB 116.0b5 (updated in the meantime):

Connection to server failed :(, ... or in our $LANG: "Verbingung zum Server fehlgeschlagen ;). An unexpected error occured.

Are you sure you got that version working with a TB version >= 115?

[creative-coder]

my mistake, im running 102.13.0.. but works at this version (sry was late tonight)

[vsatmydynipnet]

Found out in https://github.com/thsmi/sieve/pull/768 too. too. Looks like we need help of a dev to overcome this and be able to use 115.

[thsmi]

There was a pretty massive and breaking change in Mozilla's socket implementation. Refer to the following bug for more details: https://bugzilla.mozilla.org/show_bug.cgi?id=1791633

Beside this the following change also needs to be addressed https://bugzilla.mozilla.org/show_bug.cgi?id=1780695

And of course the other bugs which are blocking the 102 release...

So don't expect a release anytime soon, unless the unlikely happens and someones steps up and provides fixes...

Until then the only suggestion I can give is to use the standalone app.

[sachaz]

Thanks for the reply & letting us know whats going on <3

[jobisoft]

Hi,

the drop of Services.jsm is not much work: we do not have to include it at all anymore. It is now a globally available variable.

For the socket, I would like to move away from the internal implementation and use a JavaScript library. But I need a server to test against. Is anybody here able to provide me a test account/access? You can reach out to me directly via john@thunderbird.net

Thanks!

[jobisoft]

@thsmi

I have not understood why the WebExtension cannot use the same trick you use in the Web App, to connect to the server, or why the WebSocket API cannot be used.

I must be missing something crucial and would be happy if you could help me to understand the situation better. If your new Web App is based on node.js, should you not be able to webPack it into a Firefox/Thunderbird extension?

[thsmi]

@jobisoft well it is for the same reason why Thunderbird does not use the WebSocket API for IMAP, POP3 or SMTP. Those protocols are like SIEVE based exclusively on TCP not on HTTP. Thus WebSockets can't be used, they work only with HTTP Servers.

Thunderbird does not provide an API for extension to create TCP connections. So you need to use an experiment to open TCP Sockets. But every major release introduces a breaking change to the Socket API. And because the socket API is completely incompatible to the node.js socket implementation webpack don't work.

The WebApp works like a webmailer which talks to you browser in http while it talks to your mailserver in imap and smtp or respectively SIEVE. It is basically a WebSocket to SIEVE translator or proxy. The WebApp provides both the http server and the proxy. This allows a webbrowser to magically communicate with a SIEVE endpoint.

But this does not work with Thunderbird, because it is a client and not a server. Thus the only option to use websockets is that you have somewhere a server where you could run such a WebSocket to SIEVE proxy. Thus the only real world option is to use a very classic socket connection. And this is why you have to use an Experiment.

And yes you could try to standardize Sieve over WebSocket. But then the hard part starts, how to convince the mail server implementations to implement this? There is no use-case other then Thunderbird for this. Normal Applications just use a standard socket connection and WebApp anyhow need an HTTP Server to provide the app to the browser and thus you can easily provide a proprietary websocket to sieve proxy as side channel....

I guess you see why Thunderbird is so special. It officially offers no sockets support to extension, only HTTP or WebSockets. And Mailservers typically don't provide HTTP or Websockets endpoints for security concerns. Without Experiments there is no overlap and the Extension would be long dead.

The real pain point here is that Mozilla introduces with each major release a breaking change for their socket implementation. I don't know why but it seems to be a long standing tradition. Node.js in contrast provides a rock solid API for TCP Sockets. there haven't been any breaking changes since many many years. But this is something which can't be changed...

[madduck]

Does anyone know of any progress on this?

[ecogit]

Is there a way to donate as a token of gratitude for the efforts by @thsmi and his fellow contributors? ☕️

[madduck]

See here: https://github.com/thsmi/sieve#status

I've offered to @thsmi that my company could pay him to get sieve up to speed and am hoping to get a reply soon.

[thsmi]

Found some time to look into this and prepared a patch.

But keep in mind it is not really tested. It seems to connect but could fail at other spots And the drag in drop in the graphical editor is still badly broken in Thunderbird. Which makes it unlikely that there will be an official release soon.

You can get a nightly build from this link. https://dev.azure.com/thsmi/1164eec3-b870-4973-8110-5e275d208aac/_apis/build/builds/19977/artifacts?artifactName=XPI%20-%20Thunderbird%20WebExtension&api-version=7.1&%24format=zip

This build does not work with thunderbirds daily builds. Because of the following bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1851294 . It will instantaneously crash Thunderbird Daily Release when starting the TLS session.

[madduck]

I gave it a bit of a whirl, and it runs very nicely. Drag and drop works too, or at least I have not run into any problem thus far.

Thank you! I've just sent over a little donation. Get in touch at mail ät martin-krafft dot net if you want to explore options to fund an official release…

[blacklion]

Nightly build works for me in 115, thank a lot! I'm using only text editor (not visual one), so can not report status of drag'n'drop

Thank you again.

[tuxick]

Found some time to look into this and prepared a patch.

But keep in mind it is not really tested. It seems to connect but could fail at other spots And the drag in drop in the graphical editor is still badly broken in Thunderbird. Which makes it unlikely that there will be an official release soon.

Great, works for me on 115.4.1. Thanks a lot.

[elfloo]

Found some time to look into this and prepared a patch.

But keep in mind it is not really tested. It seems to connect but could fail at other spots And the drag in drop in the graphical editor is still badly broken in Thunderbird. Which makes it unlikely that there will be an official release soon.

You can get a nightly build from this link. https://dev.azure.com/thsmi/1164eec3-b870-4973-8110-5e275d208aac/_apis/build/builds/19977/artifacts?artifactName=XPI%20-%20Thunderbird%20WebExtension&api-version=7.1&%24format=zip

This build does not work with thunderbirds daily builds. Because of the following bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1851294 . It will instantaneously crash Thunderbird Daily Release when starting the TLS session.

Didn't manage to catch your link up, will be glad if you'll be able to share fresh sieve webext build again, current link shows that build cant be found.

Big thank you in advance for your time and work!

[SuperTux88]

@elfloo I still had the file I downloaded in October in my downloads folder: XPI - Thunderbird WebExtension.zip

[adlerweb]

Instead of using a potentially outdated XPI you can also download the latest nightly as described in the FAQ:

1. Head over to https://dev.azure.com/thsmi/sieve/_build?definitionId=5
2. Click on the latest (successful) build
3. Click on Jobs→Web Extension
4. Click on "1 artifact produced"
5. A "XPI - Thunderbird WebExtension" should be published. Open the menu at the right side and choose "Download artifacts"

Keep in mind: These builds are untested and might contain severe bugs.

[ploink]

@elfloo I still had the file I downloaded in October in my downloads folder: XPI - Thunderbird WebExtension.zip

Thanks! I tried several versions, including the latest nightly but only this one worked. Others do not load the server configuration.

[elfloo]

@elfloo I still had the file I downloaded in October in my downloads folder: XPI - Thunderbird WebExtension.zip

God bless you! Will try out a bit later, you're a lifesaver

[TonyGravagno]

The latest build from 13-Dec-2023 hangs on the Connecting process. As seen in the mention from #891 I'm trying to shift to using the Windows App while the WebExtension in Thunderbird is getting attention, but without config info that is difficult.

[thsmi]

To avoid confusion the patches which try to restore thunderbird compatibility are only available on the branch linked to this ticket https://github.com/thsmi/sieve/tree/893-does-not-work-with-tb-115

Means only nightly builds which are triggered by a change on this branch will contain any kind of patch regarding Thunderbird 115 compatibility.

Concerning Thunderbird 115+ compatibility issue and why it crashed on Thunderbird nightly builds there are some updates. The problem is that the Thunderbird core changed. You can trigger a startTLS only from the network thread otherwise it will crash. At the same time java script can only execute stuff on the main thread otherwise Thunderbird will crash. So it is a self-fulfilling prophecy always ending in a crash.

Technically it would have been a dead end for the extension but there are good news at this point. SMTP and other protocols are reimplemented in JavaScript by the Thunderbird developers. Which means they need as a base a maintained and reliable Socket implementation. Which can be reused by this extension.

So I started an experiment I changed the backend to this TCPSocket implementation. And it was way less work than expected.

The link to a nightly build which supports TB115+ is below: https://dev.azure.com/thsmi/1164eec3-b870-4973-8110-5e275d208aac/_apis/build/builds/20027/artifacts?artifactName=XPI%20-%20Thunderbird%20WebExtension&api-version=7.1&%24format=zip

The zip files for nightly builds are deleted on Azure DevOps after 6 weeks here is also the zip file: XPI - Thunderbird WebExtension.zip

Keep in mind when testing it, it still is very rough and will contain bugs.

While this is some kind of a step towards a release. The big show stopper is still the broken broke graphical editor. This is work in progress and there is no ETA when it will be fixed..

[tessus]

While this is some kind of a step towards a release. The big show stopper is still the broken broke graphical editor. This is work in progress and there is no ETA when it will be fixed..

I might be out of line here, but let me suggest something (while explaining how and why I came to that idea).

I loved the TB extension for one reason: I was able to edit my sieve filters directly from my mail app without having to SSH into my server. I never cared for the graphical editor. In fact I have never even tried to use it. I only use the Source tab in the app.

I do understand that the graphical editor was meant to be a major reason for this app/extension. I don't have a problem to use the standalone app, but using it from within Thunderbird is just a lot more convenient.

How about you get rid of all your headaches by removing the graphical editor from the web extension? Currently there is no compatible version on the add-on website, and you refer to the standalone app anyway. Just call the web extension a "light" version of the standalone app and add a note to the extension to use the standalone app for that psrticular feature.

I believe this is a viable solution and IMO it makes perfect sense to use a "light" web extension.

[thsmi]

The sentence you quoted was phrased a bit sub-optimal, and leaves room for interpretation thus some clarification.

The show stopper number one for a TB115 release is, that you crash TB when you start a secure socket connection from javascript. The only fix is to migrate the implementation to TCPSocket API. First step, a prototype, is done but it is still very very rough and a long way to go. And this is work in progress but there is no ETA when it is fixed. It also depends e.g. if and when TCPSockets are exposed directly to webextension.

... and of course there is the graphical editor issue where a light extension version would be debatable option. But concerning this no work on this really started.

[tessus]

Thanks for the clarification.

The only fix is to migrate the implementation to TCPSocket API. First step, a prototype, is done but it is still very very rough and a long way to go. And this is work in progress but there is no ETA when it is fixed.

Is that the zip file you posted on Dec 28th? It works on my installation. Haven't had any issues with it yet. (I can't change the port, but I use the standard port anyway.) Personally I wouldn't mind to just use a beta version until all the kinks have been resolved. I believe it is even possible to mark add-ons as beta or pre-release, if you want to expose more people to it and have them send you bug reports.

there is the graphical editor issue where a light extension version would be debatable option.

I think it's better to have a light version than none at all. Since I am not using the graphical editor, this is of course subjective. But maybe you can start a poll to gauge the public opinion.

[thsmi]

Concerning the zip from Dec 28th, yes it was just a quick hack to test if it is feasible. It worked out way better than expected. Therefore I published it. As good as it may look, it does not care about any kind of a errors. Means in case you run into any kind of connection related errors you won't get useful error messages and the extension stalls in a partly connected and disconnected state. The exceptions are either swallowed or are not caught. And fixing and cleaning this up needs lost of time the classic 80/20 rule...

Beside this I started in January at Thunderbird the proposal to expose the TCPSocket API directly to Addons. Th whole process moves due to lack of time and resources way slower than wanted. I promised a proposal for exposing it to Extensions which is still work in progress and this proposal will most likely will change several times before it the hopefully gets stable and integrated into Thunderbird. If this lands the extension does not need to use experiments anymore to implement features. Which should make it more robust for in the future.

But this means it is the classic dilemma short hack vs long term solution. From a short term view, which focuses only on the WebExtension, it would be just stick with experiments and use The TCPSocket API from the core. And then whenever the WebExtension API for socket lands, redo the work and switch everything to this API. The long term view is to first focus on getting a stable API into Thunderbird and then update the Extension to this update. So that work does not need to be done twice.

And as especially the resource time is limited, only the second option seems to be viable to me. Trying to get a stable Socket API into Thunderbird WebExtension API and then investing time to use this from the Extension.

When the Extension has again a reliable base, it would make sense to release a light version. But this is not too likely to happen soon, because it requires a ton of work to be completed...

[sachaz]

Thanks very much for all you efforts to make it possible <3 <3 <3 As we can see: Thunderbird is not really careful on all the community of plugins developers /o\

[tessus]

@thsmi I am grateful and happy to use the zip file until then. It is more than sufficient to quickly make a change in your sieve file. I only kindly ask that you create one more/last zip file where the add-on uses a big compat version number in the manifest, so that it doesn't stop working when the next (and the next, ...) TB release lands.

[thsmi]

@sachaz The Thunderbird is community driven. Mean if you want something as a developer, you need to just propose and do it. But the downside is that it will eat up time from your project. But this is the nature of non commercial opensource...

@tessus good point as there hasn't been a release for long time, I have not touched the manifest for long time.

[sachaz]

@thsmi sure I was related to the lasts Thunderbird changes making the rebuild of this plugin.

[sa0mad]

So, I took the latest build for a test-spin, and it seems to work just fine for me. Will just have to see if the new mails also is acted by the sieve, but it seems the others are just fine. Hope this ends up returning to the Extentions as available before.

[ForsakenHarmony]

@thsmi seems like the nightly build expired as azure probably only stores them for a certain amount of time, any chance to get a new build or even better a release?

[sphakka]

Ping! Where did the nightlies end up? The build list is indeed empty... And the XPI mentioned above seems to dislike my TB-115.8.0: it says it's corrupt :-/

[TonyGravagno]

A note for @thsmi : Thanks.

I look at how hard Thomas works on this stuff and I really appreciate it. Thanks man. You are also patient with your users here. You have explained the fundamental issue clearly so that we can understand your challenges. Your frustration with Thunderbird core is evident, and how it affects extensions like yours.

Yeah, this is FOSS. We get it. The kick in the pants is that some "Free and Open" code is really not so free (liberty) for forking to avoid problems like this, and the people and processes are not as "open" about the internal processes as we'd like. This goes for most major platforms that are owned and managed by big entities - they are free and open for us to see, understand, and maybe work around the problems, not so free or open for us to do much about them. That's just the way it is. I appreciate your position in this awkward situation.

[TonyGravagno]

I actually came here today just for this comment:

The pull https://github.com/thsmi/sieve/pull/917 into master includes changes that would be valuable for the TB 115 branch. Actually there are a number of good changes that have been made and that branch has fallen behind. It would be nice if master could be merged back into TB 115 to rebase the branch.

[TonyGravagno]

While looking at filters, I was thinking about the issue with TCP Sockets. The way I always work around these issues is with a proxy. If the only connectivity/protocol available is HTTPS, then HTTPS a JSON request to a middle-tier service that then makes the actual socket request, and return a structured JSON response which is parsed for subsequent processing and rendering. Coming back here, I see @thsmi already mentioned doing something like this - could you expand on your thoughts in this area?

Thinking out loud (not always a great thing to do but here we go...) :

What I'm thinking there is that ManageSieve requests can be proxied through a local instance of Node which then makes the actual protocol-compliant TCP request. For those who don't want the local option, proxy through a remote connector. That option makes me squirm because of the security implications, but it's an option in the right environments including secure VPS and containers. This could even be hosted by email service providers - or at least by those of us who self-host our MX.

I know that's a huge diversion from the elegant simplicity of the tooling in this project. But if the Thunderbird platform makes using this tooling prohibitive, then at some point it will be necessary to consider repurposing the tooling into other environments.

If I'm being too vague there, I'll try to clarify:

• The problem is that this code is trying to use a TCP socket direcly from Thunderbird, to use the ManageSieve protocol with a remote mail server.
• A local or remote webservice endpoint can be setup, and transactactions sent there via HTTPS. That webservice itself makes a new TCP socket connection to the mail server. The result is received and proxied back to the Thunderbird extension.

It's slower, inelegant, more complex, and not suitable for many users. But it may be "a" solution or "the" solution for "some or many" people, if core changes in Thunderbird keep getting more prohibitive of TCP protocols other than HTTP(S).

I hope this wasn't a lot of misguided text that shows my ignorance on the topic. While a seasoned developer with network communications, I'm not skilled with browser extensions - and this note might be significantly premature.

[stefangweichinger]

I am on TB 115.9.0 (Fedora 40) and the sieve plugin does not work for me.

It shows version 0.6.1, I think I already sideloaded an xpi from this thread (months ago).

What's the status, is there a working release somewhere? Thanks for your work!

[Lichtmacher]

I am on TB 115.9.0 (Fedora 40) and the sieve plugin does not work for me.

It shows version 0.6.1, I think I already sideloaded an xpi from this thread (months ago).

What's the status, is there a working release somewhere? Thanks for your work!

Did you try this XPI? https://github.com/thsmi/sieve/files/13632170/XPI.-.Thunderbird.WebExtension.zip It's form 7.10.2023

On ubuntu 20.04 with thunderbird 115.10.1 (64-Bit) it is working. (Connection, activation, deactivation of scripts, create new one.)