Make SmartTemplates compatible with Thunderbird 115 (SuperNova UI)

RealRaven2000 commented 2 years ago

let's use this issue for uploading test versions for Thunderbird betas.

This first one is for Thunderbird 105:

smartTemplate-fx-3.14pre5.zip

Installation: download zip file. drag zip file into Thunderbird Add-ons Manager.

RealRaven2000 commented 1 year ago

here is a version that can be tested with the current beta THunderbird 110.0b4.

smartTemplate-fx-3.16pre21.zip

To do: fix the template popup menus on the write / reply / forward buttons.

RealRaven2000 commented 1 year ago

So I am just looking again at the dropdown menus (for the attached external templates). I found that clicking the dropdown chevron doesn't display the dropdown menu with attached Smart Templates. Comparing with the built in "Get new messages" button I find the following differences:

<toolbarbutton id="button-getmsg" class="toolbarbutton-menu-button"
  is="toolbarbutton-menu-button"  type="menu" .. />

<toolbarbutton id="button-reply" class="toolbarbutton-menu-button"
  is="toolbarbutton-menu-button"  type="menu-button"  wantdropmarker="true" .. />

Sure enough, just changing type to "menu" makes the dropmarker work and generate the menu. So this will be my next task: change the attribute type to make the menu work again in 110.0b4

RealRaven2000 commented 1 year ago

To compare I am looking at the rendering of these toolbarbuttons in Thundebird 102.8 ESR - and apparently there it's type="menu-button" for all these buttons with (working) dropdowns. So the type definitely has been changed in 110 (and possibly officially for the next ESR 115)

RealRaven2000 commented 1 year ago

Here is a version that fixes the dropdowns ion 110.0b4:

smartTemplate-fx-3.16pre86.zip

It think this should also be compatible with 102, at least it worked in a quick test. I only patch the buttons that do not already have the type="menu" set to avoid double patching on the forward button,

RealRaven2000 commented 1 year ago

I am currently working on concrete versions that are compatible with the new Thunderbird SuperNova user interface for quickFilters here - there are some versions that that run (partly) with Tb beta 113. One of the biggest problems with SuperNova compatibility is the toolbar button integration. I successfully created a button with a dropdown menu using a mix of mx technology (browser action) and experimental code, so I am hopeful that I can recreate the template dropdowns for the various write / reply / forward buttons.

However I may have to create these as separate buttons which have to be added by the user manually to replace the existing ones. Obviously I want to have the update from 102 to 115 as smooth as possible, so I will look into alternative ways to do this. One might be to have everything in one SmartTemplates button which then gives the option to either Write / Reply or Forward and then select the template in one operation, but it needs to be an acceptable workaround that doesn't add unnecessary extra work / actions on the side of the user. To be discussed below.

RealRaven2000 commented 1 year ago

Created new branch ESR115 for all SuperNova related development. This work will not be backwards compatible with ESR91 / ESR102 due to the dramatic internal changes of this year's version of the Mozilla code base.

RealRaven2000 commented 1 year ago

Here is a build that can be installed in beta 113.0b1; based on all the latest changes committed to the basic functionality of ESR91, it supports applying account templates when writing emails and editing (some) SmartTemplates settings (From Add-ons Manager). The additional icons for changing template, insert smart snippets and cleaning up variables are also displayed from within Composer:

This goes to show that the actual functionality of selecting a template from a drop down button could potentially stil be supported in Thunderbird SuperNova.

The status bar icon is also still working, but by default the bar is not displayed anymore. It may be a good idea to add a toolbar button for the unified toolbar first because it could serve as a starting point for both settings and selecting templates...

smartTemplate-fx-4.0pre1.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

Next steps:

add a browser_action button (possibly create a svg version of the icon for this)
fix layout problems in settings dialog
test functionality (change template, insert snippet) from composer

RealRaven2000 commented 1 year ago

Made a browser action button with a placeholder panel as starting point:

(the links don't do anything yet, this is just a design prototype)

The main question here is whether I should invest a lot of time to refactor the existing menu code to integrate (or call up) from these write / reply / forward links to test end to end behavior first or invest a lot of time in integrating into the existing write/ reply / reply all / reply list / forward buttons etc. and might find out that it isn't supported in Thunderbird 115 anyway?

Questions:

Would having all options from the single button - and then branch on into template selection - and acceptable solution?
Should the template selector serve as a "future writing" switch instead? (Use the new button for selection of templates only and then remember this decision until another template (or account default template) is selected? This may be superior for writing many emails in batch but may be regarded as cumbersome for "normal" email use that requires different templates for every reply...

smartTemplate-fx-4.0pre4.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

RealRaven2000 commented 1 year ago

Fixed some issues in the license field of preferences (includes pasting license key) - otherwise testing can get a little tedious. I intend to rewrite the whole preferences screen in html and so that it lives in a content Tab, so not spending too much time fixing all the layout glitches in the existing window using xul. tested in 113.0 beta 3.

smartTemplate-fx-4.0pre6.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents). Next step: somehow integrate the external templates into the SmartTemplates menu panel.

RealRaven2000 commented 1 year ago

Merged in latest changes from current branch (ESR91). This includes features from #236, #237, #238, #232

smartTemplate-fx-4.0pre12.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents). Next step: somehow integrate the external templates into the SmartTemplates menu panel.

RealRaven2000 commented 1 year ago

New test version with a more functional "real" popup menu:

The next thing will be populating the sub-menus (write / reply / forward) with templates, to see if opening templates this way may work...

smartTemplate-fx-4.0pre18.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents). Next step: somehow integrate the external templates into the SmartTemplates menu panel.

RealRaven2000 commented 1 year ago

Merged templates into the write, reply and forward submenus on new unified toolbar button. I added a command handler to execute the appropriate compose commands from the Tb core buttons ( cmd_newMessage, cmd_reply, cmd_forwardInline).

UI integration for every template submenu will be governed by the new array SmartTemplate4.fileTemplates.uniMenus :

  uniMenus : [
    {
      id : "smartTemplates-write-menu",
      composeType : "new",
      command : "cmd_newMessage",
      templates : "templatesNew"
    },
    {
      id : "smartTemplates-reply-menu",
      composeType : "rsp",
      command : "cmd_reply",
      templates : "templatesRsp"
    },
    {
      id : "smartTemplates-forward-menu",
      composeType : "fwd",
      command : "cmd_forwardInline",
      templates : "templatesFwd"
    },
  ],

I will have to add more dedicated menus for "reply all" and "reply list" etc. but I need to make a call of when and where to show. Current menu looks something like this (the items last template and account templates don't do anything yet):

Trial version: smartTemplate-fx-4.0pre30.zip

However, in Tb115 the event NotifyComposeBodyReady doesn't appear to be called, here is a comparison of the flow with Thunderbird 102.10.1:

This looks like the compose-window-init event isn't fired or caught, leading to the templates not being loaded. If I set some breakpoints in debugger, it seems to work fine: this points to some SmartTemplates code maybe running too soon - or too late.

RealRaven2000 commented 1 year ago

Fixed the following regressions:

broken icons in composer toolbar and setting (deprecated -moz-image-region)
in settings dialog the inactive decks were not hidden - note that we still plan to migrate the complete settings dialog to html in order to show it in a Thunderbird tab instead of a dialog
fixed issue #240

smartTemplate-fx-4.0pre40.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

Fixing external templates is still WIP.

RealRaven2000 commented 1 year ago

Ok, after tonight's Add-on developer call we resolved a few problems during startup with external templates, as result we can now read the headers from the new email again and this restores most of the functionality when processing templates work again. Latest version, tested in Tb113.0b6:

smartTemplate-fx-4.0pre45.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

TO DO / known issues

changing identities doesn't fully work yet (the composer global bodyModified flag can be set and so swapping account templates may not work)
changing the template using the dropdown in composer now works
need to test "late resolved" fields (such as the "to" address in New emails (write case)
when clicking on a template on the new unified SmartTemplates menu, the menu unnecessarily pops up once more while composer is loading
implement the "Last template" menu item
implement an option for having "sticky" templates (reuse the last external template that was used)
implement the "reset template / account template" option to forget the last external template.

RealRaven2000 commented 1 year ago

Here is an interim version which fixes the issue of the new popup menu not being built unless a mail tab was active - this was something that probably broke in TB114.0b1, maybe earlier. It is now done via a TabListener that checks whether a mail tab is selected (for the first time) and patching in the menu (unless it already was patched successfully on Add-on Startup). I also added some fixes to the register dialog layout.

smartTemplate-fx-4.0pre56.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

RealRaven2000 commented 1 year ago

Fixed styles for expired / invalid licenses, added "check license" menu item, update action button label + tooltip using API
Included latest version of WindowListener helper API
Fixed license button flex layout in settings screen
tested in Tb 114.0b4

smartTemplate-fx-4.0pre79.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

RealRaven2000 commented 1 year ago

fixed layout of action button (made the colored area smaller to allow space between it and other buttons
made highlighted menu items consistent with status, e.g.:
added translatable strings to dropdown menus (l10n still to do on a few of these)
added tooltips for the expired / will expire / has news status of the button
button now changes state as expected when license runs out / status changes / news are read etc.
prepared "marching ants" animation for highlighted menu items. animation so far doesn't work. See https://dabblet.com/gist/8f6732f7664f777c331dd90121cac257 for some proof of concept css that appears to work in Firefox.
tested in Tb 114.0b5

smartTemplate-fx-4.0pre89.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

RealRaven2000 commented 1 year ago

Latest version:

Added reply list and reply all template menus (To do: translate st.menu.replAll and st.menu.replList)
Implemented menuitem for reusing last template - see #243
layout fixes on button (text color with news item)
implemented TabSelect event in tab listener to update menu after new version is loaded from extension manager
tested in Thunderbrid 114.0b6
To do: implement message changed listener to show/hide appropriate reply menu items.

smartTemplate-fx-4.0pre117.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

RealRaven2000 commented 1 year ago

Latest version:

translated all menu items on the action button menu
Implemented compose with default template (account)
tested in Thunderbird 114.0b6
To do: implement message changed listener to show/hide appropriate reply menu items.

smartTemplate-fx-4.0pre123.zip

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

RealRaven2000 commented 1 year ago

How should the reply / forward commands of the header bar be implemented? We only get to add one single messageDisplay action button, and we are not supposed to modify the existing reply / forward / reply to list etc. buttons. The most rudimentary way of implementing these given the current limitations would be by adding a dropdown with submenus for each action (reply / reply all / reply to list / reply to sender only / forward)

but this will add some additional required actions and strain especially when mailing out many emails where we may want to just forward or reply to many emails in a row. I would be grateful for some suggestions by my users on how to solve this in a better way within the given limitations.

speedball2001 commented 1 year ago

We only get to add one single messageDisplay action button, and we are not supposed to modify the existing reply / forward / reply to list etc. buttons.

Sorry if I hijack this bug. Did Thunderbird's developers made a statement about this? I also develop an addon that modifies the "Write", "Reply" etc. buttons, which means that I'm affected by this as well.

RealRaven2000 commented 1 year ago

We only get to add one single messageDisplay action button, and we are not supposed to modify the existing reply / forward / reply to list etc. buttons.

Sorry if I hijack this bug. Did Thunderbird's developers made a statement about this? I also develop an addon that modifies the "Write", "Reply" etc. buttons, which means that I'm affected by this as well.

Yeah I am pretty sure it is one button / Add-on. You could join the Add-on developers call next Thursday and hear it from the horse's mouth. What Add-on are you working on?

speedball2001 commented 1 year ago

Bummer. I'm the developer of Identity Chooser. Modifying the Write button is kindof the whole point of this addon.

RealRaven2000 commented 1 year ago

Bummer. I'm the developer of Identity Chooser. Modifying the Write button is kindof the whole point of this addon.

well you could hijack the click event and show your identities dropdown regardless, if you still have legacy (experimental) rights.

RealRaven2000 commented 1 year ago

Added the popup menu in the header bar above the message (3pane)

Also added popup menu in header bar of single message tab:

smartTemplate-fx-4.0pre172.zip

(This is still rather WIP, I built in some delays for better debugging)

to install in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

Next Steps:

support single message window
optimise & speed up the code
when the button is clicked (or a new email selected, in the case of single message tab) hide any submenus that do not apply (reply list, reply all). Do we need a “reply to sender”?
investigate if/how the menu can be made more efficient for bulk mailing events (keyboard shortcuts? Positioning of menus / buttons)
Ideally we should have one button / action (reply, forward, reply list etc.) but i don’t think this is allowed. I am still considering whether to hack the main buttons anyway and integrate the menus into the actual reply button. But the old pattern of the combined push/dropdown button has sadly been deprecated by the Thunderbird UI overlords.
One idea: we could append the 5 most recent actions to the bottom of the menu, e.g. reply with template X, forward with template Y, reply all with template Z. This could be a workflow for optimising a multi-language email campaign…

RealRaven2000 commented 1 year ago

Bummer. I'm the developer of Identity Chooser. Modifying the Write button is kindof the whole point of this addon.

@speedball2001 I had quite a few discussions with @jobisoft on this topic, the main point of contention being that the combined pushbutton/dropdown pattern goes away, probably because it looks too cluttered or is nonstandard (?). One of the main guys behind UX decisions like this is Alessandro, we could try and lobby together with other Add-on devs who use the pattern to bring it back.

I am not sure whether they would listen to just a single Add-on developer. The ideal long term solution would be a messageReaction API, that allows asking for Addon specific options before opening compose (either via dropdown or some other panel) but that’s probably even hard to specify, let alone implement.

If you can, do join the weekly Add-on developer Zoom meeting - its every Thursday at 8:00pm BST and can be accessed from the Thunderbird Community page. There is also a link to the Matrix Chat where we can chat in real time, I am there most of the time asking questions & sharing dev experience.

RealRaven2000 commented 1 year ago

Another version, there are still some issues especially with the main button not being initialized properly when the Add-ons tab is shown while it starts up. I have added support for single message windows.

smartTemplate-fx-4.0pre190.zip

To Do:

make sure the menu it properly injected even if a non-mail tab is active while the Add-on is started / updated. For testing, the easiest way to ensure proper loading is to switch to the first (mail) before clicking Add on the extension permissions popup:

RealRaven2000 commented 1 year ago

Ok, I managed to fix the previous problem with the non-mail tab having focus while SmartTemplates is started. The solution was to go through the background page and send a message to the messenger script to get the correct window context (messenger, not about:message) for the patch script.

smartTemplate-fx-4.0pre199.zip

So this should now mostly display the template submenus as expected. There may some additional work for updating all the menus when they are edited from preferences and also updating the "Last template" entries for each of the menus for the cases forward / reply and write.

Next steps

As we see there are still a few minor issues with the popup menus, which sometimes lose the extra space necessary for the selector chevron >.
I would like to add accelerator letters to the menus Reply, Reply All, Reply to List and Forward - they should be pulled from the messenger localization file to be consistent with the threadPane context menu.
I would also like to add accelerator shortcuts for Default Template (0) and Last Template. The Last template accelerator should possibly be configurable (personally I would choose a key close to the bottom row, such as \ , . / but these are obviously locale dependent.)

To install in this version Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

RealRaven2000 commented 1 year ago

Ok, I managed to fix the previous problem with the non-mail tab having focus while SmartTemplates is started. The solution was to go through the background page and send a message to the messenger script to get the correct window context (messenger, not about:message) for the patch script.

I was wrong - for some unfathomable reason I am still ending up in a closure where WindowListener.injectElements (even though it is called from an event handler in the st-messenger.js script) still has window.document being "about:message". I am not sure how I can influence the window/document context of this script other than calling it from a messenger context. Maybe I need to ask @jobisoft how to pull this off.

At this stage I can only imagine that my WL gets overwritten by one of my tab scripts in st-messagePane.js ???

RealRaven2000 commented 1 year ago

I made a second instance of the WindowListener for patching the header area above the message (the about:message window), this seems to fix the issues:

smartTemplate-fx-4.0pre205.zip

@jobisoft will give me some help converting these menus into API-compliant ones. Then I will also not be tempted to mess with the original reply / forward buttons anymore.

RealRaven2000 commented 1 year ago

More stability changes for header Pane and l10n for the included new feature #243

smartTemplate-fx-4.0pre229.zip

To install this version in Thunderbird beta, download the zip file and drag into Add-ons Manager (do not extract contents).

RealRaven2000 commented 1 year ago

Release 4.0b1 yesterday. which still has some unresolved issues with the menus, such as:

If the Add-on is updated or installed from Add-ons Manager the unified toolbar menu may not be initialised with template items
same happens if a folder is opened in a background tab (via right-click open in the new)
if an email is opened via the context menu in a background tab (right-click an email "open message in new tab")

in all cases the template menus may appear unpopulated with templates. the beta version is based on prerelease 4.0pre237

The above issues will be fixed in the next beta. Here is a preview with some quick fixes for menus:

smartTemplate-fx-4.0pre256.zip

Note that the menu handling will be rewritten using API compliant methods instead.

RealRaven2000 commented 11 months ago

Added #259 for conversion of settings to HTML.

MarkyMarkDE commented 11 months ago

@RealRaven2000: you can check vue? All modern Addons work with this "Framework" and maybe we have so less of work in converting our contents into html ... e.g. see here: https://github.com/fastaddons/WebTranslate Maybe we can win him for our project, he seems much skilled in vue

benibela commented 11 months ago

messageRaw appears to be broken. It always returns something like `

??`

RealRaven2000 commented 11 months ago

messageRaw appears to be broken. It always returns something like <smarttemplate hdr='messageRaw' st4variable='messageRaw(1000)' title='messageRaw(1000)'>??</smarttemplate>

can you provide the text of the template so I can test? Also since this is part of the scripting feature (which is not officially supported for standard use) let's continue this discussion on #266 - I am sure it is related to being async now. messageRaw tries to stream the raw message body, it may not be the right way to go about it in the new environment.

benibela commented 11 months ago

can you provide the text of the template so I can test? Also since this is part of the scripting feature (which is not officially supported for standard use) let's continue this discussion on #266 - I am sure it is related to being async now. messageRaw tries to stream the raw message body, it may not be the right way to go about it in the new environment.

it can be used without scripting like %messageRaw(1000)%

RealRaven2000 commented 11 months ago

I will hopefully get around to looking at this over the next couple of days, once #263 is complete.

RealRaven2000 commented 11 months ago

can you provide the text of the template so I can test? Also since this is part of the scripting feature (which is not officially supported for standard use) let's continue this discussion on #266 - I am sure it is related to being async now. messageRaw tries to stream the raw message body, it may not be the right way to go about it in the new environment.

it can be used without scripting like %messageRaw(1000)%

It appears this used a deprecated class SmartTemplate4.classGetHeaders() which doesn't work in 115 anymore and where the "content" function needs to be reimplemented. Code:

SmartTemplate4.getHeadersWrapper = function(originalMsgURI) {
  let rv;
  // streaming the message works up to Tb91, in Tb102 we need to load the headers during when starting ComposeStartup() [async]
  if ((SmartTemplate4.Util.versionSmaller(window.SmartTemplate4.Util.Appver, "102"))) {
    rv = new SmartTemplate4.classGetHeaders(originalMsgURI); // legacy headers
  }  
  else {
    rv = SmartTemplate4.MessageHdr; // new headers in TB102 - uses SmartTemplate4.getHeadersAsync()
  }
  if (SmartTemplate4.Preferences.isDebugOption("headers") ) {
    console.log(rv);
  }
  return rv;
}

RealRaven2000 commented 11 months ago

@benibela here is a version that makes %messageRaw()% work again:

smartTemplate-fx-4.3pre83.zip

it iterates all mime parts and looks for "body" elements, then concatenates them.

Note that if you parse a plain text message, currently double newlines between paragraphs are collapsed to single spaces (I tested while replying in HTML mode). Not sure if they should be converted top <br> automatically?

benibela commented 11 months ago

@benibela here is a version that makes %messageRaw()% work again:

It works

Note that if you parse a plain text message, currently double newlines between paragraphs are collapsed to single spaces (I tested while replying in HTML mode). Not sure if they should be converted top <br> automatically?

For scripting, it should return everything unchanged

I think it was called messageRaw, because it would not convert anything. Otherwise, it could have been called message (actually I only figured out this year, how to get the actual message out of messageRaw in the old version)

RealRaven2000 / SmartTemplates

Make SmartTemplates compatible with Thunderbird 115 (SuperNova UI) #213