Try out new withExEditor which embedded WebExtensions

[asamuzaK]

@ssokolow @wavexx @Celti @ariel-co @maxigaz

Hi there.

I am migrating withExEditor to WebExtensions (it's still embedded WebExtensions yet), and planning to release a new version of withExEditor along with the release of Firefox 52.

So, I would like to ask you for your cooperation. If Firefox Nightly, Developer Edition, or Beta is installed, will you try out the development version of withExEditor?

Note 1: The new withExEditor is targeted for Firefox 52 or higher. You can not install it on the current release version of Firefox (51) or ESR.

Note 2: The new withExEditor is incompatible with the old version. Also, when you install the new version, settings for the old version will be removed. Therefore, prepare a new Firefox profile for testing.

Thank you for your cooperation. Regards.

[Celti]

Configured and testing with this very comment. You might add to the instructions that (on Linux using Firefox Developer Edition, at least) you need to restart Firefox after adding the manifest to the appropriate path. Got an absolutely inscrutable error in the browser console before doing that.

Seems to work just fine for me after that, though. Going to see about writing a host adapter for it in Rust, maybe make the editor configurable.

[asamuzaK]

Thanks for testing and filed a PR. #24 Does that PR make it somewhat easier to understand?

Going to see about writing a host adapter for it in Rust, maybe make the editor configurable.

Can you create a sample in Rust?

[asamuzaK]

On Bugzilla, I was advised that to read/write files should be done on host, not add-on side. 1246236 – Implement local filesystem read/write access. I have to rewrite the source again, sigh...

Therefore, I'll suspend the test. Please wait for a while.

Thanks

[asamuzaK]

@ssokolow @wavexx @Celti @ariel-co @maxigaz

Released v2.0.0-rc2, please try it and tell me if it works. withExEditor :: Versions :: Add-ons for Firefox

You need Node.js and asamuzaK/withExEditorHost: Native application host for withExEditor

[asamuzaK]

Confirmed working: withExEditor: 2.0.0-rc6 withExEditorHost: 1.0.0-b.1

• with Node.js: 7.5.0 Win10 (64bit) + Firefox Nightly Win10 (64bit) + Firefox Dev Edition

• with Node.js: 6.9.5 LTS Win7 (32bit) + Firefox Beta Ubuntu16.04 LTS (VM) + Firefox Nightly

[Celti]

I can't for the life of me get it working on Arch Linux with Node 7.5.0 and Firefox Dev Edition.

Error: Attempt to postMessage on disconnected port
Stack trace:
portHostMsg@moz-extension://f4cc9116-fa8f-46a9-95dc-d10e48971394/js/background.js:152:20
handleMsg@moz-extension://f4cc9116-fa8f-46a9-95dc-d10e48971394/js/background.js:567:25
  background.js:83:5
    logError moz-extension://f4cc9116-fa8f-46a9-95dc-d10e48971394/js/background.js:83:5
    (Async: promise callback)
    handleMsg

[asamuzaK]

Any other stderr or stdout message from the host in browser console?

[Celti]

Apparently I needed to restart Firefox three times after editing all the files, not just once or twice. Dunno what exactly changed but it's working after one more restart. Sorry for the noise.

[asamuzaK]

Never mind, thanks for your contribution!

Maybe did you clone the host from GitHub? If so, I was working on host for past several hours, that may be the cause.

[asamuzaK]

Maybe the real cause was that I was throwing error on promise reject. I thought it will be caught in process.on("unhandleRejection"), but maybe it got up and caught in process.on("uncaughtException") too.

Refactored rejection handling. Releases · asamuzaK/withExEditorHost

[asamuzaK]

Released rc7 withExEditor :: Versions :: Add-ons for Firefox

Please check that even after updating the add-on, you can use it without having to restart Firefox. However, you should need to reload on the existing tab that is opened before update.

[Celti]

New version is working flawlessly, but I have a feature request — can the storage location for the files be made configurable? The old version stores them in the profile folder, which is persistent in case of a system crash for easy recovery. The new version stores them in /tmp on Linux, which on most desktops is not persistent.

[ssokolow]

@Celti What about /var/tmp?

I remember being told that transient vs. persistent is the distinction between /tmp and /var/tmp and that'd still make it easy to flush stale files if there's a bug in withExEditorHost that leaves them lying around.

...or, as a user-local alternative, perhaps a withExEditor folder inside ${XDG_CACHE_HOME:-$HOME/.cache}. (The path contained in the XDG_CACHE_HOME environment variable if set, falling back to ~/.cache if unset, as specified in the XDG Base Directory Specification.)

[ssokolow]

This Node.js package looks like it would do the trick:

https://www.npmjs.com/package/env-paths

const envPaths = require('env-paths');
const paths = envPaths('withExEditor');

const drafts_dir = paths.cache || paths.temp;

I haven't tested this, but the API docs say it should work and both paths.cache and paths.temp will include withExEditor in the path so there's no worry about filenames colliding with those from other applications.

I don't know if paths.cache can return null, but better safe than sorry. Hence, the || paths.temp.

It doesn't appear to ensure the target path exists, so you'll also want to consult StackOverflow: Node.js create folder or use existing for the solution most matched to the existing coding style within withExEditorHost.

[asamuzaK]

Filed Make tmp file storage location configurable · Issue #2 · asamuzaK/withExEditorHost

[ghost]

I installed version 2.1.0a4 along with version 2.0.0-a.15 of the host, and have not been able to get it to work. The problem seems to be that when I open the preferences of withExEditor in Firefox, the editor config path is not saved. I've definitely put the right path there, but the Editor Label field always remains disabled, so maybe somehow the editor config json file is not being found?

[asamuzaK]

Do you have any logs in Firefox browser console (Ctrl + Shift + J) related to withExEditor or withExEditorHost?

Also, go to your Firefox profile folder and look for storage.js On Windows it's there:

C:\Users\[UserName]\AppData\Roaming\Mozilla\Firefox\Profiles\[YourProfile]\browser-extension-data\jid1-WiAigu4HIo0Tag@jetpack\storage.js

Is there editorConfigPath key in it? Something like:

"editorConfigPath":{"id":"editorConfigPath","app":{"executable":true},"checked":false,"value":"C:\\Path\\To\\editorconfig.json"},

[ghost]

Here are all the console messages that I see related to withExEditor.

I checked the storage.js files (for some reason the profile I am using has two of them) and neither contains the editorConfigPath key that you mention.

If it's relevant, I'm using linux, not Windows.

[asamuzaK]

Those errors seems to be related to GitHub page, not withExEditor itself. Will you check below?

0. Close Firefox, remove both storage.js files. (It will be generated again on restart). I don't know why, but there should be only one storage.js file, not two.
1. Is there No matching message handler for the given recipient. logged in browser console? 1.1. If so, save withexeditorhost.json in ~/.mozilla/native-messaging-hosts/withexeditorhost.json. It should be saved there, not global, I think.
2. When you start Firefox, is Node.js process executed too? 2.1. If not, make sure there is $PATH environment variable for Node.js. Or, change shell script to path/to/node.js /path/to/withexeditorhost/index.js 2.2. Also, make sure withexeditorhost.sh has executable bit.

[ghost]

Still no luck but I've fixed some things that needed to be fixed, hopefully this can be sorted out.

0. I removed the two storage.js files, but they've both been regenerated. For some reason, there are two directories sitting under /home/<username>/.mozilla/firefox/<profile-name>/browser-extension-data. The two files differ.

1. I did not see the No matching message handler for the given recipient message in the browser console.

2. There is no node.js process executed that I can tell from looking at the processes listed in the system monitor.

2.1 On Ubuntu it seems that the name of the executable is nodejs, not node as it was written in the shell script. I changed it to nodejs, but that still didn't fix the problem. The executable is in /usr/bin so it is definitely already in my PATH.

2.2 The executable bit was not set on withexeditorhost.sh, so I set it with chmod a+x withexeditorhost.sh, but that still didn't get the addon to work, unfortunately.

[asamuzaK]

I removed the two storage.js files, but they've both been regenerated. For some reason, there are two directories sitting under /home//.mozilla/firefox//browser-extension-data. The two files differ.

withExEditor files are stored under ~/.mozilla/firefox/<prof>/browser-extension-data\jid1-WiAigu4HIo0Tag@jetpack/ jid1-WiAigu4HIo0Tag@jetpack is the ID of withExEditor. Remove storage.js under this directory.

There is no node.js process executed that I can tell from looking at the processes listed in the system monitor.

Check withexeditorhost.json file. Is path field correct? It should be

"path": "path/to/withexeditorhost/config/withexeditorhost.sh",

Note that comma at the end of the line.

Attached index.js for debugging. Replace this file with your path/to/withexeditorhost/index.js Restart Firefox. Tell me the log message in browser console. Please test with a clean Firefox profile.

index.zip

[asamuzaK]

2.1 On Ubuntu it seems that the name of the executable is nodejs, not node as it was written in the shell script.

On my VM Ubuntu, it is node (usr/local/bin/node), though.

[ssokolow]

Accoding to the answers on StackOverflow, it's a naming-conflict thing similar to ack vs. ack-grep.

There is a naming conflict with the node package (Amateur Packet Radio Node Program), and the nodejs binary has been renamed from node to nodejs. You'll need to symlink /usr/bin/node to /usr/bin/nodejs or you could uninstall the Amateur Packet Radio Node Program to avoid that conflict.

It'll apparently create the node symlink for you under certain circumstances and you can force it with this command:

sudo update-alternatives --install /usr/bin/node node /usr/bin/nodejs 10

I've confirmed that node is indeed a symlink provided by the "alternatives" system on my Kubuntu 14.04 LTS.

[asamuzaK]

How about adding a comment like this?

#!/usr/bin/env bash
# Replace `node` command to `nodejs` according to your environment.
# Fill in the path of the index.js file of the withExEditorHost.
node /path/to/withexeditorhost/index.js

[ghost]

Still not working but I made some progress. I started with a fresh profile and decided to go over the setup steps one by one carefully, and realized I missed something: I neglected to save the withexeditorhost.json file in the right spot. I've fixed that now. Sorry to waste your time before, that's totally my bad.

However, it's still not working. When I put the path for the editorconfig.json file into the box in the options page for the addon, the Editor Label field is still always disabled, and the path is not saved if I close the options page and open it again. Is it supposed to be the full path to the file or the path to the directory containing the file? (Either way it doesn't work for me.)

I still don't see a node.js process when I open the system monitor, although I could be overlooking something. Would it perhaps be called something else?

I did see more in the firefox console log this time:

That seems to happen every time I start up. Also, if I try to edit a text box with the extension, this is what pops up on the console:

[asamuzaK]

Are you sure your using the proper node.js version which withExEditorHost requires?

[asamuzaK]

And some more:

Suppose you saved withExEditorHost in /home/[yourprofile]/Documents/withexeditorhost

withExEditoHost will try to look for ./config/editorconfig.json relative from index.js by default. Which is /home/[yourprofile]/Documents/withexeditorhost/config/editorconfig.json in absolute path.

So save your config folder (which contains withexeditorhost.sh and editorconfig.json) under /home/[yourprofile]/Documents/withexeditorhost/. Then, you don't need to fill in editor path field in withExEditor options page.

[ghost]

OK, I will try changing the location as you say and get back to you. I had it in /home/[username]/.config/withexeditorhost previously.

I have installed version 4.2.6 of node.js, which version is required?

[asamuzaK]

OK, I will try changing the location as you say and get back to you. I had it in /home/[username]/.config/withexeditorhost previously.

Under .config/ should be OK, it can be saved in arbitrary location. But you should remove version string from the directory name. Not withExEditorHost-2.0.0-a.15, but something like withExEditorHost or withexeditorhost.

I have installed version 4.2.6 of node.js, which version is required?

Requirements are noted in release. For v2.x (Current v2.0.0-a.15): Node.js >= 7.6.0

For v1.x (Current 1.0.10): Node.js >= 6.9.5

[ghost]

Installing node.js from nodejs.org did it! Sorry, I must have missed the version requirement in the notes. Obviously that's my own fault, but it might help others to note the version needed with the setup instructions rather than just on the releases page. (Also, crazy that the version of node.js in the Ubuntu repositories is so outdated.)

This was typed in emacsclient! I'm so happy this works, and thanks again for all your help, I really appreciate that and your hard work to make the extension in the first place. Seems like it's working perfectly now. I'll let you know if I encounter any more problems.

[asamuzaK]

Glad to hear that.

it might help others to note the version needed with the setup instructions rather than just on the releases page.

That makes sense. Thanks.

[asamuzaK]

While waiting for a review for v2.0.0, I was told that some error happend in AMO and requested to update a newer version for another review. So, I uploaded 2.1.0, and it's reviewed now. withexeditor

Thanks to all of you for your contributions.

[asamuzaK]

Note that you may need to update manually switching from dev version to release version.

[wavexx]

On Wed, Jan 25 2017, asamuzaK wrote:

I am migrating withExEditor to WebExtensions (it's still embedded WebExtensions yet), and planning to release a new version of withExEditor along with the release of Firefox 52.

Late to the party, but I just realized a "host" is now needed. Why exactly is this required?

I have to say it honestly here, I'm not running nodejs just to run an external editor.

[ssokolow]

@wavexx Because Mozilla is killing off the old Firefox extension APIs and the Chrome APIs they're copying have no way to write to arbitrary files or call arbitrary programs.

withExEditor has two options:

1. Trigger the editor by performing an HTTP request (requires a host or editor plugin to listen on an HTTP port and there's always the chance something like NoScript's ABE would block local requests generated while viewing a remote page)

2. The nativeMessaging API, which requires a specially crafted host.

withExEditor chose the latter but, either way, the reality is that, come version 57 later this year, Firefox will refuse to load any extension which uses the APIs needed to do this without a host.

EDIT: That said, if I wind up writing an extension which needs a host, I'm just glad that I have some experience coding in Rust, because I do value lightweight applications and will do anything within my means to avoid contributing to the "Browsers and Thunderbird are the heaviest resident apps on my system by an order of magnitude" problem.

[asamuzaK]

As @ssokolow described, that is the way that WebExtensions requires. The browser interacts with the native application via messages, and the external editor can not be executed directly within an add-on anymore.

[wavexx]

On Thu, Mar 23 2017, asamuzaK wrote:

As @ssokolow described, that is the way that WebExtensions requires. The browser interacts with the native application via messages, and the external editor can not be executed directly within an add-on anymore.

Sigh, as a developer, I knew this would come, but I didn't expect it to hit me so soon :/

FF for me means nothing without extensions. I'll hold back to ESR. Mozilla is making a huge mistake here.

[ssokolow]

No argument there. Lately, when I've been able to make time, I've been working on a "Disaster Plan" which could result in me rewriting significant parts of my extension load-out as external applications (eg. an HTTP proxy for privacy/security features, XGrabKey+XTEST proxying for keyboard overrides disallowed for "security reasons", etc.) and/or switching to Chromium, SeaMonkey, or Pale Moon, depending on what my research uncovers.

(Chromium may be superior to Firefox+WebExtensions for my use, given that it's closer than Australis to what I currently use Classic Theme Restorer to attain. In the long term, the ideal solution would be for the WebExtensions API to become ubiquitous enough for QWebEngine to add support for it so I could reinvent and viably maintain something close to classic Firefox using PyQt5.)

I already know I want to move all of my data (bookmarks, saved sessions, etc.) into a native application connnected by a shim extension so it's as easy as possible to migrate between browsers. I'm also hoping that they'll properly support GNOME's convention for a single native menu (not a wrapped toolbar in a panel), because I can probably use an LD_PRELOAD hook to re-create the bug which causes it to appear outside GNOME 3 in GTK+ 3.x apps under *buntu 14.04. (Until then, I'll have to build a habit of tapping Alt to pull out the menubar they have to maintain for MacOS.)

[wavexx]

On Thu, Mar 23 2017, Stephan Sokolow wrote:

working on a ["Disaster Plan"] (https://gist.github.com/ssokolow/2fb8debc7dfdbf1d9a0da90284f5d73a)

That's quite detailed.

(Chromium may be superior to Firefox+WebExtensions for my use, given that it's closer than Australis to what I currently use Classic Theme

Chromium is nowhere near FF when it comes to customization, sadly. I don't want to run ServiceWorkers with the current execution model. The Caches API is horrendous from the user's perspective. I don't want to enable WASM. I don't want EME. The list goes on and on.

Somehow, many of these tweaks are still possible with plain FF. The rest can often be covered with extensions.

The (short term) future looks bleak.

[ssokolow]

I'd be patching Chromium anyway to comment out the bits which hide http:// in the address bar.

Given that, I'm saying it'd probably be easier to NOP out things like Service Workers and EME than to maintain a patchset to restore a classic UI in Firefox.

[asamuzaK]

Help wanted Update host setup instructions