Port to WebExtension

[SammysHP]

Sören Hentzschel has published a roadmap regarding multiprocess architecture and addon compatibility: https://www.soeren-hentzschel.at/firefox/so-geht-es-weiter-mit-dem-multiprozess-firefox-2/

(See also https://blog.mozilla.org/addons/2016/11/23/add-ons-in-2017/)

Especially two releases are important for Vimperator:

Firefox 53 (2017-04-18): Only WebExtensions allowed on AMO. This means that new developers cannot create new unlisted addons for signing.

Firefox 57 (2017-11-28): Only WebExtensions allowed in Firefox and AMO. Vimperator will not work with this release anymore.

Please note that the developer version of Firefox might drop support for XUL- and SDK-addons already some months earlier, maybe in June!

I have a really bad feeling about this. Considering the development activity during the last months I don't think that we will have ported Vimperator to a WebExtension in time. This process requires almost a complete rewrite of Vimperator and a lot of features aren't possible with WebExtensions at all. Don't get me wrong, but I fear that this is the death of Vimperator. :(

See also #211 for some more details, which focusses on e10s compatibility in general.

Chatroom

I created a chatroom on Gitter where we can organize and talk about the next steps: https://gitter.im/vim-webextension/Talk

[robehickman]

To me the current UI modification is absolutely essential to the value of this add-on. I've looked at similar things done in Chrome and found them unreliable and unusable.

[nachoparker]

Hello everyone,

I have been reading this and other threads and following the whole situation for a while. This is indeed worrysome. I do not want to create a wall of text, so I will try to be brief

• Mozilla is focusing the non-power user, and for that and other reasons, it has given up having powerful extensions that can do anything. This makes sense to try to recover user share in the market but leaves power users in a bad spot. It has conceded defeat.
• Web extensions or XUL it does not matter, but what you can do with it.
• There are many projects that need to reimplement the same thing to an extent
• Breaking compatibility is a high price to pay, but FF internal architecture needs a rework to be more performant. e10s, servo and rust look really good and I think FF will come back really strong at the price of addons. Painful but worth it.
• A browser is a huge thing and we need to work over existing open source. It is just impossible to build a full fledged browser just to have vim-like navigation. Nowadays, browsers are as complex as operating system, and creating from zero is stupid. Many young developers like starting from zero and reinventing the wheel but it will go nowhere.

Given these facts, I think that the way to proceed is clear, but there is a need of manpower and cannot be done if we only argue about we need to have a command line or we need to remove the navigation bar.

• Get together (all projects) and create the basic minimum that every party agrees on. jklh navigation, follow hints... the most basic stuff. We should do this with web extensions to the extent firefox API allows us to.
• Once this is done, each project can fork and do their own particular taste, but we can share patches for the core.
• Then the ideal would be to FORK firefox. I am not talking about Pale Moon, which is a FF 27 and is not getting the latest things. We need the latest things. We need to take advantage of all the work that is being done that is open source and build on top of it. This is not a small project, and a browser needs to be performant and secure. We need to be smart. We should create a fork of firefox that allows power users to get what they deserve and install vimperator (or whatever) as an addon to that browser.
• On top of that power-user-firefox we can further develop the addon to have control of the UI and any other things that mozilla is trying to lock down in the name of safety for non power users.
• We should rebase new firefox upstream in a regular basis to keep the browser secure, performant , and up to date

If it were under this vision I would love to help. I could start analyzing the source code of FF, or maybe I could contribute to the addon itself.

I really think we need to be smart about this or we will get locked down to a Chrome clone that targets non technical users.

[The-Compiler]

A browser is a huge thing and we need to work over existing open source. It is just impossible to build a full fledged browser just to have vim-like navigation. Nowadays, browsers are as complex as operating system, and creating from zero is stupid. Many young developers like starting from zero and reinventing the wheel but it will go nowhere.

I'll have to disagree. Building your own browser backend (rendering, JS, network stack, etc.) would be "stupid", but building a browser based on an existing engine such as CEF, QtWebEngine (both based on Chromium) or WebKit is very much possible. As a very simple example, take suckless surf which has basic vim-like browsing on top of WebKitGTK (an outdated version of it, sadly) and is some 2000 lines of code.

Of course, it is a lot of work, but I started such a project (qutebrowser) three years ago, and it's still going strong. It's exactly what you would imagine - a browser built on top of either Chromium (QtWebEngine) or WebKit (QtWebKit), with a minimal UI, and a lot of vim(perator)-like functionality. I've been using Vimperator and then dwb (another minimal browser project which died with WebKitGTK's massive API changes with WebKit2) for years, and then after not finding an alternative, started my own thing.

There's still always a lot of work to do (for example, it doesn't have something NoScript-like yet, but that'll hopefully come in the next few months), but I wouldn't say it's stupid. No intentions of stopping either :wink:. It has a solid day-to-day userbase by now - probably >1000, but it's hard to tell (it doesn't phone home). Other than other small browsers, it also works on Windows, OS X and others.

For larger things, crowdfunded full-time work is also a possibility. I did it very successfully (>7000€ raised, 2 months fulltime work on the new QtWebEngine backend based on Chromium) last year, and I'm already planning to do another one this year, to drive things forward as much as I can.

Foolish? Maybe. Stupid? Nope. Fun? Definitely (usually). :wink:

(That being said, of course I'd love Vimperator or something comparable for Firefox to live on - diversity is always a good thing.)

[robehickman]

I agree with 'The-Compiler' above, modern browsers are split into components. Most of the complexity is in the back-end engine which actually renders the HTML, handles JavaScript and what not. Different interfaces can be built on these engines fairly easily.

Nothing I've seen so far though has reached a level of being genuinely usable largely due to lacking extensions. The biggest one that is absolutely critical for me is LastPass. Coming close second uBlockOrigin.

The web today is unusable without a blocker of some description. The NoScript approach is too heavy handed for todays world IMO.

[nachoparker]

I apologize, definitely the word 'stupid' was not appropriate.

I agree that it is really fun and interesting starting something from zero. I know your project, I have tried it (congratulations! :) )

That being said, I keep seeing fragmentation and people reinventing the wheel instead of joining forces.

Do you want to start your thing from zero, or from webkit? by all means! please, I do not want to deter anybody from doing that, let alone call them stupid (my apologies again).

But the case that matters for me right now and for many people is not to learn and have fun doing a browser (not only), but to have the power of FF and Vim together.

In my point of view it is a problem with open source that there is little manpower to compete with the big guys, for instance, Chrome, and before you know it you are locked with a propietary solution or free but managed by a big company ( I like to call that freetary ). Fragmentation is our weakness.

I do not want to reprogram what already has been done, I want to use the muscle behind Mozilla to keep FF my daily browser, but just change what I don't like, because that is what open source is about.

As a final note, if I wanted to use webkit or qtwebkit or whatever as a basis I would not be writing on a FF plugin github

All the best with your project, again, it is really cool

[nachoparker]

To further explain myself, I am getting inspired in this issue by the Xposed Framework in Android.

I ignore if something like this would be possible in Firefox, but imagine we can find a way to intercept some of the basic "primitives" in the inner workings of FF in a standarized way. Typically this can be done by wrapping some of the inner functions between parts of the browser in our own functions. We could do any processing and then call the original function. It would be relatively clean to work this way and to update from upstream as we would be trying to stay at said API boundary level.

Then, plugins could use this API (like Xposed android apps) and give back the power of FF to power users.

In this manner, we could get back noscript, vimperator, tab groups and all the things that we are going to lose really soon

I hope it makes sense :P

[haasn]

Mozilla is focusing the non-power user, and for that and other reasons, it has given up having powerful extensions that can do anything. This makes sense to try to recover user share in the market but leaves power users in a bad spot. It has conceded defeat.

Except this doesn't work in practice, because chrome users are already happy using chrome - not firefox. So trying to “regain” chrome users by turning your browser into a carbon copy of chrome does nothing except alienate the rest of your user base as well.

Instead of recovering market share, the only thing Mozilla has accomplished is killing off their market share even faster.

[nachoparker]

Please, I would like to keep the politics out of this. Mozilla's decisions are something we cannot control and complaining about them here will only bring noise to this thread.

[insidewhy]

Firefox had more 500 contributors committing in the last 60 days. Chrome had almost double that. Forking it would be great if you could amass several hundred regular contributors, but you almost definitely can't. Palemoon couldn't. The fork would then inevitably end up eating dust. There's something to admire about foolhardiness but the case for pragmatism here is overwhelming.

[nachoparker]

I know it's not likely to happen... I guessed I could at least probe the crowd here.

Anyway, the idea is not to develop a new browser, but to closely follow the upstream and patch just enogh to get what we need, UI changes, noscript like capabilities, XPCOM like capabilities...

Again, until we look at the architectural changes we won't know if it is even feasible

[lord-re]

Another way to explore could be starting from servoshell ? or plain servo and a fork of browser.html ? Or something like miserve ? Going toward these will lost many firefox users using other addons but would lead to more flexibility and probably on a longer timescale be less dependant on mozilla.

[nachoparker]

Servo is Mozilla, also the best from Servo will be gradually integrated into FF. If we follow upstream we will eventually get big parts of Servo.

In my opinion we need manpower and that is why I suggest going for a popular browser with already good development and just change the minimum to meet our needs.

If we go too experimental I think the project will die as having a usable browser is a lot of work that needs hundreds of developers as it has been pointed out. Also FF has a nice userbase so we know that it is well tested, usable and safe.

[insidewhy]

The best thing to do is port vimperator/pentadactyl or at least parts of them to WebExtensions API, compromising and removing features where necessary due to limitations. Each of those removals or changes would be documented and then the community would vote on which are most important. We could then use that evidence and the strength of the community to advocate for changes in the WebExtension API, building prototypes as patches against Firefox where necessary to strengthen the case (and also for us to use).

Any solution involving forking or custom browsers will just end be being a hobby project, at best there will enough interest to maintain it for years, but it won't have the longevity of Firefox or ever be a pragmatic choice for most developers, let alone mainstream users.

[The-Compiler]

Anyway, the idea is not to develop a new browser, but to closely follow the upstream and patch just enogh to get what we need, UI changes, noscript like capabilities, XPCOM like capabilities...

As someone essentially writing a browser (see my post above), it's probably easier than forking Firefox.

I've more experience with this with Chromium, but I'd guess Firefox' situation won't be too different - there is a new Chromium major release all 1-2 months, and each release has millions of changed lines. In QtWebEngine, updating the subset (!) of Chromium they depend on from 53 to 55 meant over 5 million added and 3 million deleted lines (let that sink in for a second... 😆).

Keeping up with this (even with multiple active people, which is unfortunately rather rare) as a community project seems... difficult, to say the least. Qt says they need about a man-month for every Chromium update, and this time Chromium did some big changes in its build-system and it's been almost 2 months (with multiple full-time employees!).

[nachoparker]

@ohjames that is what I propose for the first step for sure... join forces

The forking thing would definitely be the last thing to do, only for those features that are not possible to do anymore with the webextensions API

Actually is not even worth discussing about forking anything until we get step one done, so yeah let's see if we can get a common base and definitely it would be ideal that we join forces with pentadactyl, vimfx and the rest

[SammysHP]

Please let us not talk about forking Firefox or developing an own browser. This is a completely different project (and I really doubt that we will find enough developers to create a modern browser that is feature complete compared to Firefox or Chrome).

[bovine3dom]

@cmcaine and I have sketched out how Vimperator might be ported to WebExtensions and @cmcaine has talked on IRC with andym, a Firefox developer, about the more contentious bits.

The short of it is that we think a WebExtension Vimperator can be made to work a bit better than a cVim-port in Firefox, if certain APIs that the devs have accepted on principle are implemented and if the theming API allows the hiding of basically all the standard UI elements.

The good part is that cVim is really quite nice, and we have a lot of freedom to do our own thing.

The problems are these, in our order of priority:

1. The new theming API will not permit UI elements to be hidden, nor are any other APIs planned that would do that.
2. Some features on restricted pages (about:*, addons.mozilla.org) will not work, hinting and autocompletion on the commandline are the most important features we'll lose (we'll also lose Visual, Insert and Caret). We should be able to keep tab, page and history navigation and searching (if a keyboard API is implemented by FF devs: https://bugzilla.mozilla.org/show_bug.cgi?id=1215061; UI might be available with https://bugzilla.mozilla.org/show_bug.cgi?id=1215064).
3. We won't be able to access the filesystem as easily for e.g. plugins, unless a filesystem API is accepted and implemented. There is already a storage API which can synchronise with Firefox Sync: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/storage
4. Won't be able to run external processes.

We consider 4 and 3 to be unimportant issues; 3 because it can be worked around and 4 because we never use it.

The only workaround for 1 that we have so far is if the theming API is flexible enough that we can implement auto-hiding tree style tabs or otherwise hide Firefox UI, but that seems unlikely.

We are hoping to get a productive discussion going on whether it will be possible to use content scripts on all pages here: https://bugzilla.mozilla.org/show_bug.cgi?id=1342379

The theming API is being tracked here, but the developers say that they aren't ready to discuss it yet: https://bugzilla.mozilla.org/show_bug.cgi?id=1330328

I think it is important that we engage productively with the Firefox developers on this. We should be able to reach a reasonable compromise.

TL;DR: We can already do most of the stuff we need/want to in WebExtensions, with the exception of hiding Firefox's UI, and getting trapped on special pages (e.g about:config).

[The-Compiler]

and 4 because we never use it.

What about Ctrl-i to spawn an external editor (i.e. vim for example) to edit textareas and other inputs?

[robehickman]

"What about Ctrl-i to spawn an external editor (i.e. vim for example) to edit textareas and other inputs?"

I use this all the time.

[dpaetzel]

What about Ctrl-i to spawn an external editor (i.e. vim for example) to edit textareas and other inputs?

We could try to get something similar to surfingkeys' integrated vim-like editor (https://github.com/brookhong/Surfingkeys#vim-editor)? Just a suggestion as I don't know much about firefox and/or chrome extensions…

[robehickman]

The value of vimperator to me is that it completely replaces the stock UI with one that's superior. The stock UI is now useless and consequently I don't want to see it.

[bovine3dom]

What about Ctrl-i

I've never used it - if I'm writing something big enough that it warrants Vim, I generally write the whole thing in Vim and then copy/paste it into the text box, because I'd be concerned about losing it, and I want access to e.g, Undotree. I did that with my previous comment.

As @dpaetzel says, there do seem to be plenty of JavaScript vim-like editors (e.g, http://coolwanglu.github.io/vim.js/emterpreter/vim.html) that could fill that gap.

[lord-re]

Maybe splitting vimperator into sub-addons ? One for messing with the UI (when the new api will permit it) : vimperator-ui One for the keybindings : vimperator-keys One for the external|vim-like editor : vimperator-editor

[nachoparker]

ctrl+i is very nice. I also use ctrl+t quite a bit.

In my oppinion, let's get the basics done and then we'll discuss more advanced features.

Thanks to @cmcaine and @bovine3dom for taking the iniciative! Let me know how I can help. I can code.

Should we start a new github project? fork vimperator? fork cvim?

[nagisa]

A somewhat important feature to me that hasn’t been mentioned in the list above is VIMPERATOR_INIT environment variable, which may be… difficult… to make work.

One of the most used about: pages is about:downloads for me, and it would be a shame if I had to resort to tabbing to navigate that page.

[SammysHP]

We should create a list of all people (from Vimperator, Pentadactyl...) who want to contribute to the new addon. We had maybe five contributors for Vimperator last year, not enough to make a new addon.

[basvdheuvel]

Yes, please add me! I'm not experienced in writing addons but I do have a lot of programming and JS experience. I actually have contributed 2 years ago. Don't have a load of time but am willing to invest some into this project!

[nachoparker]

alright, let me count:

nachoparker klaplong tfrangio apheleia svarogg cmcaine bovine3dom

Anyone else?

Better get moving or we'll be talking here forever. I am not aware of any repo that has been created for this (correct me if I am wrong) so created a repo

There we go

https://github.com/nachoparker/vimperator2

We can discuss over there how to start... from cvim codebase? vimperator codebase?

cmcaine and bovine3dom already have been looking into this, so they probably have an informed opinion to where to start

[mikejkjr]

If there's room, I'd love to help as well. I've used Vimperator for quite a while and would hate to see it die. I'm on vacation right now but after March 4 I'll have spare time to contribute.

I'm fairly experienced in JS development, but haven't done any plugin work before.

[Segaja]

I would try to help aswell. And maybe it makes sense to make this repository in the vimperator group?

[tecfu]

@gkatsev @maxauthority @SammysHP @timss

I suggest that a Vimperator WebExtension port should be created under the custody of the current github account used for Vimperator.

• Will be easier to find
• Leverages access to the existing user community
• Encourages leadership and participation from existing contributors
• Reduces likelihood of fragmentation

@nachoparker Therefore I'd like to suggest a different approach: ask the current maintainers to create a new vimpertator/vimperator-labs repository and then petition for push privileges (or to become an additional Vimperator maintainer).

[nachoparker]

@tecfu I am fine with that, as long as we get moving :)

updated list :

nachoparker klaplong tfrangio apheleia svarogg cmcaine bovine3dom segaja mikejkjr

[tecfu]

@nachoparker Cool. Let's see if we can get one of the current maintainers (@gkatsev @maxauthority @SammysHP @timss) to respond and facilitate.

[nachoparker]

sure, my apologies to the maintainers. I'd also rather do it under their leadership :)

[tecfu]

@nachoparker Don't sweat it dude, I don't think anyone sees your suggestion as being subversive - actually I think its the norm and the spirit of Github in general. But - in my observation projects that retain brand, contributor, and user continuity seem to outperform those that don't.

I have a recent experience of this on my own trying to get buy-in on a fork. I think it can become a big competition for attention and recognition rather than a collaborative effort to improve and maintain a product. I just didn't want to see Vimperator go the same route.

[bovine3dom]

If people are motivated right now, it would be useful to compile a list of features that they feel are important to reimplement, because apparently there are still lots of them I don't know about :)

[nachoparker]

@tecfu agree, like I said from the beginning, the more the merrier. I actually linked this conversation on the pentadactyl github. I think it would be ideal that we (pentadactyl, vimperator, vimfx and whatnot) created a solid base and then everyone can fork to their taste if they want different things.

[tecfu]

I actually linked this conversation on the pentadactyl github.

@nachoparker Promoting a unified effort to create a Vim-like browser add on based on WebExtensions is a great idea since:

"To a large extent the system is compatible with the extension API supported by Google Chrome and Opera." - https://developer.mozilla.org/en-US/Add-ons/WebExtensions.

I just suggested this idea at VimFx, Vimium, Vrome, and cVim.

[SammysHP]

@tecfu I'd like to work together with the developers of other similar addons like Pentadactyl. Thus a repository in https://github.com/vimperatur might not be the best idea. ;)

Chatroom

I created a chatroom on Gitter where we can organize and talk about the next steps: https://gitter.im/vim-webextension/Talk

[nachoparker]

@tecfu thanks for linking all these projects here

For everyone that is new to this conversation,

• the first goal is to have a minimal common ground for "vim implementation in web extensions" that we can all contribute to and use.
• after this, we can all go back to having our own fork if we have different points of view, but we can still reuse the core

[cmcaine]

@nachoparker, @bovine3dom and I already have a repo with some relevant notes in it. I've renamed it and removed some unrelated stuff. It's available here: https://github.com/cmcaine/tridactyl

Almost all notes rather than live code, because we're waiting to find out what APIs and permissions the Firefox developers are willing to support.

The most interesting documents will be design.md and bug-message.md, probably.

[philc]

Hey folks, note that we intend to release Vimium for Firefox in the near term. You can track the work here.

[tecfu]

So far I've discovered that there are two codebases that already support a Vim-like add-on using the WebExtensions API:

• https://github.com/shinglyu/QuantumVim
• https://github.com/mrmr1993/vimium/tree/firefox-merger

I'll also make mention to a repo which currently has only notes but which I will watch for development:

• https://github.com/cmcaine/tridactyl/

For now I'm going to watch these projects and revisit this issue in a month or so to reassess progress then.

[nachoparker]

I aggree, does not seem that we need to create yet another thing. Those look good

[cmcaine]

The existing chrome vim addons should work with Firefox with minimal changes.

My main interest in the tridactyl repo and in my bug reports on bugzilla is in encouraging the FF devs to allow a better UX than is currently possible with the chrome WebExtension APIs. There's definitely interest in enabling this from Firefox developers.

There is a need for developer interest in a keyboard API proposal for Firefox: the developers are OK with maintaining it but the initial implementation and definition was started by lydell (vimfx), but he's given up on it for now.

See: https://github.com/lydell/webextension-keyboard https://bugzilla.mozilla.org/show_bug.cgi?id=1215061

If you're interested in working on the keyboard API maybe get in contact with me and we can collaborate (email or issue on cmcaine/tridactyl).

@bovine3dom and I don't think there's much to be gained by outright forking an existing project, though we will certainly be reading the existing projects and adapting code. We will be experimenting with potential architectures this weekend and continuing to talk to the Firefox developers. If you're interested in working with us, please do get in contact, as above.

We'll post progress updates to this thread occasionally as @bovine3dom did today (unless anyone asks us to stop!).

Finally, it's worth noting that although Vimium is very good at what it does, it's a lot more lightweight than Vimperator: it's more like e.g. VimFx. I say this just to manage the expectations of some in this thread :) A closer WebExtension relation to Vimperator would be something like cVim or vrome.

[robehickman]

Just tried CVim on chromium. It has the same unreliability issue I've experienced with these add ons generally. There are instances where pressing a command key does nothing for no apparent reason. Clicking on the page tends to fix it but that also defeats the point of the extension.

[dpaetzel]

There are instances where pressing a command key does nothing for no apparent reason. Clicking on the page tends to fix it but that also defeats the point of the extension.

While I do not find the source for that, I remember reading something about the reason for such a behavior being that the whole keybinding stuff for these chrome extensions is (always, irc) implemented as a user script that gets injected into every page and sometimes there seem to occur problems with the asynchronous loading of these scripts...

[robehickman]

While I do not find the source for that, I remember reading something about the reason for such a behavior being that the whole keybinding stuff for these chrome extensions is (always, irc) implemented as a user script that gets injected into every page and sometimes there seem to occur problems with the asynchronous loading of these scripts...

Whatever causes it, it needs resolving if Firefox is going to have anything resembling Vimperator. I have never had a problem with it not responding.

[cmcaine]

@robehickman I didn't experience that. Are you able to identify any correlated circumstances?

As @dpaetzel says, if it happened shortly after the page was loaded it could be because the content script hasn't been loaded yet.

[Segaja]

It would be resolved if Firefox implements a normal keyboard API so that the addons don't have to rely on such hacky workarounds.