Remove the need for users to set up Node.js themselves

[ssokolow]

As a means of doing something useful with my spiteful urges against the limitations of the WebExtensions API, I've been researching options for making native messaging hosts as easy for end-users to install as possible.

I was wondering whether you're be receptive to the idea.

For withExEditorHost, it would require two steps at minimum:

1. Removing the need to install Node.js separately

This can be accomplished using pkg or nexe. (Or possibly node-compiler or node-packer but I hadn't heard of them before.)

Both tools allow you to bundle your project, the Node.js runtime, and all of your dependencies into a single executable. (And to generate Windows, Linux, and MacOS versions from the same machine as long as none of your dependencies are compiled C modules. They can download the ready-made runtimes from the Node site.)

2. Giving Windows users something friendlier than setup.js

The simplest way to accomplish this would be to build an installer. (Which will also allow you to use LZMA (7-zip) compression without requiring users to have 7-zip installed.)

There are three tools I know of which are free and open source, good enough that major companies are using them, and powerful enough to be scripted into autodetecting which browsers are installed and enabling/disabling entries in the components list based on that:

1. InnoSetup (Basic functionality is defined via a .ini-like format. Advanced functionality is supported by a Pascal-based scripting language added in version 4.x. You could either have a "component" for each browser and disable ones that weren't found or use CreateInputOptionPage to create a wizard page full of checkboxes... I recommend the former.)
2. NSIS (Uses its own internal scripting language, but has the longest history of being scriptable and NSIS installers can also be created on Debian/Ubuntu/Mint Linux without using Wine via the nsis package.)
3. WiX Microsoft's toolkit for building .msi files. (While this is technically the best solution for the end-user, I find it harder to learn than the other two choices and haven't learned enough about it to know how to actually use it in this situation yet... but it looks like the dialog set you'd want to use is WixUI_FeatureTree.)

As for checking for updates, the best solution would probably be to have the browser extension responsible for doing that, with the host just telling the extension its version number when asked.

[ssokolow]

Oh, to clarify, the reason I only mention Windows users is that "just put the output of pkg/nexe/etc. in a Zip file" is the recommended option for developers who don't have the Mac and Apple developer license needed to use their proprietary tools.

(The closest I've found is node-appdmg for generating .dmg files, and their effort to support licensed Apple developers who want to cross-compile is more or less stalled on finding a way to sign things without using Apple's signing utility.)

[asamuzaK]

Is it possible to provide a package without Node.js to those who already has Node.js?

[ssokolow]

In what sense? Offering two separate downloads to save download bandwidth or offering a single download which only installs the bundled Node.js if it's not already present on the host system?

The latter is of questionable value (having people download a copy of Node.js even if they're not going to keep it) but the former is certainly possible.

That said, one advantage of having each thing bundle its own Node.js is that it simplifies technical support by standardizing the version and configuration of the Node.js runtime used. (ie. No bugs caused by running on top of an incompatible Node.js version.)

[asamuzaK]

I meant the former.

• prepare package with node:
  • node executable
  • node_modules
  • host scripts
• prepare package without node:
  • node_modules
  • host scripts

If possible, only including dependencies modules (omit devDependencies) should be better.

[ssokolow]

I'm not a Node.js expert, but wouldn't the second option just mean packaging the project folder after asking npm to install dependencies non-globally?

(I use "packaging" as a general term to refer to building an installer or a more ordinary archive as appropriate.)

[asamuzaK]

Uploaded test binaries. withExEditorHost/dest at pkg · asamuzaK/withExEditorHost

To execute setup script, run index --setup

[ssokolow]

Monday was a bit of a mess, and I'll be out on Wednesday, but I'll see if I can fit in some testing on Tuesday.

Testing in VMs which don't have Node.js already installed will have to wait for at least Thursday though.

[asamuzaK]

Oh, thanks. I've confirmed with

• Win7 32bit + Firefox, Chrome
• Win10 64 bit + Firefox

[ssokolow]

Good to hear. I'll start working on an installer as soon after testing in VMs as possible if you can send me a reference listing...

1. Each supported browser's name (for the list in the installer wizard)
2. A list of things that --setup does to set up support for that browser (so it's possible to rely on the installer's internal uninstall logging mechanism and do the setup without opening a command prompt window.)

Another possibility I could research, if you're interested, is setting up Travis-CI and/or AppVeyor so that, when you push a new tag like v2.3.1 to GitHub, they automatically generate installers, and then create a new GitHub release entry, with attached downloads.

(I'll probably use NSIS for the installer because, that way, it'll be your choice where the installers get built, rather than limiting them to building in a Windows environment like AppVeyor.)

[ssokolow]

OK, here's my impression so far on a Kubuntu Linux 14.04 LTS system with Node.js already installed:

1. The index file in linux.zip is not marked executable by default. Probably a good idea to build it on a Linux system like Travis-CI where it can be marked executable and then stored using an archival utility that will store the permissions flag in the archive. (tar or the Linux version of zip)

2. Once I remedied that issue, it worked perfectly... with one caveat that I'll file another bug for. (#27)

[ssokolow]

Oh, given that every platform has a utility to open a file in whatever's application is associated with it (start on Windows, open on OSX, and xdg-open on Linux or BSD), it'd probably also be a good idea to offer that as default by convention.

(That's actually how Chromium on Linux does things. When you click a file in the downloads bar, it feeds it to xdg-open.)

I'll plan to include a screen in the installer which presents a choice between two radio buttons:

• Use whichever application is set to open text files at the time
• Always use [text field pre-filled with the path to the default text editor as an example]

(With a button that calls up the application-picker dialog Windows uses for "Open With..." if I can figure out how. Failing that, I'll provide a browse button.)

[asamuzaK]

The index file in linux.zip is not marked executable by default

Hmm, I created zip file by jszip because it had client api jszip-cli. jszip seems to be able to add permissions, see file(name, data [,options]), but it's unclear how to add that option in command line.

Maybe

--unixPermissions 701

or

--unix-permissions 701

will do?

[asamuzaK]

Probably a good idea to build it on a Linux system like Travis-CI where it can be marked executable and then stored using an archival utility that will store the permissions flag in the archive. (tar or the Linux version of zip)

Sounds good, but in that approach, I can't test built file locally?

[asamuzaK]

Good to hear. I'll start working on an installer as soon after testing in VMs as possible if you can send me a reference listing...

Each supported browser's name (for the list in the installer wizard) A list of things that --setup does to set up support for that browser (so it's possible to rely on the installer's internal uninstall logging mechanism and do the setup without opening a command prompt window.)

Setup is done by another module asamuzaK/webExtNativeMsg: Helper modules for WebExtensions native messaging host for the host part, and by withExEditorHost/setup.js at pkg · asamuzaK/withExEditorHost for the editor config part.

You can get the supported browser data from webExtNativeMsg/browser-data.js at master · asamuzaK/webExtNativeMsg

webExtNativeMsg will create:

• config directory
• shell script within configuration directory
• manifest json of the host
• registry (windows)

withExEditorhost/modules/setup.js will create:

• editor config json file within config directory

BTW, Is it possible to create installer by communicating with those scripts?

[ssokolow]

BTW, Is it possible to create installer by communicating with those scripts?

There are four problems which make that more trouble than it's worth:

1. If the scripts copy/move/create files, I know of no way to have NSIS add them into the list of files that get uninstalled by the Add/Remove programs option. You don't want the bad PR of having an installer/uninstaller pair which intentionally doesn't uninstall properly. (That's what bundled malware does.)
2. The whole point of making an installer is to have a friendly, familiar wizard GUI to walk users through the install process. That means that, to use the scripts, the installer would have to reinvent their prompts as wizard pages and then feed the info to the scripts in a potentially fragile way.
3. Windows isn't very good about having programs which sometimes produce terminal output and sometimes don't. Whether or not to pop up a terminal window is a compile-time switch.
4. Users are used to associating terminal windows during install processes with a lack of professionalism because they're most often used by people who slap together an installer where the heavy lifting is done by using .bat files to glue together existing command-line utilities. (eg. cracked software... and efforts have been made to associate cracked software with malware in the minds of end-users.)

[asamuzaK]

Got it, thanks.

[asamuzaK]

Test build Releases · asamuzaK/withExEditorHost

• Added scripts to add unix permission in .travis.yml
• Can download from Releases

[ssokolow]

I just checked the new linux.zip. The permissions triple you chose for index is a bit odd, but unzip does create it in an executable state.

(I've never seen anyone set the permissions to 701 (rwx --- --x) because it's rare to find a use for "allow read/write/execute (7) for the owning user, deny everything (0) for the owning group, and then allow execution without reading (1) for everyone else. Typically, for something like this, you'd permission it 755 (Only the user can write to the file, but everyone can read or execute).

...or, for something containing private information, such as your SSH key files, 700 (user can do anything, group and "other" can do nothing.)

If you're not familiar with numeric-form permissions on POSIX, here's a chart:

 u   g   o
rwx rwx rwx
421 421 421

• The permissions are represented in the form of an octal (base-8) number.
• The first digit is the user permissions, the second is the group, and the last is everyone else
• Because it's base-8, each digit is made up of three bits.
• For each digit, the 4 bit is read permissions, the 2 bit is write permissions, and the 1 bit is permission to execute a file or list the contents of a folder.

Hence, read + write + execute = 4 + 2 + 1 = 7

...just bear in mind that the octal representation is what humans and the chmod command expect. Raw system-level APIs expect an ordinary integer variable, so, if you're interacting with the APIs directly, you'll have to use one of the following options:

• An octal literal like 0755
• An integer-parsing function like parseInt('755', 8);
• Combining constants using bitwise OR. (Gets very verbose when you're building permissions from scratch rather than modifying specific permission bits on existing files/folders.)

[asamuzaK]

Thanks for pointing out. Fixed permission to 755 and uploaded a new try builds.

[ssokolow]

...and I just realized that I forgot to respond to this:

Sounds good, but in that approach, I can't test built file locally?

It shouldn't be difficult to set up an NPM script which runs equally easily on your machine or in Travis.

(You could then do local testing of any inherently Linux-specific bits using VirtualBox with a VM from OSBoxes. That's how I plan to do my "Node.js not installed" testing once I finish making time for it.)

If you want to go a step further, you could use Selenium WebDriver to test how everything integrates together.

(Selenium WebDriver lets you use a jQuery-like API for launching and remote-controlling browsers. Travis-CI has a guide for running such tests from your .travis.yml. If you choose the SauceLabs option, they offer a free option for open-source projects.)

GeckoDriver, ChromeDriver, and OperaChromiumDriver should all support installing extensions during the test process and you wouldn't need to puppet the external editor if you set it to a custom test script.

[asamuzaK]

...and I just realized that I forgot to respond to this:

Thanks, I tried myself, and it is now packaged, zipped, and deployed from travis.

[ssokolow]

Oh, on a related note, If you want to test on Windows versions you don't have, Microsoft offers VMs with a special "only for testing purposes, but you're allowed to reset the Windows trial period" license.

They're intended to let web developers test their creations under various versions of Internet Explorer and Edge, but I don't specifically remember the testing license prohibiting you from using them to test other browsers.

(They currently offer Windows 7, 8.1, and 10. They used to offer XP and Vista to cover versions of Internet Explorer back to 6.0, but those were end-of-lifed.)

[asamuzaK]

Do you see anything which still needs to be fixed in the linux binary?

[ssokolow]

I haven't tested it in some VMs without Node.js yet. I'll try to get that done tonight.

[ssokolow]

OK. I did a quick check and the bundled withExEditorHost starts up perfectly well on a 64-bit Linux VM with no Node.js.

However, I noticed that it's an ELF64 binary, so, given that you have win32.zip and win64.zip, you'll want to rename it to linux64.zip and, ideally, also offer a linux32.zip.

If you have a Linux VM, the simplest way to check that is to run file index and it'll print out a string like this:

index: ELF 64-bit LSB  executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, BuildID[sha1]=753f0807e65ea3f1900fff77e86cea65698c9e3c, not stripped

I'll start working on a Windows installer once I've answered some needinfo requests for Firefox bugs I reported.

[asamuzaK]

Thanks. For the file name, linux32, linux64, macos64, win32, win64 or linux-x86, linux-x64, macos-x64, win-x86, win-x64

Which do you think is more preferable?

[ssokolow]

The former would probably be more understandable for your average Windows user (though that can be improved upon even more) while the latter would be more accurate.

Do you plan to do either of the following things?

1. Keep your options open for eventually offering non-Intel/AMD bundles (eg. ARM for Raspberry Pi)
2. Offer some kind of auto-updater which downloads those builds and would be broken by having the filename change.

[asamuzaK]

Keep your options open for eventually offering non-Intel/AMD bundles (eg. ARM for Raspberry Pi)

Looks like the binary for ARMv7 is available. Releases · zeit/pkg-fetch But native messaging is possible on portable apps? At least on Android, it's not possible now and looks like Google won't let it happen?

Offer some kind of auto-updater which downloads those builds and would be broken by having the filename change.

I did not think of auto-updater at this time.

I decided file names to be win64.zip and win32.zip for Windows users, and more accurate for Linux users, linux-x86_64.zip and linux-x86.zip.

[asamuzaK]

Merged #26

[ssokolow]

Looks like the binary for ARMv7 is available. Releases · zeit/pkg-fetch But native messaging is possible on portable apps? At least on Android, it's not possible now and looks like Google won't let it happen?

Devices like the popular Raspberry Pi series of development boards run ARM builds of ordinary Linux desktops. (The original Raspberry Pi was ARMv6, but all the later revisions can run ARMv7 code.)

Also, there are various other kinds of single-board or ultra-compact computers which can run desktop Linux on an ARMv7-compatible chip, such as:

• The Cubox-i4Pro sitting under my TV
• The OpenPandora in my pocket and its upcoming successor, the DragonBox Pyra
• Various other development boards like the ODROID series.

It's common practice for Linux distros to provide their own builds of Firefox (often the ESR channel), so you can run desktop Firefox on those ARM devices fairly easily.

For example, Debian lets you sudo apt-get install firefox-esr on i386 (x86), amd64 (x86_64), armhf (ARMv7), armel (little-endian ARM), arm64, mips, mipsel (little-endian MIPS), mips64el (little-endian 64-bit MIPS), ppc64el (64-bit little-endian PowerPC), and s390x (IBM Mainframes)... and their unstable channel adds 32-bit big-endian PowerPC, x86 and x86_64 sitting on top of a FreeBSD kernel, and unofficial ports for 64-bit big-endian PowerPC, HP-PA, and PowerPC SPE.

That said, at the moment, I'm not equipped to do a full test of an ARMv7 bundle:

• My Raspberry Pi is the original ARMv6 model.
• My OpenPandora is currently running a distro that, for software-compatibility reasons, turns off the "hardware floating-point math" part of the ARMv7 ABI.
• My Cubox-i4Pro is running a distro that has Kodi talking directly to the GPU without X11, so there's no GUI API for Firefox to talk to.

However, if it's good enough, I could do a basic "Does it start to the point of printing the "I'm ready" JSON?" test by manually running the index binary on the Cubox.

[asamuzaK]

Added linux-ARMv7 Binary looks like it was created without error, but there are some warnings... Job #284.2 - asamuzaK/withExEditorHost - Travis CI

[ssokolow]

Well, ARMv7 is the correct architecture, but those warnings are show-stoppers:

OpenELEC:~ $ unzip linux-ARMv7.zip index
Archive:  linux-ARMv7.zip
  inflating: index                   ./
OpenELEC:~ $ ./index 
module.js:542
    throw err;
    ^

Error: Cannot find module '/snapshot/withExEditorHost/index.js'
    at Function.Module._resolveFilename (module.js:540:15)
    at Function.Module._resolveFilename (pkg/prelude/bootstrap.js:1269:46)
    at Function.Module._load (module.js:470:25)
    at Function.Module.runMain (pkg/prelude/bootstrap.js:1298:12)
    at startup (bootstrap_node.js:227:16)
    at bootstrap_node.js:649:3

Also, architecture strings in Linux package filenames are generally kept lowercase.

(ie. Instead of ARMv7, it'd feel more natural to use armv7, similar to how it's written in things like the LLVM target string for the platform: armv7-unknown-linux-gnueabihf)

[asamuzaK]

I found that linux-x86 and win32 build had same warnings in travis job log.

Found Warning Failed to make bytecode for file · Issue #310 · zeit/pkg

[asamuzaK]

So, it's an upstream issue. For a workaround, I can locally build lnux-x86, linux-x86_64, macos, win64, win32 zipped packages on my win10. But it fails to build linux-armv7 package.

I will omit linux-armv7 until issue is fixed on pkg. Filed #29

[ssokolow]

Understood.

[asamuzaK]

I am considering to release v3.0.0 at this point, and when the installer for Windows is ready, let's release v3.1.0. How does that sound to you?

[ssokolow]

Sounds good.

Sorry for the delay on the installer. I've had other things drawing on my time and NSIS isn't exactly the friendliest option. (It's very low-level, as installers go, so I've been considering trying node-innosetup-compiler plus Wine as a way to build an installer on Travis-CI using a more high-level language.)

[asamuzaK]

I could not come here without your cooperation. I appreciate that.

You can create it without worrying about time.

[ssokolow]

Sorry for going silent for so long. I'm still trying to get other concerns under control, but, to make sure it doesn't get lost, here's the research I've done toward the goal of building an installer:

InnoSetup is the most comfortable to work with, having a nice and well-documented .ini-style declarative format, optionally extended with PascalScript:

• Inno Setup Downloads (Contains the Windows IDE and compiler, for doing the initial development work)
• node-innosetup-compiler (An automated means of downloading and running the InnoSetup compiler from a Node.js-hosted build script which is explicitly compatible with both Windows and, when Wine is installed, other OSes)
• Inno Setup Help (Especially "Setup Script Sections > [Registry] section", which has an example of population a registry key via variable substitution, and "Pascal Scripting", which is necessary for displaying custom wizard pages and dynamically filling the install path in the manifest file.)
• A StackOverflow page which has examples of subbing values into a JSON file (ie. setting the install path within the manifest) during installation using either a regexp or a JSON-manipulating InnoSetup extension.

NSIS could also do the trick, but it's got a very low-level scripting language (even flow-control constructs you'd typically expect to be core language functions are implemented as macros) and something as simple as automatically recording what got installed for use by the uninstaller doesn't come bundled, so you have to grab a user-contributed implementation from the wiki... but, for what it's worth, it's the only open-source option which combines scripting and support all the way back to Windows 95.)

• The Ubuntu nsis and nsis-pluginapi packages enable cross-building of Windows installers if using Travis CI without AppVeyor for package-building.
• NSIS Users Manual
  • Scripting Reference: Registry, INI, File Instructions
  • Appendix D.2: Add uninstall information to Add/Remove Programs
  • Appendix E: Useful Headers
• NSIS Modern User Interface (reference and examples for frontend stuff)
• NSIS Wiki:
  • Best practices
  • Category: Tutorials
  • Category: Code Examples
  • Uninstall only installed files
  • Add uninstall information to Add/Remove Programs
  • Auto-uninstall old before installing new
  • Adding custom installer pages
  • NsJSON plug-in
• Nsis: Modifying a Json file using NsJson - Stack Overflow

Finally, Microsoft's recommended solution is to use the Microsoft Installer (ie. build a .msi file) but the only open-source solution for that is their WiX Toolset, which isn't worth the hassle unless you're building a big, compiex project with a bunch of dependencies like DirectX, the .NET framework, or Visual C++ redistributables that need to have their installers bundled in... and even if it is worth the hassle, the free documentation I examined was a pain in the butt. (It's no wonder InstallShield remained a profitable product after switching to using MSI as a backend.)

[Eli-the-Bearded]

So, about this effort. It's noble and a worthy thing to make this plugin easier to install, I find it to be very fragile on my system, and I know how to use node.

But...

Please don't check the binaries into the git project. .git/objects/pack/pack-8e791fc6259fcab0903eb0d29aeb4a6d69c7ce9c.pack is up to 168 megabytes now, the dest/ dir is another 20 megs, while the total of the source, test, and other text is about 176 kilobytes.

Use the "Releases" method to include binaries, so that we don't git clone a history of every artifact file.

https://help.github.com/articles/creating-releases/

[asamuzaK]

You can also get a package from npm. It may not be the proper usage of npm, but how about trying below?

>npm i withexeditorhost -g
>cd /path/to/npm/node_modules/withexeditorhost
>node index --setup

Updated: NOTE: Do not run npm run setup.

[asamuzaK]

This issue about need for users to set up Node.js themselves is solved by providing binaries. The only thing left is to have an installer, but there seems to be no progress for more than a year. So I will close this at this moment.

If an installer is available, please make a new issue or pull request. Thanks

[ssokolow]

Yeah, sorry about that.

Life became a mess and I'm still trying to get caught up on things.

I agree with the decision to close this, but it's still in my TODO list so I hope to get to it eventually.

[asamuzaK]

I appreciate your cooperation.