Improved Linux Install and Overall Experience

[kantai]

The current Linux install experience and instructions are a little bit lacking. We give users a script which tries to setup a docker environment for running the browser service and a local api endpoint.

There's a routes to go towards improving this experience.

1. Improve the Docker Experience a. Issues with the script -- it seems like the biggest issue is in password selection. We could just try to auto-select a password, and then pass this along to xdg-open -- or output it after startup. b. Other script issues? Sometimes blockstack-api fails to start, I believe because the registrar lock isn't getting cleaned up. c. The images are huge. Like 1GB. d. Docker is a pain to set up on a bunch of different platforms

2. Ship a Virtualbox image or QEMU image a. We need to still perform the setup -- so the password is still an issue. b. Would users access the browser from inside the VM? c. How do we automate building these images?

3. Recommend pip plus a package for the browser? a. pip works fine, but there's still an issue of configuration and setup. b. would also need to solve the problem of packaging the browser -- we have an apt package, which would work, but again, that's something that needs to be attended to. c. alternative to pip is also just to do a deb package but with the full virtualenv, dhvenv I think is the route people go here.

4. An orchestration app like what macOS has now.

[kantai]

Tagging @jcnelson @jackzampolin for insight on this.

[jcnelson]

Re Option 2:

• What setup do we need to do? Can't we just automatically generate API passwords and encryption passwords?
• Yes. The VM would simply start Chrome on login, pointed to localhost:8888.
• It's not too bad. We can just kick off a script that will pull directly from git and run some initial setup commands.

I'm happy to take a stab at this option. I think it offers the most consistent user experience, especially in light of how may corner-cases and gotchas there are to getting Docker working on a given Linux distro.

[vsund]

I'm not sure whether option 2 is good for usability. Isn't it too much effort to start a VM everytime you want to access the browser?

[kantai]

On using an in-VM browser for option 2: UI things inside of a VM are consistently Very Annoying, like Copy-and-Paste

[kantai]

What setup do we need to do? Can't we just automatically generate API passwords and encryption passwords?

Yes, basically just that. Those can just be generated on the VM's first boot I suppose.

It's not too bad. We can just kick off a script that will pull directly from git and run some initial setup commands.

Is that when the VM boots? Or does the VM image already have everything installed?

[jcnelson]

I'm not sure whether option 2 is good for usability. Isn't it too much effort to start a VM everytime you want to access the browser?

The status quo is that most users have difficulty getting Docker installed and running correctly. Even for the ones that have Docker installed correctly, not everyone has the XDG tools installed and configured. This breaks protocol handling in hard-to-debug ways. Shipping a VM with everything inside already configured would increase UI consistency.

On using an in-VM browser for option 2: UI things inside of a VM are consistently Very Annoying, like Copy-and-Paste

Since we control everything inside the VM, we can just script it to install the Virtualbox guest add-ons. This would enable copy-and-paste on all OSs.

Is that when the VM boots? Or does the VM image already have everything installed?

My approach would be to set up a vanilla QEMU image, and create a CI script that would:

• launch qemu
• ssh into it
• install Blockstack and all related components from git
• auto-generate the wallet and API passwords
• convert the QEMU image into a virtualbox image

The vanilla QEMU image would be set up to auto-login, start Blockstack, and and run Chromium full-screen on localhost:8888. The CI system would simply install Blockstack from git to generate a virtualbox image for each release.

[jackzampolin]

So for me this just seems like we are adding yet another install path that will be not maintained, will be under documented, and is soon-to-be depreciated. This sounds to me like a classic bad idea. If you are coming to VBox with no context (which I would say is a lot of users) that and docker both have a very similar amount of learning overhead.

With this solution we will be switching one virtualization engine for another with limited gains. I understand the criticisms of the current install path, but I feel like putting in the effort to do another one that will have equally bad support and docs moving forward just makes our problems worse not better.

@jcnelson @kantai

[ErikBjare]

@jackzampolin nailed it.

You should try to avoid virtualization and containerization if you want to make it easy for end users; switching from Docker to a VM won't make it easier. Not to be an ass about it, but I can't believe you actually thought shipping an end-user application as a VM would be a good idea.

Why do you need containerization in the first place? I'm legitimately curious, I sure you have some reason, I just doubt it's a good one in the long run.

Why can't you just put in the work and do what every other application does: simply ship a zip with precompiled binaries, or a proper deb package. Is it really that hard to get it working?

The best solution would be if you could build an AppImage or flatpak. That would be a great solution not too far from how .app bundles work on macOS.

[kantai]

With this solution we will be switching one virtualization engine for another with limited gains. I understand the criticisms of the current install path, but I feel like putting in the effort to do another one that will have equally bad support and docs moving forward just makes our problems worse not better.

Yeah, I'm kind of inclined towards the "let's not try to fix an old problem with a slightly different trick" argument, especially if we haven't finished fleshing out the exact problems we're trying to solve with a new path.

Positives of new path:

1. Less situational-dependent: volume mounting in aufs in Docker is legitimately confounding of blockstack-core in a number of different ways (the secret passing file checks, pid files, /tmp/ directories), and these manifest differently in different host OS's.
2. Skirts any issues with the protocol handler using xdg-open which doesn't exist on all Linux distros.
3. Virtualbox is easier to install and setup than Docker
4. No question anymore of "is blockstack running?" -- if the VM is running, blockstack will be running.
5. No port binding issues

@jackzampolin -- do you think these benefits are still too minimal to merit a VM?

[jackzampolin]

1. These are issues I believe need to be fixed in blockstack-core to enable larger deployments. I've been opening issues to try and fix some of the problems you are mentioning here to enable the docker usercase. We know these issues exit and I think we should fix them so that we can run in docker. This will also fix some issues that we run into with the normal pip install.
2. Valid
3. See above. I don't believe this is a valid criticism
4. This is prob not necessarily true. People will have to dig into the VM to debug
5. Valid

I don't think the minimal gains merit doing the work for the VM.

[jcnelson]

Here are my issues right now:

• Multiple unreliable containers. Can we please try to consolidate all Blockstack processes into one container? I want the stop directive to kill everything, and I want start to always work correctly. Right now, sometimes the stop directive has a tendency to leave some containers running. This prevents the subsequent start directive from working correctly.

• Unreliable clean-up. Sometimes some registrar state gets leaked to the host's /tmp directory, which prevents start from working after stop.

• Shared ownership of ~/.blockstack. Some users have a ~/.blockstack directory already, since they were CLI users. We need to make sure the Blockstack Browser script has its own, separate ~/.blockstack directory (let's say ~/.blockstack-browser), so they don't accidentally erase their old wallets.

• Unreliable Protocol-Handling. There are lots of corner cases with XDG tools not handling blockstack: URIs correctly, and us not being able to figure out why.

• User-generated Core Password. Users get confused as to why they need to know two passwords (12-word phrase and this one). Can we generate this automatically for them?

• Docker is hard to install. This isn't to say that Virtualbox is easy. But, if you look at our forum, you'll see this is a pain point for users. Docker does not have a GUI, and does not ship with all major distros. The installation instructions require users to drop into the CLI, add 3rd party package repositories, install Docker, and then add themselves to the right user group. The exact opposite is true for Virtualbox--it has a GUI, it ships with all distros, and no further work is required to use it once the package manager installs it. It also has a generic Linux installer that puts it into /opt that works with anything newer than RHEL5 and ships with glibc.

I am of the opinion that a full VM makes these problems easier to address, but I don't care too much either way as long as they are addressed.

[jcnelson]

@ErikBjare I am happy to write a "How to install Blockstack from Source" document if you prefer.

[jackzampolin]

Comments on a couple of the points above:

• Multiple unreliable containers: This is the right way to build applications using docker. Personally I think that one of the reasons that the integration tests are so hard to debug is that everything is in one container. I would love to be able to dig into each process. This also reduces the size of the containers and gives other benefits as well. I think the containers are unreliable because the processes running inside are unreliable. We could fix some of these issues by specifying a docker-compose file, but last time I suggested that it was not liked.
• Unreliable clean-up: This can likely be fixed by not mounting the tmp directory. I had this working on some version of this installer that's been lost to git somewhere.
• Shared ownership of ~/.blockstack: This was a requirement for the installer not a bug. Also that is the canonical location of the Blockstack folder in linux. Why would it be elsewhere?
• User-generated Core Password: We could do this for them, or even just automatically use a set string. This wallet is no longer used so it can be safely ignored. This was not possible when the installer was written and has since changed.

A number of these issues could be fixed by fixing the things that make running in docker a bit painful. That seems like it would be just as much work as throwing it all out and starting anew.

[ErikBjare]

@jcnelson That would be great.

[jackzampolin]

So after meeting with @jcnelson and @kantai we decided to take a 2 pronged path: improve the docker install and document the installation from source. The docker install has the primary downside of docker being difficult to install. The from source option has the the primary downside of requiring full python (2.7) and nodejs (6.9) dev environments setup.

@jcnelson will document the from source install @jackzampolin will make the following changes to the launcher script:

• Add better error messages and eliminate silent failures
• Change install path from ~/.blockstack to ~/.blockstack-browser to avoid messing up CLI users
• Remove anything stored in the /tmp directory on start or stop
• Switch to using blockstack ping to generate client.ini on first run then use the API pass as the wallet password when running create-wallet
• Eliminate the need to input password when starting (use the client.ini API pass for this)
• Improve usage documentation to fully document all available commands.

[larrysalibra]

@kantai where are we on this?

[jackzampolin]

I need to pick this work back up.

[arisgk]

Some more feedback here: I tried to login with Blockstack using Ubuntu and it was not clear that I had to run the bash script before being able to use Blockstack Browser. I am attaching a screencast of the experience before running the script and following its instructions (xdg-open links just do nothing):

[blockpac]

Any progress on the Linux install? (Preferably without using Docker)