Cross-platform installer for holochain software

Holochain Apps have two parts, that are version dependent, the Holochain Core code and the App code. It is conceivable (and virtually guaranteed) that either of these may update during the lifecycle of an app install.

hard requirements

The app must specify which version(s) of the core it is compatable with
Having a copy of that version of the core on the host machine
Running the app with that version of the core
Enabling the app to update, without changing its core dependency
Enabling the app to update and require a new version of the Core

soft requirements

holochains dependent on other holochains
"Identities" (which could well be running different versions of the same holochains (identities are pretty much the step "above" the package manager
Automatic testing of all Holochains, and automatic version rollback or hard security shutdown on 0.0.x security/miniVersion updates (of the core)

go

It seems to me that if "go" is going to be used as the base tool, then "gx" is probably the best tool to use as a base for holochain package management, or else holochain itself, if thats possible. GX is a uses the IPFS distributed filesystem, and hash based addressing to guarantee package contents.

docker

Im currently looking into how far its possible to take the docker image update system. I see no reason why if apps are distributed in docker images, and the core is distributed in docker images, that a host machine couldnt have 100's of versions of either/both stored in a few megabytes or so. Plus docker is completely cross platform. I think leveraging the incredible AUFS (or the new OverlayFS): here have a look at this: https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/#image-layering-and-sharing-with-overlayfs-overlay (there's a diagram in it), removes hundreds of complications of "package management". And it uses content hashes for everything. Cute. there is no github emoticon for Cute. geeks.

When we are discussing the "holochain of holochains" (lets call that HCHC, and talking about I believe @zippy there is some chat about some linked list of "hash types"? is that called "identifications"? So that, e.g., it is "easy" to seach a DHT for all the "posts" by a certain user (and I assume by whatever "index" the app wants).

Seems to me the versions of the core/apps in HCHC could be dockerhub hash id's, and thats almost the entire package management issue taken care of.

Blasted heck, its just occured to me that there may be forks of the core.

Well... thats also just as sorted by docker filesystem.

The thing to clearly understand is that the docker filesystem can exactly mirror the git commit tree, with differential content only. Not quite as cool as git, since it only goes to the INode granularity, but that will do afaiac.

So lets make the "docker design" clear

The host machine contains all the versions (not all the commits as separate versions, all the release commits) of the core and the app, stored in overlay fs (literally pulled from dockerhub/IPFS/HCHC by hashID)
The job of the "package manager" is to ensure that the app volume is mounted into a container based on the correct docker image (i.e. an image containing a version of the core that the app is compatable with). This is trivial.
Updates to the core / app can be rolled out on the machine, and tested, all without destroying any history at all, and rolling back is simply a matter of spinning up a container from an earlier image.
The package manager has enough information to be able rebase onto new initial docker images, where HCHC says that it is possible, when it can determine that no apps on its system are bound to legacy requirements.
Yeah, thats simple
Also, we can dispense with "support for windows", and "using anything other than bash for sys admin"

In my opinion, closing #174 [this issue] is a pre requisite for:

The design is to build on top of two protocols

git
docker

Design for Holochain App package information stored as part of the app's filesystem

.hc - the .hc directory is in the root of the git repository that holds this Holochain App
- immutable - directory !do not change what is in here! - possibly enforced by some link to the git repository name, or some hash or uuid or something
- name - file contain the human readable name of the App
- dockername - file containing the docker tag name of the app (lowercase, restricted regex)
- aboutThisCommit - directory containing information related to current state of the files in the git repository
- releaseVersion - file containing the release version number of this git commit of files
- releaseVersionMap - file containing the history of mappings between git commit hashes and release version numbers
- holochainCoreVersionRegex - file containing the regex specification of which Holochain Core versions this commit of the App is compatable with (run tests automatically to put entry in here?)
- holochainCoreVersionMap - file contain history holochainCoreVersionRegex against git commit hash
- publishingStatus - directory containing information regarding
- Are we publishing? Have we uploaded to git? What about Dockerhub? Or IPFS? or HCHC?
- In Design. Specifically asking for input on this issue . It seems to me that all four of those tools could easily be involved in a large scale distribution network. With the goal that "eventually" (a few hours to a few months, depending on how popular the App is), HCHC would take care of distribution (probabaly using IPFS, with Dockerhub, then github fallbacks)

Notes on .hc

Immutability

Changing the name of a holochain app should be considered a security relevant chane and require explicit forking of the repository

aboutThisCommit

There may well be more things to go in here, and its possible publishingStatus would also have some entries in this directory. Once a commit is published, holochain should be very pissy about changing the filesystem without changing the version number id's in here. I.e. it shouldnt let it happen. If we integrate a bunch of git commands into holochain, then thats pretty easy, and the validation is pretty simple.

Release Versions

When the user wants to release a version of the App into an ecosystem (private, edge, alpha, beta, production), the package manager will update these files to represent the user's actions. In essence all that is required is to update these files, an update to be pushed to github (or some other git service), and then the relevant people and/or services update themselves in response. In the future we can implement a distributed version of github, "git clubs on holochain", if it is determined that github's centralisation is not useful / is damaging. Irrespective, information infrastructure in .hc is still just as valid, and the evolution path is not painful. Certainly at this moment, I dont see github ceasing to be part of the development cycle in the next 3 to 6 months to a year or so, and perhaps never completely.

So what if I want to actually run a holochain app goddamit

Install the holochain package manager (requires docker)
1. downloads the HCHC's .hc directory, pulls the requisite Core image, and spins up an instance with "public central" HCHC genesis mounted onto it.
2. connects to HCHC (is this centralised? if not, are there "known" high speed servers with the entire contents of HCHC on them?)
3. Seems pretty clear looking at this, that HCHC should be HCHCHC initially. The holochain of holochain holochains, since that will like run into MB over years, rather than GB.
browse the "repository"
1. By community? GeoLocation? Interest? Genre? Send it your web history and let it pick for you?
select an app
1. Select an community? Apply to join a community? Start a community?
download app specification (at minimal, this is just the current/desired version of the .hc directory, plus genesis data, and could be held reasonably on HCHC without IPFS support. nice.)
run app
1. use docker "FROM" command to get the docker image of desired version of the holochain Core from Dockerhub/IPFS
2. download the app source code (dockerhub/ipfs/github)
3. Spin up a docker container using the relevant holochain Core image with the App image mounted onto it.
  1. Either genesis or join, with the HC from the container Core
  2. possibly connect through some new user boost protocol. The spiderweb can come in handy in a number of ways

Easy Package management, 174

BTW, as pointed out by @zippy, the description of how HCHCHC would work is not a minimum working solution, it is a bit of fun to show how it interacts well with the docker/git design paradigm.

As a minimal tool, all that is necessary is to grab what is on HCHC / HCHCHC and display it nicely to the user as a list (like chrome extensions). As long as the data on HCHC links the app source to the correct release commit of the core, which will be available as a docker image. Its paramount to recognise that the docker images for each realease of the core would only contain the delta's from the previous image, meaning that a system can easily host all versions of the core, and serve apps from the correct ones for as long as needed.

The only hard requirements with any complexity or depth are the need to manage genesis and sharing of genesis blocks. Thats not too hard, and I'm not sure if it makes sense to progress without people being able to form their own community from a holochain, and offer other people to join existing ones.

I really like this approach if we can make it user friendly enough -- especially the ability for docker to host multiple versions with stacked deltas. There are a few things I've been planning for the HCHC/Holochain of Holochains that you should probably know about. An entry in the directory not only has Name, Description, tags, bootstrap servers, public gateways, index servers, etc. for supporting function, but also links to other holochain entries for "Replaced by" and "Replaces" for providing continuity between versions of applications regardless of name. And at some point, those entries could validate that any internal code governance procedures were followed (e.g. signed by the benevolent dictator, or m of n signatures from the code committee, or a threshold of votes from users, or whatever...).

Also, with the use of public gateway servers for HCHC we could enable file downloads without IPFS, and IPFS gateways could be listed when files are shared that way.

And in light of this proposal, that HC admin web interface we were talking about the other day, probably should not just run out of a single HC core, but instead be able to list the core versions, apps and versions, storage space being used by them all, and probably also recent performance statistics (like amount of cpu/network use).

I like where you're headed with this to try to keep it from becoming a version dependency nightmare.

End-User experience

I really like this approach if we can make it user friendly enough

install docker
- no need for doing anything with users on the machine or anything like that
run the holochain installer with sudo (*nix) or as an administrator user (windows)
- seems plausible the holochain installer could install docker itsself if its not found on the system
- also a specific installer might check first on a web server, for what version of docker is compatible with the latest/desired version of HCHCHC
open the browser shortcut created by the installer to connect host/localhost:
- documentation to support anyone developing remotely (which is what I do, so no biggy)
sign up / sign in
interact with the HCHCHC interface

HCHC design

An entry in the directory not only has Name, Description, tags, bootstrap servers, public gateways, index servers, etc. for supporting function, but also links to other holochain entries for "Replaced by" and "Replaces" for providing continuity between versions of applications regardless of name. And at some point, those entries could validate that any internal code governance procedures were followed (e.g. signed by the benevolent dictator, or m of n signatures from the code committee, or a threshold of votes from users, or whatever...).

I agree with all of this, the design seems straightforward, and evolvable without dependencies on anything narly

Centralised access for HCHCHC and HCHCs

Also, with the use of public gateway servers for HCHC we could enable file downloads without IPFS, and IPFS gateways could be listed when files are shared that way.

I agree

Basic description of HCHCHC app

And in light of this proposal, that HC admin web interface we were talking about the other day, probably should not just run out of a single HC core, but instead be able to list the core versions, apps and versions, storage space being used by them all, and probably also recent performance statistics (like amount of cpu/network use).

The web part of the HCHCHC App I see to be what you are describing here

The HCHCHC App is installed automatically by the admin container, in the same way as any other Holochain App
The admin container is run as a special user on the system (exactly the same design as www-data), and that user is part of the "docker" group (and probably runs as administrator on windows)
The HCHCHC app is granted special access to an interface middleware go program running in the admin container
The installer can inject a unique 2048 byte salt into the configuration of HCHCHC during installation, which HCHCHC uses in requests to the middleware interface
- or perhaps HCHCHC should be installed in the same container "the admin container". My intuition is that is messy, but Im not sure about the security implications, though nothing we code is going to create any possible security issues.
Inside the admin container is a go middleware, which is an interface between HCHCHC and bash scripts which control the docker daemon or that interacts directly with docker via JSON RPC.
- the argument between bash script or RPC stuff, whether for everything, or some things or no things is not immediately apparent, but shouldnt be too narly.
The middleware interface will read such datas about installed Apps, to pass to the HCHCHC App, and take care of all downloading / docker image creation (Holochain Core and App installation), as requested by HCHCHC.
All actual App Data to be stored on the host filesystem in the .holochain directory, and "identity management" simply a case of swtiching the location of .holochain for any specific container/App spin up
This is sane, and secure, afaict.

HCHCHC is a Holochain App, therefore it will be running inside a docker container as an app applied to a core, and accessible through a web browser on some port on the host machine. The installation of HCHCHC will be the process of (programmatically) getting the correct Core and App stuff, and spinning them up like any other holochain app, and represents one of the final stages of the "Holochain Installation Wizard"

One thing I'd like to add to this discussion is that people have been talking about equinox as potentially relevant to this.

holochain / holochain-proto