split static structure (scripts) from modifiable/updatable user data (stones)

[dalehenrich]

@DamienCassou is working on building a Nix package for GsDevKit_home as a prerequisite to working on Issue #22.

The model for GsDevKit_home that @DamienCassou is using is based on having the static structure (primarily scripts) installed in a readonly component while the modifiable data (stones, pharo images, git repos, etc.) would be installed in a user's directory structure ...

I agree that this is a good structure to use.

The plan is to create a sysAdmin branch with the static structure and a user branch with the modifiable data .... over time, the _home project will migrate toward using this model by default ...

[dalehenrich]

Here's the top-level directory structure for a full installation of the master branch:

▸ bin/
▸ dev/
▸ docs/
▸ etc/
▸ server/
▸ shared/
▸ sys/
▸ tests/
▸ travis/
  license.txt
  README.md

Here's the directory structure in the raw master branch:

▸ bin/
▸ docs/
▸ etc/
▸ shared/
▸ sys/
▸ tests/
▸ travis/
  license.txt
  README.md

These are the bits that should be in the raw sysAdmin branch:

▸ bin/
▸ docs/
▸ tests/
▸ travis/
  license.txt
  README.md

These are the bits that should be in the raw user branch:

▸ shared/
▸ sys/
  license.txt
  README.md

An initial install of a the user branch will use symbolic links to the sysAdmin installation to reconstitute a raw master branch appearance:

▸ bin/ 
▸ docs/
▸ shared/
▸ sys/
  license.txt
  README.md

[DamienCassou]

I plan that my nix package create a /etc/gemstone/paths.sh which will contain environment variable assignments. That way, the scripts in the user directory just have to source this file

[dalehenrich]

@DamienCassou I assume that you are referring to the GS_* env vars? Hmmmm, users are supposed to be able to change some (or all) of the env vars to customize bits and pieces of the environment .... or are you proposing that /etc/gemstone/paths.sh will simply call $GS_HOME/bin/defGsDevKit.env? ... or are you proposing that /etc/gemstone/paths.sh being a summary ... the scripts are built to fill in missing pieces (lazy initialization) of the structure in the users structure .... and of course some of those pieces are optional ...

I guess I'm saying that a single, central /etc/gemstone/paths.sh file will make things quite a bit more difficult to implement....

[DamienCassou]

I'm sure we can find a solution. Let me explain the problem. I have a functional package manager where a single package can be installed multiple times on the system. Each package goes to /nix/store and is named with a hash. For example, Emacs is at /nix/store/4y284924qmks6ry0ppj0pmspi8fhw7ph-emacs-24.5. Each time I install Emacs, because of a new version of Emacs itself or of one of its dependencies, a new directory is created under /nix/store with a new hash. This means it is going to be difficult to create a symlink from HOME to a package directory because the directory will move very often. I guess one way to solve that is for a package to register itself in a globally known directory such as /etc. Nevertheless, I never tried that so I'm not sure it will work.

[dalehenrich]

I read a little bit about Nix yesterday I did see that each release is independent of every other release and that it is possible to have multiple releases available on the same machine concurrently, but I'm afraid that my fundamental approach is the antithesis of a "functional approach".

The system I am building with GsDevKit_home is really an extension of the image-based Smalltalk development model expressed on disk ... I expect the system to evolve organically and I have built in optional chunks that can be controlled/extended by the developer ... I expect that both the scripts and the user structure will evolve over time, so there is a synergy between the scripts and the disk structure that is owned by the user. The structure/scripts/env vars have all changed significantly in the last month.

I really see the disk structure as an extension of the Smalltalk image and I expect that any piece of the system can be modified to meet the needs as they evolve, so to pick one segment of the system and call that read only is equevalent to saying that "you can't add or change methods in Object in a Smalltalk image" ...

I'm basically using git as my package manager. The script $GS_HOMNE/bin/updateGsDevKit is designed to update the system to a later version. I chose git because I want to encourage users to contribute to the project and share any changes that they might make. The system is evolving in concert with the Smalltalk code in the image, so the entire system (user structure and scripts) needs to evolve in step, just like a Smalltalk image evolves as an organic unit ...

As I said, I've read a bit about Nix and I sorta wish that I had known about it before I started, because I do see the value ---- But I'm afraid that the route I've taken is not very friendly to the Nix package manager --- It seems to me that in order to be compatible with the Nix package manager I will have to re-architect the entire system ... and even then I'm sure that I can fit the system that I am building into the Nix model.

[dalehenrich]

The alternative that is certain to work, is to not try to package things up using Nix ... the model that GsDevKit_home was designed for was one in which each user would clone the GsDevKit_home repository (as described in the README) for each set of stones that are sharing a common code base ...

I am certain that other deployment architectures can be designed for GemStone, but I barley have the bandwidth to maintain the GsDevKit_home approach:)

[DamienCassou]

Thanks for the explanation about GsDevKit_home architecture, it makes sense. I will think about it.

[dalehenrich]

Sure, I think in principle that I like the idea of packaging things with NIX and perhaps when the evolutionary period is over and things stabilize structurally, it will be more appropriate....

How are data bases like Mongo or Postgres packaged with NIX ... these are products that have a combination of fixed scripts/functionality and user-owned writable data .... perhaps there's inspiration to be had there ...

[DamienCassou]

database executable files are in a root-owned directory under /nix/store and cannot be modified. The database data are in /var/db/ (see https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/databases/mongodb.nix)

[dalehenrich]

Since I don't speak nix:) How are per user databases and multiple databases per user handled if they are in /var/db ... is there a separate user data structure underneath /var/db?

[DamienCassou]

The mongo service runs under the mongodb user. All files under /var/db/mongo are owned by this user. When I use the mongo CLI, I send queries to the local mongo service. This is a standard client/server architecture.

[dalehenrich]

How are multiple mongo databases handled in this scenario .... is there a single mongo data base or can there be multiple data bases?

I create multiple GemStone databases just like you might create multiple Pharo images ... do you have a pharodb user that manages all of the Pharo images, or are Pharo images owned by the user the creates them?

[DamienCassou]

You can create as many databases as you want.

[DamienCassou]

The traditional Smalltalk way I guess is to have the Smalltalk images under the home dir of each user. But traditional databases don't work like that. Applied to GemStone, this would mean a GemStone server always listen and is able to create GemStone databases on demand by clients.

[dalehenrich]

Well, GemStone isn't really a traditional database:) "listening for requests to create databases on behalf of clients" is technology that does not exist ... The best way to think of this is that GsDevKit_home is the first step on a long path that will ultimately lead to a model that might behave the way you suggest ... this is why I talk about evolution .... at the moment there is no reason to treat a GemStone stone differently than a pharo image --- GemStone is closer to a pharo image than a traditional database anyway ---

I do not oppose moving GemStone to the point where it can be used more like a traditional data base, but there is a significant amount of work required to support this new usage model and I don't have the bandwidth to go down that route right now ...

So for the next 6 months or so (at least) I will be riding the GsDevKit_home model and expanding the services that surround GsDevKit_home and GemStone ....