Build docker based multi-node integration teststing

[zippy]

Assume a simple layout of the test directory in any particular holochain:

test -testfile1.json -testfile2.json -...etc...

• scenario1
  • role1
    • testfile1.json
    • ...
      • role2
      • ...
      • scenario2
      • ...

json files at the top level get processed when you execute 'hc test mychain' (this is the current standard, i.e. non-integration tests)

But the other files (in the directories) should get processed when you run 'hc test mychain scenarioX roleY' and the docker magic that would make true integration testing possible, would be docker scripts to inspect that directory structure, and do as follows:

• for each scenario, 1) create a set of docker instances identified by each of the roles 2) clone/join each of the roles 3) execute 'hc text on each of the docker instances 4) collect up and join together the ouptut from all the instances.

This would require a couple standards, 1) choosing one of the roles to be the clone, and the rest to be the join, lets just say earliest alphabetic is the clone. 2) timing. The testing commands have to know about time to be able to say some like: wait till start + 200ms This means that somehow we have to communicate start time to all the instances. I'm not quite sure what's the best way to do this. Some options 1) I could have all the of test instances be in a tight loop watching for a file on a shared drive to change state, (kind of like dropping the flag to start a race), 2) if we know approximately how long it will take to fire up all the instances, there could be an absolute start time passed in as one of the args to hc test and all the instances just wait until that time to get started. 3) each testing instance waits on a TCP socket for a signal from the starting script to start. 4) we use a unix signal like SIGINT (i.e. ctl-c) that the script fires off in quick enough succession (is this even possible?)

Note that this scenario/rol version of 'hc test' is slightly different than the usual one, because it doesn't do an implicit clone, which the regular hc test does.

[christopherreay]

Id suggest a generic, simple tcp port based communication between the hc peers for message passing on a "testing" port. Port disabled by default, and possible to compile without it completely

The decision contrasts for this seem to me to be around how far the gossip protocol would need testing before it can be relied upon to be part of the testing mechanism. There might also be useful or valid tests in userspace which would want to determine the efficacy of a certain gossip implementation or design in the working context.

This could all be implemented using docker magic, but I think the implmentation would be brittle and difficult to extend to evolving usecases, especially when addressing sequences of iteractions between peers

If there is a holochain of holochains way of doing it, I cant see it right now xx

[GerryG]

For testing, you are talking about multiple instances each in a container, no? They can all just use the same port and you just need to use locally defined hostnames bound to each container. Canonical local hostnames get defined by the container tools you use (I'm most familiar with docker compose) I haven't tried any configs that call for multiple instances of a type, but the docker-compose generated names have a number at the end.

When you start doing production scaling, you really want typically one container per provisioned cloud server. You might have a couple more containers to perform functions related to the main container, but you want to try to saturate the virtual server with each app instance, and then scale the number or size of those with demand. Now you are talking a different set of tools to configure load balancing and other resources. But for local (testing and dev) It is all one host. If you develop on Windows or MacOS, then you are good running testing/dev between containers, but between the native environment and the docker containers is tricky. All inside or all outside are straightforward.

[GerryG]

Reading in more detail. For this,I would use docker-compose to configure the instances. Tests scripts would be able to turn setting in the docker-compose.yml file. You might even write it with top level 'environments' having the whole scenario/role matrix into configs (you could share when different scenarios repeat the same role(s).

You set up a link section in each scenario that names all the roles it uses, then:

docker-compose up -d scenario1

starts all the containers in the scenario. Some containers won't need to run anything beyond the startup CMD setting. A test script can use docker exec to run the actual test sets on one of more of the containers. If you need to run groups of tests in sequence and in parallel against a group of containers, that should be fairly straightforward.

Then: docker-compose down scenario1 And repeat for the rest in a script ...

[christopherreay]

how about having a "testserver" go program that runs alongside hc in a container and manages synchronous communication

[christopherreay]

I feel like im thinking about implementing a sychronous communications protocol in order to make tests for gossip. Is that in synergy with the intent?

[zippy]

So I follow two maxims: YAGN (Ya Aint Gonna Need it) and "make it as simple as possible, but no simpler" The first means: don't build over-generalized solutions. The second means, don't build under-generalized solutions. In this case, I don't know what test tooling app developers are going to need, so I'd recommend sticking with building the easiest simplest thing, which just allows allows the hc app to listen someplace so as to start the tests at roughly the same time. Everything else should be handled by the app itself, because the idea is to test the app (and any synchronization issues) in as much of a real-world situation as possible, i.e. with no other other synchronization tools available, because they won't exist in the holochain world, unless that becomes part of the holochain ecosystem, in which case the apps will use that, and it shouldn't be part of the testing harness.

[christopherreay]

What about having a spiderweb protocol as a permanent fixture between HC instances. Synchronous accesss, no guarantees to delivery. Absolute addressing. Only the sending of integers to on connected node. I think that looks beautiful in the big picture of holochain. Give the fundamentals for any kind of sychronised operations (e.g. testing)

I an make a simple "pid.lock" kind of widget which notifies all the hc containers when all the containers have been built.

[christopherreay]

Design

• docker-compose service "post.up"
  • manages a single .lock file accessible to all hc containers
  • container "hc serve" holds until the .lock file is deleted.
  • all containers "hc serve" start within milliseconds of each other

or "hc test" or whatever

[zippy]

That sounds great.

[christopherreay]

It'll be brittle as hell woot woot

[GerryG]

I think this pretty much agrees with my points. YAGN is more or less, don't worry about it that much until you do need it. I think we have established a need for:

• Configuring various multi-container test scenarios, probably with docker-compose
• Running a number of test case sets in each scenario config
• Need to coordinate beginnings, endings, overlaps and recovery for each test sets (a test-runner)

I like the post-up idea, and I think it could be extended to be a 'test-coodinator'

Implementation wise for docker, would you just mount the shared resource into all the containers with the 'volumes' directive in docker-compose? You could just mount the single lock-file, or you could mount a directory in anticipation of allowing for any number of named locks in addition to app start up.

I expect that a common developer use case will be one or more plain 'hc' containers to establish a test DHT environment that the application containers can test against. Then you'd have one or more containers that run a custom app with the holochain library.

If I understand the ceptr architecture, that could be a whole swarm of ceprts each with distinct code (dna), and heterogeneous swarms of these might be tested for how they coordinate.

The plain hc containers probably are more or less generic until you put something in them, like a DB container in a typical application.

[christopherreay]

@zippy Get on to me about pulling the logs in together. There is a mechanism there already, but it will be much easier to fine tune with actual log output you want

https://github.com/metacurrency/holoSkel

editted: and btw it doesnt inspect the tests directory as of yet lolz.

@GerryG Hey Gerry. Ive created a generic tool. have a look, what do you think?

[christopherreay]

Assume a simple layout of the test directory in any particular holochain:

• scenario1
  • role1
  • testfile1.json
  • role2
• scenario2 The other files (in the directories) should get processed when you run 'hc test mychain scenarioX roleY' and the docker magic that would make true integration testing possible, would be docker scripts to inspect that directory structure, and do as follows

for each scenario, 1) create a set of docker instances identified by each of the roles 2) clone/join each of the roles

• seed service operates before the hc services are started and a shared file system has the seeded chain in it.
• there is an example which does the join in the source 3) execute hc test chain scenario role

• easy peasy 4) collect up and join together the ouptut from all the instances. Aye, we can do this np, put all the log output into a unique directory on the host machine with a single line of code. Something like

  hc test $chain $scenario $role | tee -a /host/logs/test.$chain.$scenario.$role.log

@zippy Im leaving matthew's tomorrow. Give me a shout if you want anything today. Do you have some tests in mind to test this with? Or shall I work with you on that? How is gossip?

[christopherreay]

OK @zippy

Can you put in a pull request for holochain/holoSkel that has a latest working sample holochain in it?

What im thinking as the next step is this:

• [ ] each scenario gets run as a cluster. Each container in the cluster represents a role. Each role has a set of test files.
  • this is a couple of for loops in bash
• [ ] create a script which makes it easy to indentify and log into the running "role" containers whilst the test is going on
  • this is easy peasy with docker, its just a question of identifying the correct container name. We already know that the container name will be "devchainhc#" where # is the scalar of which container we want to attach to
  • using docker ps you can list the running containers
  • using docker exec <containerName> /bin/sh is how to join a running container
• [ ] push all the log files into a shared space
  • there is already a shared space and a unique identifier for the test run.
• [ ] wait until each cluster has completed all its tests before spinning up the next scenario/role
  • bash should take care of this

item 2 could be useful a lot for mini development cycles for working with hc serve's outside of the proscribed testing mechanism.

[zippy]

First I'm going to get scenario/multi-role testing working in hc, then I'll do the pull request.

[zippy]

as of 5c1161b0 on master you can now do hc test <chain> <scenario> <role> and it runs the test files in the chain subdirectory: test// so now I'll move on to actually creating some scenarios and roles for a test chain!!

[christopherreay]

as of d693cf45ba596a41928caab08433d327b1360b7e in holoSkel one may supply as an argument the name of a scenario to the script Scripts/testScenario and thusly cause to be happening stuff of the correct nature

[zippy]

Currently scenario/role testing does a gen chain but makes no sense. Fix that with option to not gen..

[zippy]

While implementing the last fix, I realized that it makes no sense to have multiple test files under each role in the scenarios. This is because each testfile in the normal test directory is an implicit scenario in that causes the chain to be reset and regened. In the multi-instance setting each scenario is what causes that. Thus, the directory structure really needs to be like this:

• scenario1
  • role1.json
  • role2.json
  • ...
• scenario2
  • ...

[christopherreay]

acknowledged. I'll update the scripts in the morning my time xx

[christopherreay]

so the script, works "as is", and but Ive removed the trailing ".json" from the filename before it is passed to hc test

[zippy]

this is now working as of 86a75e9e2d5fa1fa162b2e04721de03856518b7d