Add integration tests for 'stable env'

[srenatus]

Turns out this is probably the lowest-hanging fruit right now. This makes use of the previous commit's "run in temp dir" feature.

Part of #28.

Happy to discuss the approach taken in this PR's first commit -- using the shell wrapper to cd into the right place. I've tried getting this to work in some other way -- using an exclusion group for side-stepping concurrency issues in @chdir, but couldn't get it to work. After that, the present solution seemed OK enough 😉

Curious about all the input I can get here :) What I dislike most is the duplication in the new env tests, but I couldn't come up with a better structure.

[srenatus]

@mfelsche @jemc if anyone of you got a moment, I'd appreciate a glance. 😃

[mfelsche]

I am fine with creating a temporary directory in the current folder in general as long as it is guaranteed to not clash with anything existing.

I wont have much time in the next months but ill try to have a look at this soonish.

Thanks for your efforts in bringing stable forward.

[srenatus]

🤔 I'm a little concerned that this has failed on CircleCI for some reason -- this indicates that the tests are flakey, which will be annoying long-term.

However, I can run make integration in a loop without any failures. Just tried a few hundred iterations. This makes me suspect that it could be an actual timeout, i.e. the execution on the (presumably shared) infrastructure provided by circleci may just be too slow every now and then. This hypothesis is difficult to verify, but I suppose doubling timeouts for integration tests is a thing we could try (100ms -> 200ms).

[srenatus]

Huh, this happened again right away. Trying to dig in...

Running the tests locally using the circleci tool:

$ make clean # will pick up wrong arch binary otherwise (not rebuild it for linux)
rm -rf build/release
$ circleci build --job vs-ponyc-release

====>> Spin up Environment
Build-agent version 0.0.5895-01cdb92 (2018-05-29T20:07:10+0000)
Starting container ponylang/ponyc:release
  using image ponylang/ponyc@sha256:765b1184c04f69d46cd7174790634b5badfbc3e6345459337c1dba0f9f3a3872

Using build environment variables:
  BASH_ENV=/tmp/.bash_env-localbuild-1529996007
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=sr/integration-test/env
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=vs-ponyc-release
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://github.com/ponylang/pony-stable
  CIRCLE_SHA1=8d8a906c69921050b18eded39159c1073afb071d
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1529996007
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/sh
mkdir -p /root/project && cp -r /tmp/_circleci_local_build_repo/. /root/project
====>> make test integration
  #!/bin/bash -eo pipefail
make test integration
mkdir -p build/release
ponyc  --debug -o build/release stable/test
Building builtin -> /usr/local/lib/pony/0.23.0/packages/builtin
Building stable/test -> /root/project/stable/test
Building ponytest -> /usr/local/lib/pony/0.23.0/packages/ponytest
Building time -> /usr/local/lib/pony/0.23.0/packages/time
Building collections -> /usr/local/lib/pony/0.23.0/packages/collections
Building files -> /usr/local/lib/pony/0.23.0/packages/files
Building buffered -> /usr/local/lib/pony/0.23.0/packages/buffered
Building term -> /usr/local/lib/pony/0.23.0/packages/term
Building promises -> /usr/local/lib/pony/0.23.0/packages/promises
Building strings -> /usr/local/lib/pony/0.23.0/packages/strings
Building signals -> /usr/local/lib/pony/0.23.0/packages/signals
Building capsicum -> /usr/local/lib/pony/0.23.0/packages/capsicum
Building .. -> /root/project/stable
Building json -> /usr/local/lib/pony/0.23.0/packages/json
Building format -> /usr/local/lib/pony/0.23.0/packages/format
Building options -> /usr/local/lib/pony/0.23.0/packages/options
Building integration -> /root/project/stable/test/integration
Building process -> /usr/local/lib/pony/0.23.0/packages/process
Building backpressure -> /usr/local/lib/pony/0.23.0/packages/backpressure
Building regex -> /usr/local/lib/pony/0.23.0/packages/regex
Generating
 Reachability
 Selector painting
 Data prototypes
 Data types
 Function prototypes
 Functions
 Descriptors
Writing build/release/test.o
Linking build/release/test
build/release/test --exclude=integration
1 test started, 0 complete: stable.Bundle started
1 test started, 1 complete: stable.Bundle complete
---- Passed: stable.Bundle
----
---- 1 test ran.
---- Passed: 1
ponyc  stable -o build/release
Building builtin -> /usr/local/lib/pony/0.23.0/packages/builtin
Building stable -> /root/project/stable
Building files -> /usr/local/lib/pony/0.23.0/packages/files
Building time -> /usr/local/lib/pony/0.23.0/packages/time
Building collections -> /usr/local/lib/pony/0.23.0/packages/collections
Building ponytest -> /usr/local/lib/pony/0.23.0/packages/ponytest
Building buffered -> /usr/local/lib/pony/0.23.0/packages/buffered
Building term -> /usr/local/lib/pony/0.23.0/packages/term
Building promises -> /usr/local/lib/pony/0.23.0/packages/promises
Building strings -> /usr/local/lib/pony/0.23.0/packages/strings
Building signals -> /usr/local/lib/pony/0.23.0/packages/signals
Building capsicum -> /usr/local/lib/pony/0.23.0/packages/capsicum
Building json -> /usr/local/lib/pony/0.23.0/packages/json
Building format -> /usr/local/lib/pony/0.23.0/packages/format
Building options -> /usr/local/lib/pony/0.23.0/packages/options
Generating
 Reachability
 Selector painting
 Data prototypes
 Data types
 Function prototypes
 Functions
 Descriptors
Optimising
Writing build/release/stable.o
Linking build/release/stable
STABLE_BIN=$(pwd)/build/release/stable build/release/test --only=integration
1 test started, 0 complete: integration.Usage() started
2 tests started, 0 complete: integration.Usage(help) started
3 tests started, 0 complete: integration.Usage(unknown arguments) started
4 tests started, 0 complete: integration.Version started
5 tests started, 0 complete: integration.Env(no bundle.json) started
6 tests started, 0 complete: integration.Env(empty bundle in same directory) started
7 tests started, 0 complete: integration.Env(bundle in same directory) started
8 tests started, 0 complete: integration.Env(calling a program) started
9 tests started, 0 complete: integration.Env(bundle.json in parent dir) started
10 tests started, 0 complete: integration.Env(invalid bundle.json in nested dir) started
10 tests started, 1 complete: integration.Version complete
10 tests started, 2 complete: integration.Usage(help) complete
10 tests started, 3 complete: integration.Usage(unknown arguments) complete
10 tests started, 4 complete: integration.Env(empty bundle in same directory) complete
10 tests started, 5 complete: integration.Env(bundle.json in parent dir) complete
10 tests started, 6 complete: integration.Usage() complete
10 tests started, 7 complete: integration.Env(invalid bundle.json in nested dir) complete
10 tests started, 8 complete: integration.Env(calling a program) complete
10 tests started, 9 complete: integration.Env(bundle in same directory) complete
10 tests started, 10 complete: integration.Env(no bundle.json) complete
---- Passed: integration.Usage()
---- Passed: integration.Usage(help)
---- Passed: integration.Usage(unknown arguments)
---- Passed: integration.Version
---- Passed: integration.Env(no bundle.json)
---- Passed: integration.Env(empty bundle in same directory)
---- Passed: integration.Env(bundle in same directory)
---- Passed: integration.Env(calling a program)
---- Passed: integration.Env(bundle.json in parent dir)
---- Passed: integration.Env(invalid bundle.json in nested dir)
----
---- 10 tests ran.
---- Passed: 10
Success!
$

Running this in a loop for a while; let's see if the failure comes up locally...

Update: ran this 25 times with no issue. (It takes a lot longer than a make integration loop; so the n is smaller here; didn't wait that long to try this forever.)

Next up: limiting the docker resources locally to the CircleCI specs -- 4G mem, 2 cpu -- and running docker run -v $(pwd):/src/main ponylang/ponyc:release make integration in a loop.

Update: no issues after having run that ^ 100 times.

Trying to restrict resources further, I've found that giving the job the minimum memory possible in docker, 4M, reproducibly gives timeouts:

(Note that this only works if you've run docker run -v $(pwd):/src/main ponylang/ponyc:release make integration before -- you cannot build the test binary with 4M mem)

$ docker run -m 4m -v $(pwd):/src/main ponylang/ponyc:release make integration
STABLE_BIN=$(pwd)/build/release/stable build/release/test --only=integration
1 test started, 0 complete: integration.Usage() started
2 tests started, 0 complete: integration.Usage(help) started
3 tests started, 0 complete: integration.Usage(unknown arguments) started
4 tests started, 0 complete: integration.Version started
5 tests started, 0 complete: integration.Env(no bundle.json) started
6 tests started, 0 complete: integration.Env(empty bundle in same directory) started
7 tests started, 0 complete: integration.Env(bundle in same directory) started
8 tests started, 0 complete: integration.Env(calling a program) started
9 tests started, 0 complete: integration.Env(bundle.json in parent dir) started
10 tests started, 0 complete: integration.Env(invalid bundle.json in nested dir) started
10 tests started, 1 complete: integration.Usage(help) complete
10 tests started, 2 complete: integration.Usage() complete
10 tests started, 3 complete: integration.Version complete
10 tests started, 4 complete: integration.Env(bundle in same directory) complete
10 tests started, 5 complete: integration.Env(empty bundle in same directory) complete
10 tests started, 6 complete: integration.Usage(unknown arguments) complete
10 tests started, 7 complete: integration.Env(no bundle.json) complete
10 tests started, 8 complete: integration.Env(calling a program) complete
10 tests started, 9 complete: integration.Env(invalid bundle.json in nested dir) complete
10 tests started, 10 complete: integration.Env(bundle.json in parent dir) complete
---- Passed: integration.Usage()
---- Passed: integration.Usage(help)
---- Passed: integration.Usage(unknown arguments)
**** FAILED: integration.Version
Test timed out without completing
stdout was: 0.1.3-353733f [release]

---- Passed: integration.Env(no bundle.json)
**** FAILED: integration.Env(empty bundle in same directory)
Test timed out without completing
**** FAILED: integration.Env(bundle in same directory)
Test timed out without completing
stdout was: SHLVL=0
OLDPWD=/src/main
CWD=/src/main/stable/test/integration/tmp/P7YjXX
STABLE_BIN=/src/main/build/release/stable
PWD=/src/main/stable/test/integration/tmp/P7YjXX
PONYPATH=../local/a:/src/main/stable/test/integration/tmp/P7YjXX/.deps/-local-git-b16798852821555717647:/src/main/stable/test/integration/tmp/P7YjXX/.deps/github/c:/src/main/stable/test/integration/tmp/P7YjXX/.deps/gitlab/d

---- Passed: integration.Env(calling a program)
---- Passed: integration.Env(bundle.json in parent dir)
**** FAILED: integration.Env(invalid bundle.json in nested dir)
Test timed out without completing
----
---- 10 tests ran.
---- Passed: 6
**** FAILED: 4 tests, listed below:
**** FAILED: integration.Version
**** FAILED: integration.Env(empty bundle in same directory)
**** FAILED: integration.Env(bundle in same directory)
**** FAILED: integration.Env(invalid bundle.json in nested dir)
Makefile:53: recipe for target 'integration' failed
make: *** [integration] Error 255
$

However, bumping the timeouts to 1s even makes these go away when running in 4M mem containers.

I would thus think that this could be a compromise. The hypothesis is that a timeout of 1s is enough; and the next commit, bumping timeouts further, shall make this PR pass on circleci.

[SeanTAllen]

@srenatus i would suggest running the tests using the --sequential flag. That will run them one at a time and should fix your timeout problem. Then they aren't competing for resources, you can also put them in an exclusion group together so no more than 1 member of an exclusion group runs at a time. --sequential (like we do for ponyc tests) is probably your best option.

[srenatus]

@SeanTAllen good idea! Thanks 🎉

Update: note those two commits below -- seems like a modest timeout bump (200ms) and running sequentially makes this OK.

In any case, it's something to keep an eye on. If this becomes too annoying (it shouldn't impede contributions), I'm happy to revisit it.

[srenatus]

Hiccups on travis, I've tried re-running. If it doesn't do the trick, I can deal with this later. (Tomorrow morning the latest.)

[srenatus]

Ok the travis issue has resolved itself -- added a note re: the ponyc issue, will merge this when green.