Algo-XXX automatic environment testing

[zeryx]

This PR adds automatic environment testing to the packageset validator. It also adds an optional algorithmia API key environment variable that can be passed into the runtime container for algorithmia client API support.

I need feedback on: Testing system (files, format, etc) Structure and format (everything makes sense? sanity check)

Once I have that, I'll move forward on adding test support to all algorithm language templates & dependency templates.

TO TEST: install new dependency pycurl (and any system dependencies required) Run the following: ./tools/environment_validator.py -g python3 -s python37 -t language -n python3 -a 1

This should return a success as it validates two test cases

[besirkurtulmus]

@zeryx can you provide steps to test this?

[zeryx]

@zeryx can you provide steps to test this?

good catch, updated the description

[zeryx]

@besirkurtulmus I added example tests for all basic algorithm language templates, as well as providing support for exception based tests (algorithm exception handling, etc). I think that will conclude the scope for this. Ready for review.

[besirkurtulmus]

@zeryx I got this error when I ran the TEST command:

➜  langpacks git:(ALGO-XXX-automatic-testing) ./tools/environment_validator.py -g python3 -s python37 -t language -n python3 -a 1
completed template construction, file available at /tmp/validator_cache/validator-runtime-0e392fca-b1ac-40d9-bd9b-cc83cb3a8799.Dockerfile
building runtime image
completed template construction, file available at /tmp/validator_cache/validator-buildtime-6dd7e31d-b7bb-4772-b3cb-3f72b63a39eb.Dockerfile
building buildtime image
building final image
Traceback (most recent call last):
  File "./tools/environment_validator.py", line 355, in <module>
    automatic_testing=args.automatic_testing,
  File "./tools/environment_validator.py", line 287, in main
    raise e
  File "./tools/environment_validator.py", line 272, in main
    run_tests(client, container, test_inputs, test_outputs)
  File "./tools/environment_validator.py", line 199, in run_tests
    expected = json.loads(expected)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 19 (char 18)

[zeryx]

Ready for re-testing @besirkurtulmus when you're ready.

[besirkurtulmus]

@zeryx looks like I got two errors, but the tool is saying that all tests are successful:

➜  langpacks git:(ALGO-XXX-automatic-testing) ./tools/environment_validator.py -g python3 -s python37 -t language -n python3 -a 1
completed template construction, file available at /tmp/validator_cache/validator-runtime-cb8625d4-a872-40dd-846b-32e797b10cef.Dockerfile
building runtime image
completed template construction, file available at /tmp/validator_cache/validator-buildtime-b434da08-1ca1-4052-aa66-582f8fee019e.Dockerfile
building buildtime image
building final image
{'result': 'hello algorithmia'}
{'result': 'hello algorithmia', 'metadata': {'duration': 0.0014321, 'content_type': 'text'}}
pass
{'error': {'type': 'SystemError', 'message': 'Failed to build algorithm input from request: JSON error - expected value at line 1 column 1'}}
{'error': {'type': 'SystemError', 'message': 'Failed to build algorithm input from request: JSON error - expected value at line 1 column 1'}}
pass
All tests successful

[zeryx]

@zeryx looks like I got two errors, but the tool is saying that all tests are successful:

➜  langpacks git:(ALGO-XXX-automatic-testing) ./tools/environment_validator.py -g python3 -s python37 -t language -n python3 -a 1
completed template construction, file available at /tmp/validator_cache/validator-runtime-cb8625d4-a872-40dd-846b-32e797b10cef.Dockerfile
building runtime image
completed template construction, file available at /tmp/validator_cache/validator-buildtime-b434da08-1ca1-4052-aa66-582f8fee019e.Dockerfile
building buildtime image
building final image
{'result': 'hello algorithmia'}
{'result': 'hello algorithmia', 'metadata': {'duration': 0.0014321, 'content_type': 'text'}}
pass
{'error': {'type': 'SystemError', 'message': 'Failed to build algorithm input from request: JSON error - expected value at line 1 column 1'}}
{'error': {'type': 'SystemError', 'message': 'Failed to build algorithm input from request: JSON error - expected value at line 1 column 1'}}
pass
All tests successful

That's how it's supposed to look actually! at least at the moment, I get it to print both the real result and the expected. I'll update the print statements to be a bit more explicit.