Add a real integration test on upload [swarming]

[GoogleCodeExporter]

Proposal:
update.py have a new flag --test (and optionally --no-test depending on the 
desired default):

./tools/update.py -A <appid> --test

would do a complete smoke test after deployment:
- Fire up a bot and connect to the host with a random unique dimension
- Trigger a dummy task with this random unique dimension
- Ensure the task triggered properly.

This would take <15s to run and would catch all the "obviously broken" cases 
like 6cbbc9df88ff75c6ee0e5794f5ddc8f56e501485 which pass on a local smoke test 
but would fail a smoke test when running on prod.

This issue is about Swarming but the same idea can be done for isolate server 
so it'd be good to implement it in a generic way inside update.py, that 
triggers "<APP_DIR>/smoke_test.py --host <host>".

Original issue reported on code.google.com by maruel@chromium.org on 17 Oct 2014 at 1:15

• Merged into: #187

[GoogleCodeExporter]

Ok, here's an alternative proposal, in the form of a dream.

I dream of times when we deploy GAE apps by changing some configuration 
property in some file. In my dream we have a buildbot that watches for changes 
to current_versions.yaml file, runs pre-deploy sanity tests, deploys a new 
version, runs post deploy integration tests, and then switches the default 
version to a newly deployed one. 

In my dream, each stage of a process is a separate distinct step. And thus GAE 
update and integration tests are performed by separate scripts. 

tl;dr - I think it's better to have the test script be independent of update.py 
and perform integration test manually for now. I really want to start working 
on appcfg.py wrapper & waterfall that runs it, but have no time to do it...

Original comment by vadimsh@chromium.org on 17 Oct 2014 at 2:25

[GoogleCodeExporter]

It's not a dream, it's an auto-rolling bot. And once the integration smoke test 
is implemented, it wouldn't be too hard to implement the auto-rolling bot.

Original comment by maruel@chromium.org on 17 Oct 2014 at 2:29

[GoogleCodeExporter]

Superseded by issues 186 and 187.

Original comment by maruel@chromium.org on 9 Jan 2015 at 8:00

• Changed state: Duplicate