Concourse died - what's the best procedure to bring it back up?

[aterreno]

Hi, more a question than an issue, perhaps some docs on the subject would be great.

our monitoring says that our concourse went down

Pingdom DOWN alert: Concourse (xxx) is down since 16/12/2017 00:51:15. Reason: Network is unreachable

I didn't bother investigating and I've just ran a concourse-up deploy

Is that a 'decent' approach?

Is it possible that a new release of concourse-up broke the web worker?

Thanks.

[aterreno]

BTW, that didn't work 2017-12-16 00:36:47 +0000 time of the new version of concourse-up - Our Concourse is down since 16/12/2017 00:51:15.

Seems quite correlated...

[DanielJonesEB]

Thanks for the report @aterreno - one of our folks is looking into it. There's a problem with the upgrade path whereby newly-required values are not present in the config file in S3, and aren't generated either. Our testing wasn't good enough to catch this, so we're going to look into that.

We're looking into fixes now.

[aterreno]

@DanielJonesEB cool man, a couple of things (something I didn't bother figuring out when everything was working well)

• How do you ssh to the boxes to troubleshoot? I know about the config.json which is stored in s3 and containing the private/public key, but what's the user and is ssh enabled?

• Is there a mechanism for downgrading? I can't see it, so probably it's a nice-to-have new feature

I hope these questions will make concourse-up even better, I am happy to write some docs if you give me a couple of hints on how it works.

Cheers

[takeyourhatoff]

@aterreno Can you try redeploying with concourse-up v0.7.3? We have fixed a problem whereby new fields added to the ATC config were not properly initialised on existing deployments and have improved our testing to catch this in the future.

[aterreno]

Hi thanks, I ended up re-installing concourse from scratch as we couldn't wait for the fix, updated now and all good.

I'll close the issue.