Web hook payloads failing with 500

[meker12]

https://github.com/rackerlabs/nexus-control/settings/hooks/7379405

@smashwilson

[meker12]

Not sure if this is related to the issue, but we changed the name of the docs-cloud-arbor repo to docs-cloud-paas.

After I changed the repo name, I updated the nexus-control content.repositories, deconst.horse.com config and routes files.

Then, I updated the contentID to match the new repo name, and checked in that PR. Then, I checked in that PR. The job ran successfully for that update, but the build log shows that it ran the job in the rackerlals-docs-arbor-master workspace.

Deconst content directory: /workspace/data/rackerlabs-docs-arbor-master/job-5762de8dc97e920100795294/dev-docs
Using inferred preparer: quay.io/deconst/preparer-sphinx
Using environment variable CONTENT_ID_BASE=[https://github.com/build-7e617ea32b/rackerlabs/docs-cloud-paas/] instead of _deconst.json setting [https://github.com/rackerlabs/docs-cloud-paas/].
Running Sphinx v1.4.1
making output directory...
Using environment variable CONTENT_ID_BASE=[https://github.com/build-7e617ea32b/rackerlabs/docs-cloud-paas/] instead of _deconst.json setting [https://github.com/rackerlabs/docs-cloud-paas/].
building [mo]: all of 0 po files

That's when I discovered that the nexus-control configuration updates for the repo name change didn't get delivered.

[smashwilson]

Well that's not good! Thanks for the heads-up, I'll have a look.

Personally, I think it has more to do with this:

Looks like we filled up a disk on the build server. I'll clean that out this morning and see if that improves things.

/cc @kallimachos

[smashwilson]

The space is being taken up by the Strider workspace volume - the directories where Strider stores its git clone and build output from each build. Strider only stores the most recent build for each repository and branch, but in the case of rackerlabs/rackspace-how-to, (a) there's a directory for every branch ever created since they're created in the rackerlabs/ fork, and (b) each directory there is 200MB+:

1.5G    rackerlabs-rackspace-how-to-master
232.8M  rackerlabs-rackspace-how-to-meker12-patch-2
232.9M  rackerlabs-rackspace-how-to-meker12-patch-3
233.0M  rackerlabs-rackspace-how-to-orchestration-edit-1
233.5M  rackerlabs-rackspace-how-to-rackconnect-edits-1
233.4M  rackerlabs-rackspace-how-to-rackconnect-edits-2
233.1M  rackerlabs-rackspace-how-to-rackconnect-edits-3
232.5M  rackerlabs-rackspace-how-to-revert-759-patch-2
232.5M  rackerlabs-rackspace-how-to-revert-761-patch-1

I can blow all of those away easily right now, that's all temporary data:

deconst-drc-build-blue-001 docker # df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda9       37G   12G   24G  34% /

I'll make a separate issue presently to find a more long-term solution that I can fit into next sprint (or this one while everyone else is out at Dockercon). It'll either be adding a phase to the Deconst content build plugin to delete the workspace content when a build is successful, or to add a fix upstream in Strider to delete workspace directories corresponding to deleted branches; probably the former.

[smashwilson]

We now have:

• A successful webhook delivery from GitHub;
• A green build in Strider for nexus-control;
• And a successful load of the latest state of nexus-control into Nexus:

Looks like we're in a better state now :tada:

I'll get that other issue filed. Judging by the amount of time it took for the disk to fill this time, I should have plenty of time to get a longer-term fix in place. If I'm wrong light the bat-signal again :grin:

[meker12]

👍

[kallimachos]

Thanks @smashwilson!