add docker authentication

[dnadeau-lanl]

Add Docker authentication to images being pulled.

• [x] Pull request has descriptive title
• [x] Pull request gives overview of changes
• [N/A] New code has inline comments where necessary
• [N/A] Any new modules, functions or classes have docstrings consistent with dbprocessing style
• [N/A] Major new functionality has appropriate Sphinx documentation
• [N/A] Added an entry to CHANGELOG if fixing a major bug or providing a major new feature
• [N/A] New features and bug fixes should have unit tests
• [x] Relevant issues are linked in the description (e.g. See issue # or Closes #)

Closes #32

[jtniehof]

The check on this is weird...Github is showing the webhook as delivered to CircleCI, and the response is the same as for #50, but there's no indication of it running on the CircleCI side. I'll keep digging...the difference might be in who submitted, but #41 was fine.

There is a build on CircleCI against master, which is weird.

Ah, but I did have the "require up-to-date before merging" set; I wasn't going to turn that off until I did other work for #40, but I'll switch that now and see if it makes a difference.

Two things that would help

1. Can you add the "Closes #32" to the PR description? That's how the release notes process pulls the information; manually linking doesn't show up in there.
2. Can you add a little information on what has to be set up on the CircleCI side to inject the secrets into the environment? This can go in the CircleCI section of https://github.com/spacepy/dbprocessing/blob/master/docs/developer/github.rst and that gives a little bit of an example of how I had other stuff documented.

[dnadeau-lanl]

Will do today. I can try to retrigger circleCI with a "curl" command.

[jtniehof]

I tried the retrigger and nothing. The problem is we need to enable the "Pass secrets to builds from forked pull requests" setting in CircleCI. I just turned that on and will retrigger; that needs to go in the setup documentation, too.

[jtniehof]

Incidentally the thing with that setting is that somebody can submit a PR that changes the CircleCI config to just spit your Docker credentials to their email, so don't do anything terribly sensitive with those credentials...

[dnadeau-lanl]

I can see in the "response" X-Frame-Options: DENY, hopefully passing secrets will work.

[dnadeau-lanl]

I don't think I have access 😏 https://github.com/spacepy/dbprocessing/settings/access

[jtniehof]

You're a member of the dbprocessing group, so you have access.

X-Frame-Options: DENY is set on all CircleCI replies; it just means "don't stick this in a frame." It's being set on builds that worked, too.

I can't find any logging on the CircleCI side of what hooks it's received and what it did with them.

[jtniehof]

Ah, at least I figured out the "master" builds. When a PR is merged, that's sending a "push to master" event to CircleCI, which is then triggering a build on master. I'll make a note to disable that as part of #40, since we'll be doing the merge before running the PR tests and also doing cron-based builds of master...no sense in running the tests twice.

Still can't figure out why redelivery of the webhook isn't triggering the pipeline.

[jtniehof]

Did you just disable the integration checks?

[dnadeau-lanl]

Yes I saw a comment saying to try this. Did not work! I put it back.

[dnadeau-lanl]

I just reran it within circleci

[dnadeau-lanl]

It ran 19 hours ago, but github did not see the results...

[dnadeau-lanl]

Ok so main is problematic and not being reported back.

[jtniehof]

It ran on the branch, not on the pull request.

[jtniehof]

Needs to show up here or it won't work: https://app.circleci.com/pipelines/github/spacepy/dbprocessing

[jtniehof]

That actually might be the problem. If you have your fork set up to send stuff to your CircleCI account, it's probably rejecting the webhooks from the dbprocessing organization as duplicates.

[jtniehof]

Check https://github.com/dnadeau-lanl/dbprocessing/settings/hooks --probably just best to remove any hooks you have there for CircleCI. That would be the difference between PRs you submit and those that other people submit.

[dnadeau-lanl]

I can try that for now, but I do want to trigger circleci with my branch when I push, not only for PR.

[jtniehof]

You can always open a draft PR if you want to see how things go. I just don't think it's going to work to have two sets of webhooks against the same commit.

[dnadeau-lanl]

I tried to create another PR without my webhook and it still did not work.

[jtniehof]

I'm seeing a build on master; was that a manual trigger?

[dnadeau-lanl]

Yes.

I don't know what is going on. It used to work....

[dnadeau-lanl]

Can you submit a fake PR too. to see if you can trigger it from your github repo.

[jtniehof]

Looks like the build-on-branch-and-on-PR won't work: "Currently a push to a branch may be ran before a pull request is created and a new job will not be triggered when a pull request is opened." https://ideas.circleci.com/cloud-feature-requests/p/trigger-new-build-when-a-pull-request-is-opened

Doesn't explain why we're not having stuff work now that you've disabled your webhook.

[jtniehof]

Yes, I'll do some tests on my account, have another meeting now though.

[dnadeau-lanl]

I found out what it was. I was following my projects in circle and the forked projects. You have to follow only the fork projects.

https://support.circleci.com/hc/en-us/articles/360008097173-Why-aren-t-pull-requests-triggering-jobs-on-my-organization-

https://app.circleci.com/projects/project-dashboard/github/spacepy

[jtniehof]

The one you just pushed does, at least, link to the spacepy/dbprocessing org page on CircleCI (when I click "Details"). Looks like the postgres is still running, so hopefully life is good once it completes.

[jtniehof]

Ah, I had seen that before, and of course forgot. So we should fold that into the docs, document the setting for passing the secrets through to CircleCI, and maybe do a rebase to clean up the extra commits that were just triggering, and we're good. Thanks for pushing through that.

[jtniehof]

Hmmm, that did pick up a bonus commit that's just whitespace changes....

[dnadeau-lanl]

Just realized I missed a meeting .. sigh

[jtniehof]

I can run the documentation on this and you can check it over to see if it makes sense? (Edit, if that takes stuff off your plate, not elbowing in.)

[dnadeau-lanl]

Ok thanks!

[dnadeau-lanl]

I just git reset --hard and git force push on my branch. Seems to have worked.

[jtniehof]

I can also do some history cleanup when I write up docs (this afternoon).

[dnadeau-lanl]

The docs sounds good. Something I wonder, if I push a new branch will it trigger circleci in your spacepy repo, if not I would like to find a way to make this work.

[jtniehof]

Based on https://ideas.circleci.com/cloud-feature-requests/p/trigger-new-build-when-a-pull-request-is-opened , it sounds like if you push to a branch that runs a CircleCI job and then open a PR, the checks won't work on the PR.

Also, we know from here that you can't follow a fork and have the checks work in the organization's repo, and I don't think there's a way to get the organization CircleCI to run against pushes in a fork (it would be all forks, then, and that would be a mess.)

So I don't think there's a way to get the CI to run against your fork and have checks work on the pull request.

[jtniehof]

I'm going to go ahead and merge this and opened #53 to track the user CI issue.