CircleCI onpublish fails to make GitHub API call

[dleehr]

I released version 2.1.1 from master a few minutes ago, and the CircleCI build fails as below

See https://circleci.com/gh/Duke-GCB/DukeDSClient/34

#!/bin/bash -eo pipefail
. venv/bin/activate
./deploy/onpublish.sh
Requirement already satisfied: requests in ./venv/lib/python3.6/site-packages/requests-2.21.0-py3.6.egg (2.21.0)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.6/site-packages/certifi-2018.11.29-py3.6.egg (from requests) (2018.11.29)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./venv/lib/python3.6/site-packages/chardet-3.0.4-py3.6.egg (from requests) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in ./venv/lib/python3.6/site-packages/idna-2.8-py3.6.egg (from requests) (2.8)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in ./venv/lib/python3.6/site-packages/urllib3-1.24.1-py3.6.egg (from requests) (1.24.1)
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Creating spec file ddsclient-2.1.1-gcb01.spec.
Traceback (most recent call last):
  File "deploy/onpublish.py", line 33, in <module>
    resp.raise_for_status()
  File "/home/circleci/project/venv/lib/python3.6/site-packages/requests-2.21.0-py3.6.egg/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.github.com/repos/Duke-GCB/helmod/contents/rpmbuild/SPECS/ddsclient-2.1.1-gcb01.spec
Exited with code 1

[dleehr]

The 404 error is raised after the requests.put on line 32.

This may be an authentication error. I know in many cases GitHub returns 404 when you're not authenticated/authorized correctly.

[dleehr]

Further data on the 404 - unauthenticated curl GET and PUT for an existing file:

$ curl -X GET https://api.github.com/repos/Duke-GCB/helmod/contents/rpmbuild/SPECS/ddsclient-2.1.0-gcb01.spec | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17150  100 {   0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
17  "name": "ddsclient-2.1.0-gcb01.spec",
1  "path": "rpmbuild/SPECS/ddsclient-2.1.0-gcb01.spec",
5  "sha": "7080a0f692a4cae65d2b94a10c6a00946d4e7c9b",
0  "size": 11693,
   "url": "https://api.github.com/repos/Duke-GCB/helmod/contents/rpmbuild/SPECS/ddsclient-2.1.0-gcb01.spec?ref=master",
   "html_url": "https://github.com/Duke-GCB/helmod/blob/master/rpmbuild/SPECS/ddsclient-2.1.0-gcb01.spec",
   "git_url": "https://api.github.com/repos/Duke-GCB/helmod/git/blobs/7080a0f692a4cae65d2b94a10c6a00946d4e7c9b",
   "download_url": "https://raw.githubusercontent.com/Duke-GCB/helmod/master/rpmbuild/SPECS/ddsclient-2.1.0-gcb01.spec",
0  "type": "file",
     0  93715      0 --:--:-- --:--:-- --:--:-- 93715

$ curl -X PUT https://api.github.com/repos/Duke-GCB/helmod/contents/rpmbuild/SPECS/ddsclient-2.1.0-gcb01.spec
{
  "message": "Not Found",
  "documentation_url": "https://developer.github.com/v3/repos/contents/#update-a-file"
}
$ 

[dleehr]

Tried to debug a little inside CircleCI.

The token appears to be valid and the code structured to authorize should be correct. Even with a valid token, this returns a 404.

Since this is creating a new resource, I'd suggest it would be a POST, but GitHub's documentation suggests it should be a PUT

[dleehr]

I tried to reproduce locally with a new repo and the following script. After getting the URL right, no problem creating files.

#!/usr/bin/env python

import requests
import base64
url = 'https://api.github.com/repos/dleehr/putit/contents/api-file3/afile.txt'

commit_token = 'REMOVED'

data = {
  "message": 'hello',
  "committer": {"name": 'Dan Leehr', "email": 'dan.leehr@duke.edu'},
  "content": 'aGVsbG8K',
  "branch": 'master'
}

headers = { "Authorization": "token {}".format(commit_token)}
resp = requests.put(url, headers=headers, json=data)
print(resp.json())
resp.raise_for_status()

[johnbradley]

Looks like this issue was two fold.

1. The credentials didn't have the appropriate permissions within github. Github results in 404 in this case instead of a Not Authorized for some reason.
2. Due to some changes with python3 we need to tweak the unicode encoding logic.

[johnbradley]

Github docs about unexpected (to me at least) 404: https://developer.github.com/v3/troubleshooting/#why-am-i-getting-a-404-error-on-a-repository-that-exists