Recursive merge bug fixes and improvements

[sbesson]

Tested in the context of the ansible-roles repository (see https://trello.com/c/ruRyhHyP/22-ansible-roles-testing), this PR contains a series of improvements to increase the scalability of scc merge for repositories with submodules.

• 2de5192 fixes the regexp pattern to allow --update-gitmodules to update URLs with or without trailing .git
• 625c024 inverts the pushing order, pushing submodule branches before the upstream branches. In the case of repositories with large number of submodules where branch pushing trigger builds like Travis, this prevents race conditions as the top-level repository branch could point to not-existing submodule commits
• d610888 uses the GitHub create_fork() API before pushing branches. If the fork exists, this should be a no-op else the fork will be created.

With the systematic usage of integration branches separated by _ rather than /, the last commits means there should be no need for any manual action when new organizational repositories are created as scc merge --push --update-gitmodules should be self-sufficient.

Testing this PR:

• Continuous Integration: check Travis and Jenkins

• clone https://github.com/ome/ansible-roles (ideally without forking it) and run the merge command

  git clone https://github.com/ome/ansible-roles
  cd ansible-roles && git submodule update --init
  virtualenv /tmp/venv
  /tmp/venv/bin/pip install -r requirements.txt # yaclifw
  /tmp/venv/bin/python scc/main.py merge master --push master_merge_trigger --update-gitmodules

  The command should recursively fork the top-level project as well as all unforked submodule repositories, run the recursive merge command and push the integration branches. Check that the submodule branches are pushed before the top-level branch

  Re-running the merge command after the first execution should have the same effect but the repository forking should be skipped

  git reset --hard origin/master && git submodule update
  /tmp/venv/bin/python scc/main.py merge master --push master_merge_trigger  --update-gitmodules

Fixes #230

[joshmoore]

Could https://ci.openmicroscopy.org/view/Failing/job/SCC-merge/976/console be related?

self = <github.Requester.Requester instance at 0x17c3710>, status = 500
responseHeaders = {'access-control-allow-origin': '*', 'access-control-expose-headers': 'ETag, Link, Retry-After, X-GitHub-OTP, X-RateLi...Accepted-OAuth-Scopes, X-Poll-Interval', 'content-length': '109', 'content-security-policy': "default-src 'none'", ...}
output = {'documentation_url': 'https://developer.github.com/v3/pulls/#create-a-pull-request', 'message': 'Server Error'}

    def __check(self, status, responseHeaders, output):
        output = self.__structuredFromJson(output)
        if status >= 400:
>           raise self.__createException(status, responseHeaders, output)
E           GithubException: 500 {u'documentation_url': u'https://developer.github.com/v3/pulls/#create-a-pull-request', u'message': u'Server Error'}

(same in the last 2 runs) Did integration tests pass for you locally?

[joshmoore]

Sorry, looks like that was unrelated. See https://github.com/openmicroscopy/snoopys-sandbox/pull/32182 instead. https://ci.openmicroscopy.org/view/Failing/job/SCC-merge/978/ is now green. Re-including and re-executing.

[joshmoore]

New run failed again (This time no PRs are left open).

[sbesson]

Unexcluding as #238 might help stabilizing the integration tests. I will see how tests go over the week-end and relist for review on Monday

[sbesson]

See https://ci.openmicroscopy.org/job/SCC-merge/989/ and https://github.com/openmicroscopy/snoopycrimecop/pull/239#issuecomment-420404464

[joshmoore]

LGTM.