Migrate infrastructure repositories to opensafely-core organisation

[bloodearnest]

We want to move the open source infra projects (e.g. job-runner, job-server, output-publisher, opensafely-cli, etc) into a separate organisation, for a few reasons.

Firstly, it provides a clear security barrier between research studies and operational code. Currently, this is managed through Github's teams via manage-github.py script, but it's not very clean, and increasingly a maintain burden. Moving sensitive repos to an external org means we can differentiate researchers and developers more clearly.

Secondly, it improves our researcher onboarding capabilities. We can more easily add members to opensafely org, and split them into teams with different repo access.

Thirdly, it makes the opensafely organisation all about the users of opensafely, containing just studies and supporting documentation, and the opensafely-core organisation all about the developers of the platform.

Fourthly, it makes things simpler. It reduces the the number of repos in the opensafely org, researchers are less likely to be confused by infra repos, some things are simpler to implement - e.g. updating all repos, or job-server repo selection logic, as it doesn't have to filter out the infra repos.

[bloodearnest]

repositories to be moved

• [x] opensafely/cohort-extractor
• [x] opensafely/ctds-binary
• [x] opensafely/emis-acceptance-test
• [x] opensafely/emis-tools
• [x] opensafely/git-server-proxy
• [x] opensafely/job-integration-tests
• [x] opensafely/job-runner
• [x] opensafely/job-server
• [x] opensafely/jupyter-docker
• [x] opensafely/opencodelists
• [x] opensafely/r-docker
• [x] opensafely/base-docker
• [x] opensafely/python-docker
• [x] opensafely/docker-proxy
• [x] opensafely/sysadmin
• [x] opensafely/opensafely-cli
• [x] opensafely/output-publisher
• [x] opensafely/job-runner-dependencies
• [x] opensafely/tools-docker

private

• [x] opensafely/stata-docker - clean history, make public, then transfer

missed

• [ ] database-notebooks
• [ ] ...?

other tasks

• [x] update management script
  • auto-delete branches
  • enforce branch protection
• [x] create developers team in opensafely-core
• [ ] update website(s) (links do currently forward, so not urgent, but should be done)
  • [x] https://github.com/opensafely/documentation/
  • [ ] thedatalab.org ?
  • [ ] ...others...?
• [x] update job-runner/output-publisher checkouts on TPP servers
• [x] migrate secrets
  • [x] chase @sebbacon to transfer projects
  • [x] create opensafely account
• [x] pump release processes
  • [x] job server
  • [x] job runner
  • [x] docker
  • [ ] ...

[bloodearnest]

Migration questions

• do we want to move s/master/main at the same time?
• should we still publish the github images under the opensafely org name, because a) no migration needed b) its user facing anyway?
• who should be a member of the new org? Obviously the opensafely/developers team, but also maybe other DataLab folks? Helen/Alex/Will/Brian/Anna R ?
• is admin level repo access for every member of opensafely-core ok?

[bloodearnest]

Migration TODOs (WIP)

• update job-runner and output-publisher checkouts in TPP and EMIS servers
• update developers local repos
• update links in https://opensafely.org/code/ and probably other documentation

[madwort]

• do we want to move s/master/main at the same time?

looking like a hard no

[bloodearnest]

• do we want to move s/master/main at the same time?

looking like a hard no

Yeah, lets make the delta as small as possible

[madwort]

update developers local repos

sed -i -e 's/github.com:opensafely/github.com:opensafely-core/' .git/config or sed -i -e 's?https://github.com/opensafely?https://github.com/opensafely-core?' .git/config

[madwort]

@bloodearnest just came across this old ticket - all done I think