opensafely-core / job-server

A server for mediating jobs that can be run in an OpenSAFELY secure environment. q.v. job-runner
https://jobs.opensafely.org
Other
5 stars 10 forks source link

Enforce 1-1 relationship for repo -> workspace going forward. #1966

Closed bloodearnest closed 2 years ago

bloodearnest commented 2 years ago

We currently support creating a workspace that points to a branch on a repo.

Currently, 60 repos have more than 1 workspace linked to them:

jobserver=> select repo, count(id) from jobserver_workspace group by repo having count(id) > 1 order by count(id) desc;
                                        repo                                        | count 
------------------------------------------------------------------------------------+-------
 https://github.com/opensafely/amr-uom-brit                                         |    24
 https://github.com/opensafely/covid-ve-change-over-time                            |    11
 https://github.com/opensafely/Factors-associated-with-COVID-19-vaccination         |    10
 https://github.com/opensafely/covid-vaccine-effectiveness-research                 |     9
 https://github.com/opensafely/antibody-and-antiviral-deployment                    |     9
 https://github.com/opensafely/OS_OC_v001-research                                  |     8
 https://github.com/opensafely/COVID-19-vaccine-breakthrough                        |     8
 https://github.com/opensafely/highcostdrugs-research                               |     8
 https://github.com/opensafely/carehomes-research                                   |     7
 https://github.com/opensafely/os-demo-research                                     |     7
 https://github.com/opensafely/SRO-Measures                                         |     6
 https://github.com/opensafely/anticoagulants-research                              |     5
 https://github.com/opensafely/long-covid                                           |     5
 https://github.com/opensafely/covid-vaccine-preliminary-study                      |     5
 https://github.com/opensafely/antipsychotics-prescribing-during-COVID-19           |     5
 https://github.com/opensafely/Ethnicity-2nd-wave                                   |     5
 https://github.com/opensafely/ethnicity-short-data-report                          |     4
 https://github.com/opensafely/aftershocks-notebook                                 |     4
 https://github.com/graphnet-opensafely/opensafely-SRO-Measures                     |     4
 https://github.com/opensafely/population-outcomes-burden-research                  |     4
 https://github.com/opensafely/covid-vaccine-not-received                           |     3
 https://github.com/opensafely/SRO-smr                                              |     3
 https://github.com/opensafely/T1DM_covid_research                                  |     3
 https://github.com/opensafely/vaccine-effectiveness-hospital-admissions-validation |     3
 https://github.com/opensafely/absolute-risks-covid-research                        |     3
 https://github.com/opensafely/Uptake-of-NHS-home-interventions-during-COVID-19     |     2
 https://github.com/opensafely/ethnicity-covid-research                             |     2
 https://github.com/opensafely/hospital-disruption-research                         |     2
 https://github.com/opensafely/restoration-observatory-emis-research                |     2
 https://github.com/opensafely/os-sch-children-2021                                 |     2
 https://github.com/opensafely/SRO-pulse-oximetry                                   |     2
 https://github.com/opensafely/covid-unvaccinated-aimpaper                          |     2
 https://github.com/opensafely/bmi-short-data-report                                |     2
 https://github.com/opensafely/sotrovimab-and-molnupiravir                          |     2
 https://github.com/opensafely/antidepressant-prescribing-lda                       |     2
 https://github.com/opensafely/principle-inhaler-uptake                             |     2
 https://github.com/opensafely/post-covid-outcomes-research                         |     2
 https://github.com/opensafely/carehomes-short-data-report                          |     2
 https://github.com/opensafely/hypertension-sro                                     |     2
 https://github.com/opensafely/pincer-measures                                      |     2
 https://github.com/opensafely/immunosuppressant-meds-research                      |     2
 https://github.com/opensafely/renal-short-data-report                              |     2
 https://github.com/opensafely/non-specific-immunity-research                       |     2
 https://github.com/opensafely/BMI-and-Metabolic-Markers                            |     2
 https://github.com/opensafely/mechanical-valve-anticoag                            |     2
 https://github.com/opensafely/booster-effectiveness                                |     2
 https://github.com/opensafely/HEST_OS_study                                        |     2
 https://github.com/opensafely/vaccine-eligibility                                  |     2
 https://github.com/opensafely/sro-gastro                                           |     2
 https://github.com/opensafely/post-covid-kidney-outcomes                           |     2
 https://github.com/opensafely/covid-collateral-research                            |     2
 https://github.com/opensafely/primis-covid19-vaccine-uptake                        |     2
 https://github.com/opensafely/research-template                                    |     2
 https://github.com/opensafely/NICE-audit-lipids                                    |     2
 https://github.com/opensafely/HbA1c-levels                                         |     2
 https://github.com/opensafely/covid_mortality_over_time                            |     2
 https://github.com/opensafely/emis-qa                                              |     2
 https://github.com/opensafely/post-admission-admissions-research                   |     2
 https://github.com/opensafely/SGTF-617-research                                    |     2
 https://github.com/opensafely/surgery-research                                     |     2
(60 rows)

The original goal of this design was to support a form of code sharing between related projects, especially since we don't support forks for security reasons.

Anecdotally, there are two forms that branch-as-workspace usage seems to take:

1) starting from a known point to pursue a new line of enquiry.

2) some projects do want to share common code.

There are also some explicit drawbacks to allowing branches for workspaces.

a) For private repos, all branches will be public when the repo is made public. To comply with OS policy, we need to make private repos public 12 months after the first job on any branch/workspace ran, but a workspace based on a recent branch of that repo might not be ready to be made public yet.

b) It adds platform complexity. It makes it harder to understand the system, and makes browsing code in the right branch on github a bit more complex and easy to get wrong. If we only allowed a single repo with a main branch, then a bunch of things would we easier to implement and understand.

Suggestions

1) We should probably not allow new workspaces to use branches of private repos - they should be new repos. This will stop future problems when it comes to making those repos public.

2) We should consider not supporting branch based workflows going forward, and requiring new repos.

bloodearnest commented 2 years ago

TODO move to gdoc

bloodearnest commented 2 years ago

https://docs.google.com/document/d/1vKuekyIiQ4eJHzks6Hxn0nCihfc6ia30XM-jAmYCMUA/edit#