search

projectglow / glow

An open-source toolkit for large-scale genomic analysis
https://projectglow.io
Apache License 2.0
262 stars 107 forks source link

Test recalibration + DBR 9.1 Docker images #513

Closed a0x8o closed 1 year ago

a0x8o commented 2 years ago

What changes are proposed in this pull request?

  1. Temporarily stop Hail tests (until these are upgraded for where breaking Hail changes are presently failing)
  2. Fix broken links in Getting Started document source for Databricks Repos Sync/Git Clone
  3. Provide Docker images for DBR 9.1

How is this patch tested?

I am aiming to address three things here:

  1. Hail-IS made some changes that are breaking the Hail tests in projectglow. I'm switching off these tests until someone can fix the Hail tests.
  2. Getting Started had another broken link which I've fixed. I also have found that these tests should have a higher timeout value as frequently the tests fail due to short load timeouts.
  3. I believe we need to provide the DBR9.1 Docker images in addition to DBR10.4 images in master. I've reintroduced these to master.
a0x8o commented 2 years ago

@williambrandler @karenfeng I am trying to address three things here:

  1. Hail-IS made some changes that are breaking the Hail tests in Projectglow. I'm switching off these tests until someone (I or another engineer) can fix the tests.
  2. Getting Started had another broken link which I've fixed. I also have found that these tests should have a higher timeout value as frequently the tests fail due to short load timeouts.
  3. I believe we need to provide the DBR9.1 Docker images in addition to DBR10.4 images in master.
williambrandler commented 2 years ago

ah ok, thanks @a0x8o

I removed 9.1 when bumping to 10.4, but it seems reasonable to maintain both versions until 9.1 is obsolete

Fine to remove hail tests to get the ci passing again, can add back in once https://github.com/hail-is/hail/issues/11707 is resolved. The Hail team are waiting for EMR and dataproc to update to spark 3.2.0, after which I expect the tests to pass (once the Hail version is updated)

I see EMR is on 3.2.0 with emr-6.6.0, however dataproc is still on spark 3.1.3 (dataproc versioning)

a0x8o commented 2 years ago

@williambrandler any idea why we are seeing the following: https://app.circleci.com/pipelines/github/projectglow/glow/3131/workflows/a1e5ccf6-ada6-480b-91d8-9983139c4cd9/jobs/9682

williambrandler commented 2 years ago

@williambrandler any idea why we are seeing the following: https://app.circleci.com/pipelines/github/projectglow/glow/3131/workflows/a1e5ccf6-ada6-480b-91d8-9983139c4cd9/jobs/9682

either a transient issue with circleci or permissions related @a0x8o

Try this link to join the project? https://app.circleci.com/pipelines/github/projectglow/glow?invite=true