Spark init doesn't work... Something about the docker image was crook
The CI workflows which hit Analysis-runner were producing invalid docker image names
Proposed Changes
Changes the build base to python-bullseye, building on top of Hail's own image (itself having a pretty opaque build file) was rubbish
Changes the Hail Batch inits to use hl.context.init_spark directly, which avoids some possible weirdness with Literal python typing
Proposed changes to how the image VERSION is added using String replacement in CI workflows
A run using these changes was partially successful here (ultimately failed due to a missing slash, that's to be fixed in prod-pipes. One of the local Hail Query environments ran successfully).
@illusional specifically on the CI Workflow, for the image to use in analysis-runner, I originally had it so that the full cpg_aip:VERSION part of the docker image was under bump2version control. That worked fine, but the diff it created showed the whole line being changed (the full JSON payload being sent as a request).
That failed (e.g. here) because it uses .../images/talos:$VERSION literally. I expected a string replacement, any idea why it didn't work? This image name is inside double quotes, inside single quotes, so maybe it's not reachable?
I've tried replacing it with .../images/talos:${{ env.VERSION }}, would you expect this to fail in the same way?
Fixes
Proposed Changes
hl.context.init_sparkdirectly, which avoids some possible weirdness with Literal python typingVERSIONis added using String replacement in CI workflowsA run using these changes was partially successful here (ultimately failed due to a missing slash, that's to be fixed in prod-pipes. One of the local Hail Query environments ran successfully).
@illusional specifically on the CI Workflow, for the image to use in analysis-runner, I originally had it so that the full
cpg_aip:VERSIONpart of the docker image was under bump2version control. That worked fine, but the diff it created showed the whole line being changed (the full JSON payload being sent as a request).I changed that to a VERSION env variable, and:
"image": "australia-southeast1-docker.pkg.dev/cpg-common/images/talos:$VERSION"That failed (e.g. here) because it uses
.../images/talos:$VERSIONliterally. I expected a string replacement, any idea why it didn't work? This image name is inside double quotes, inside single quotes, so maybe it's not reachable?I've tried replacing it with
.../images/talos:${{ env.VERSION }}, would you expect this to fail in the same way?