Streaming from requester-pays buckets for CrosscheckFingerprints doesn't work in Terra

[rickymagner]

Bug Report

Affected tool(s)

CrosscheckFingerprints

Affected version(s)

3.1.0, and 2.27.5 (inside GATK 4.4.0.0 and 4.3.0.0)

Description

When running CrosscheckFingerprints in Terra and trying to stream an input using a localization_optional file, if the file is located in a requester-pays GCP bucket, the task will fail. Here is a truncated stack trace:

        at org.broadinstitute.hellbender.cmdline.PicardCommandLineProgramExecutor.instanceMain(PicardCommandLineProgramExecutor.java:37)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
        at org.broadinstitute.hellbender.Main.main(Main.java:289)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
GET https://storage.googleapis.com/storage/v1/b/broad-dsde-methods-hydro-gen-truth-data-public/o/NIST%2FGIAB%2Fv4%2FHG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz?fields=bucket,name,id&projection=full
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Bucket is a requester pays bucket but no user project provided.",
    "reason" : "required"
  } ],
  "message" : "Bucket is a requester pays bucket but no user project provided."
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:455)
        ... 17 more
Using GATK jar /gatk/gatk-package-4.4.0.0-local.jar

After I tried using Picard 3.1.0 locally with the command:

java -jar ~/Downloads/picard.jar CrosscheckFingerprints I=gs://dsde-palantir/SequencerEvaluation2022/IlluminaNovaSeq/HG001/coverage_30x/HG001_illumina_new.vcf.gz SI=gs://broad-dsde-methods-hydro-gen-truth-data-public/NIST/GIAB/v4/HG001_GRCh38_1_22_v4.2.1_benchmark.broad-header.vcf.gz H=Homo_sapiens_assembly38.haplotype_database.txt O=local_picard_test.txt

which worked fine, meaning that the difference was in the environment in Terra, not the tool itself. It also worked fine running locally with GATK 4.4.0.0, so the versioning was not the culprit.

Terra support wanted to blame the Picard tool for not having the --gcs-project-for-requester-pays flag, but it still worked locally, so it's not clear to me what exactly needs to be set in the environment to make it "just work."

Steps to reproduce

Try streaming a file from a requester-pays bucket in a Terra environment.

Expected behavior

The tool should be able to stream the bam/vcf inputs and calculate the fingerprint info from that.

Actual behavior

The tool gets an error when trying to stream instead.

[kachulis]

it seems like picard gets a project to use for a requester-pays bucket from google application default credentials, which presumably are not setup by default in terra.

edit: actually I'm not sure it makes sense that application default credentials wouldn't be setup in terra. someone who understands how all the cloud access stuff works probably has a better shot at answering this.

[kockan]

@lbergelson maybe?