ERROR - Scopes not configured for service account

[juhawilppu]

Two days ago I successfully ran my first wdl on Cromwell using the Google Pipelines API. Then I tried to change my service account and it broke. I'm not able to get it running anymore at all. Stacktrace can be seen below, the error is "Scopes not configured for service account."

stdout

...
tsv_string += '\n' + "unmapped"

with open("sequence_grouping_with_unmapped.txt","w") as tsv_file_with_unmapped:
  tsv_file_with_unmapped.write(tsv_string)
  tsv_file_with_unmapped.close()
CODE`
2018-05-25 12:55:24,629 cromwell-system-akka.dispatchers.backend-dispatcher-137 ERROR - Scopes not configured for service account. Scoped should be specified by calling createScoped or passing scopes to constructor.
java.io.IOException: Scopes not configured for service account. Scoped should be specified by calling createScoped or passing scopes to constructor.
    at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:342)
    at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:160)
    at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:146)
    at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
    at cromwell.cloudsupport.gcp.genomics.GenomicsFactory$$anon$1.initialize(GenomicsFactory.scala:18)
    at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:277)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$PAPIRunCreationRequest.httpRequest$lzycompute(JesApiQueryManager.scala:293)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$PAPIRunCreationRequest.httpRequest(JesApiQueryManager.scala:293)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$PAPIApiRequest.contentLength(JesApiQueryManager.scala:278)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$PAPIApiRequest.contentLength$(JesApiQueryManager.scala:278)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$PAPIRunCreationRequest.contentLength(JesApiQueryManager.scala:291)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$$anonfun$receive$1.applyOrElse(JesApiQueryManager.scala:110)
    at akka.actor.Actor.aroundReceive(Actor.scala:514)
    at akka.actor.Actor.aroundReceive$(Actor.scala:512)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager.akka$actor$Timers$$super$aroundReceive(JesApiQueryManager.scala:33)
    at akka.actor.Timers.aroundReceive(Timers.scala:44)
    at akka.actor.Timers.aroundReceive$(Timers.scala:36)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager.aroundReceive(JesApiQueryManager.scala:33)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
    at akka.actor.ActorCell.invoke(ActorCell.scala:496)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I also tried on another computer and another GCP project just to verify that it is not a cache problem. I don't know what is wrong. Seems like the service account has a problem, but I did everything the same way as when it worked.

Some detailed information:

I have tried Cromwell 31.1 and 31.

I start Cromwell using this command java -Dconfig.file=google.conf -jar cromwell-31.jar server

google.conf (I have changed the actual project name to generic "project")

include required(classpath("application"))

google {

  application-name = "cromwell"

  auths = [
    {
      name = "application-default"
      scheme = "application_default"
    }
  ]
}

engine {
  filesystems {
    gcs {
      auth = "application-default"
      project = "project-test1"
    }
  }
}

backend {
  default = "JES"
  providers {
    JES {
      actor-factory = "cromwell.backend.impl.jes.JesBackendLifecycleActorFactory"
      config {
        // Google project
        project = "project-test1"

        // Base bucket for workflow executions
        root = "gs://project-test1/cromwell-execution"

        // Polling for completion backs-off gradually for slower-running jobs.
        // This is the maximum polling interval (in seconds):
        maximum-polling-interval = 600

        // Optional Dockerhub Credentials. Can be used to access private docker images.
        dockerhub {
          // account = ""
          // token = ""
        }

        genomics {
          // A reference to an auth defined in the `google` stanza at the top.  This auth is used to create
          // Pipelines and manipulate auth JSONs.
          auth = "application-default"
          // Endpoint for APIs, no reason to change this unless directed by Google.
          endpoint-url = "https://genomics.googleapis.com/"
          // This allows you to use an alternative service account to launch jobs, by default uses default service account
          compute-service-account = "default"
        }

        filesystems {
          gcs {
            // A reference to a potentially different auth for manipulating files via engine functions.
            auth = "application-default"
            project = "project-test1"
          }
        }
      }
    }
  }
}

I created the service account from https://cloud.google.com/docs/authentication/getting-started and give the role: Project -> Owner.

I've downloaded Google Cloud SDK and run these

gcloud auth login juha.wilppu@gmail.com
gcloud auth application-default login
gcloud config set project project-test1
gsutil ls gs://project-test1 // This command works, so authentication is successful.

project-test1-59b66448c3ab.json

{
  "type": "service_account",
  "project_id": "project-test1",
  "private_key_id": "59b66448c3ab730097135e1dba83b375a6b57ea3",
  "private_key": "-----BEGIN PRIVATE KEY-----\n(Omitted)\n-----END PRIVATE KEY-----\n",
  "client_email": "project-service@project-test1.iam.gserviceaccount.com",
  "client_id": "104927211954691424974",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/project-service%40project-test1.iam.gserviceaccount.com"
}

[Horneth]

Hi, I see you've indeed created a service account and gotten a json file, but I'm not seeing how you're passing it to Cromwell. Your configuration uses application_default as authentication mode, and you are logged in using your personal gmail it seems. Did you use this json in any way ?

To use the service account in Cromwell you'd want to either 1 - Recommended) Change your configuration to use the service account instead of application default You can see how to do that here It is slightly outdated, instead of pem-file use json-file and the path to your json. 2) You can keep application default and use gcloud auth activate-service-account to authenticate as the service account on your machine.

Also could you print the result of gcloud auth list ?

[juhawilppu]

I got it working now by setting the service-account in google.conf. Excellent, thanks for your help!

I was looking into the wrong place: I didn't realize that Cromwell did not find the account at all, I thought it just had a problem with access rights.

---

To answers to your questions, I had set the environment variable export GOOGLE_APPLICATION_CREDENTIALS=/Users/jwilppu/cromwell/project-test1-59b66448c3ab.json by following these instructions https://cromwell.readthedocs.io/en/develop/tutorials/PipelinesApi101/ which has a link to this page https://cloud.google.com/docs/authentication/production .

The result of command gcloud auth list is

       Credentialed Accounts
ACTIVE  ACCOUNT
*       juha.wilppu@gmail.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

I have now removed the environment variable.

[Horneth]

That's good to know thanks, looks like the docs need an update :) I'm going to close this ticket, don't hesitate to re-open it if you run into the same issue again.

[multimeric]

Hmm, this seems a bit odd. The application_default authentication should still work with a service account, as long as you set the $GOOGLE_APPLICATION_CREDENTIALS variable is set, which @juhawilppu seems to have done here. I had this same issue, where my service account only worked once I used a scheme = "service_account", but that seems like something is implemented wrongly.

[freeseek]

I had the same issue. I got the same error message:

[2020-07-27 18:34:00,37] [error] PipelinesApiAsyncBackendJobExecutionActor [3d2d7a27wf_hello.hello:NA:1]: Error attempting to Execute
cromwell.engine.io.IoAttempts$EnhancedCromwellIoException: [Attempted 1 time(s)] - StorageException: xxx@xxx.iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project.
Caused by: com.google.cloud.storage.StorageException: xxx@xxx.iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project.

I had set up my credentials with:

export GOOGLE_APPLICATION_CREDENTIALS=sa.json

and had this configuration in google.conf copied from the tutorial:

google {
  application-name = "cromwell"
  auths = [
    {
      name = "application-default"
      scheme = "application_default"
    }
  ]
}

engine {
  filesystems {
    gcs {
      auth = "application-default"
      project = "xxx"
    }
  }
}

That clearly did not work. I tried to follow the logic in this post. I followed Horneth suggestion to use service-account's authorization and I took the auths configuration and changed pem-file to json-file in google.conf as follows:

google {
  application-name = "cromwell"
  auths = [
    {
      name = "service_account"
      scheme = "service_account"
      service-account-id = "xxx@xxx.iam.gserviceaccount.com"
      json-file = "sa.json"
    }
  ]
}

engine {
  filesystems {
    gcs {
      auth = "service_account"
      project = "xxx"
    }
  }
}

And I have replaced every other instance of auth = "application-default" with auth = "service_account". Now when I run Cromwell:

java -Dconfig.file=google.conf -jar cromwell-52.jar run hello.wdl -i hello.inputs

I don't get the error anymore. I do get a different error:

[2020-07-27 22:54:56,48] [info] WorkflowManagerActor Workflow 0fb5e69d-7d70-407e-9fe2-bf7cb2b2c3e6 failed (during ExecutingWorkflowState): java.lang.Exception: Task wf_hello.hello:NA:1 failed. The job was stopped before the command finished. PAPI error code 7. Required 'compute.zones.list' permission for 'projects/xxx'

I don't know what this means.

If I remove Requester pays from the bucket I can get the WDL to work using scheme = "application_default", as long as I do not export GOOGLE_APPLICATION_CREDENTIALS first. But if I use Requester pays on the bucket, using scheme = "application_default" causes error:

[2020-07-27 23:19:31,90] [info] WorkflowManagerActor Workflow 4c8a642a-19a6-486b-acad-e0adf3168820 failed (during ExecutingWorkflowState): java.lang.Exception: Task wf_hello.hello:NA:1 failed. The job was stopped before the command finished. PAPI error code 10. 15: Gsutil failed: failed to upload logs for "gs://xxx/cromwell-execution/wf_hello/4c8a642a-19a6-486b-acad-e0adf3168820/call-hello/": cp failed: gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://xxx/cromwell-execution/wf_hello/4c8a642a-19a6-486b-acad-e0adf3168820/call-hello/, command failed: BadRequestException: 400 Bucket is requester pays bucket but no user project provided.

So I still have not found a way to run the WDL with Requester pays on. I wish Cromwell could give errors explaining what steps to take to solve the issue ... I know that with gsutil I can specify the user project with -u xxx but I have no idea how to do that with Cromwell.