How to use a service account for CI deployments

[marcosscriven]

Running clasp login sets up a .clasprc file with a token that seems to last about a week.

Is there any way to get some kind of authentication working that could work in a headless setup like CI (E.g. GitHub Travis or Bitbucket Pipelines) please?

I looked at https://script.google.com/home/usersettings which has a switch for the API, but nothing about service tokens.

---

Note from @grant, please upvote this bug! https://issuetracker.google.com/issues/36763096

[marcosscriven]

Note I tried the directions at https://developers.google.com/identity/protocols/OAuth2ServiceAccount and using the resultant JSON key in place of the .clasprc, though this didn't seem to work.

[marcosscriven]

This seems to be related to https://github.com/google/clasp/pull/28 - but although it doesn't require a browser, it still requires interaction on the command line.

[marcosscriven]

Pinging @grant on this one (as a recent committer to https://github.com/google/clasp/blob/master/src/auth.ts).

I looked at the oauth2 client used here, and it seems there is a way to set creds in an env var: https://www.npmjs.com/package/google-auth-library#loading-credentials-from-environment-variables

It even mentions the deployment use case.

Also - over in gapps, looks like there was a PR for such a request by @gunar https://github.com/danthareja/node-google-apps-script/pull/46/files

[grant]

clasp auto-refreshes access token for any clasp command when it expires (~24h?). You should only need to clasp login once. (Unless you add scopes or change users).

~/.clasprc.json has these:

• access_token
• refresh_token

This request is for using a service account rather than a user account. I think using --ownkey should already solve this, but I'll have to check more and document it.

[marcosscriven]

@grant thanks for looking into this issue.

Any kind of auto refresh would then need to persist - in the context of CI then, one would have to check if the local .clasprc file (which CI would have to be generated from secret env vars during the build) changed, and then somehow update the env vars that contain the secrets.

--own-key seems to only be about having a .clasprc file in the local directory, not using a service account JWT key of the form:

{
  "type": "service_account",
  "project_id": "project-id-xxxxxxxxx",
  "private_key_id": "xxx",
  "private_key": "-----BEGIN PRIVATE KEY-----\nxxxx\n-----END PRIVATE KEY-----\n",
  "client_email": "xxx",
  "client_id": "xxx",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/xxxxx.iam.gserviceaccount.com"
}

[marcosscriven]

To expand, the code snippet from google-auth-library is:

export CREDS='{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "your-private-key-id",
  "private_key": "your-private-key",
  "client_email": "your-client-email",
  "client_id": "your-client-id",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "your-cert-url"
}'

And then:

const {auth} = require('google-auth-library');

// load the environment variable with our keys
const keysEnvVar = process.env['CREDS'];
if (!keysEnvVar) {
  throw new Error('The $CREDS environment variable was not found!');
}
const keys = JSON.parse(keysEnvVar);

async function main() {
  // load the JWT or UserRefreshClient from the keys
  const client = auth.fromJSON(keys);
  client.scopes = ['https://www.googleapis.com/auth/cloud-platform'];
  await client.authorize();
  const url = `https://www.googleapis.com/dns/v1/projects/${keys.project_id}`;
  const res = await client.request({url});
  console.log(res.data);
}

So maybe either just implicitly be able to read JWT tokens of this sort in a .clasprc file, or have an explicit --jwtkey option that expects the key in an env var like this (preferred).

[campionfellin]

Hey @marcosscriven does this PR https://github.com/google/clasp/pull/223 solve your issue? It removes --ownkey in favor of --creds

This changes clasp login so that it loads those creds from a json file. Does your CI pipeline specifically need it to load from an environmental variable? We can probably make that an option as well.

However, it still requires you to select the Gmail account you're authorizing the app for. I can take a look at that sample code there and see if we can get it to work. Looks like

  await client.authorize();
  const url = `https://www.googleapis.com/dns/v1/projects/${keys.project_id}`;
  const res = await client.request({url});

may be where it authorizes without opening up in the browser.

I hope this helps! Also feel free to open your own PR, or ask more questions.

[marcosscriven]

@campionfellin - it certainly looks close. It doesn't have to be from env vars - I'm chiefly thinking about Bitbucket Cloud Pipelines https://confluence.atlassian.com/bitbucket/environment-variables-794502608.html

One could easily generate a JSON file at build time, populating the secrets from the env vars. It would be handy to avoid that step though, as show in the snippet.

[marcosscriven]

Also @campionfellin - I don't think the snippet you highlighted is anything to do with the auth - it's just an example of going on to use any of the APIs (dns in this instance).

The bit that enables it is simply const client = auth.fromJSON(keys), with the keys in the format I posted.

[campionfellin]

Hey @marcosscriven it looks like rather than using auth.fromJSON(...) I just read and parsed the file manually (https://github.com/campionfellin/clasp/blob/64b5301ccef7dcc99a2d9690c6d52db708975e08/src/auth.ts#L60). I can go ahead and change that.

However, I don't think this would solve the issue of getting the access_token or refresh_token that you need in your .clasprc.json file unless client.authorize(...) is what does that.

[campionfellin]

Hey @marcosscriven , so it does look like using what's in the sample will work. client.authorize(...) is what does the work for us. It will take some time to make those changes though.

[campionfellin]

Question for @grant : is this what we want to do by default, or add a flag for it? I am afraid of taking away the user's ability to see what scopes they are authorizing.

[marcosscriven]

@campionfellin - The scopes would have be chosen by the user while generating the service account key, so I think just working by default would be fine (so long as it was documented how to use this rather than a token).

[campionfellin]

My question is more for users who don't generate service accounts. What about a flag like --use-service-account ?

[grant]

Another flag sounds OK. I'm getting a bit confused by all the discussion here, but it seems like this is just a FR for adding another flag like clasp login --service-account and changing authorize to work with service accounts.

[marcosscriven]

@grant - I'm not clear of the purpose of 'logging in' with a service account? Logging in at the moment is just about getting a token into .clasprc - we don't need that if we've already downloaded json service account credentials.

To be clear, I would expect to be able to provide service account credentials (created according to https://developers.google.com/identity/protocols/OAuth2ServiceAccount), in either a file or env var, and for all remaining API actions to simply use client.authorize().

I think it should work fine by simply inferring behaviour from the contents of .clasprc - if it's just a token, use that. If it's credentials with a private key etc., then use that instead.

[campionfellin]

Hey @marcosscriven and @grant I've done a bit of investigation today, so here's a follow up, please correct me on any things I am misunderstanding:

1. According to here: "The Google OAuth 2.0 system supports server-to-server interactions such as those between a web application and a Google service. For this scenario you need a service account, which is an account that belongs to your application instead of to an individual end user. Your application calls Google APIs on behalf of the service account, so users aren't directly involved."

It goes on to explain that this is "2-legged OAuth", as compared to "3-legged OAuth" which can act on behalf of the user but needs user permission (like the pop-up that we currently have).

My understanding of what you want is essentially for your Bitbucket pipeline to interact with Google Services, without you having to open a page to login.

I don't think that with a Service Account or JWT this will be possible, for most clasp commands.

Anyway, here's how I tested it:

In GCP, I made a service account, with full "owner" access to the entire project. I downloaded those credentials (in the same format as you have above and same as the documentation) and used them to authenticate my API calls, like here:

https://github.com/google/clasp/blob/d807a6c72886a3cadf06bda53227fcb106f6b858/src/auth.ts#L41

But instead of the oauth2client I used the one I created as a JWT Client. Now the first command I tried was clasp list, but unfortunately got an empty array as a response. Why? Well because the service account's Drive is empty, all the scripts are located in the user's Drive. Ok, so let's clasp create. Unfortunately, you're hit with this:

Error: User has not enabled the Apps Script API. Enable it by visiting https://script.google.com/home/usersettings then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Since most of the clasp commands either use the Drive API or the Apps Scripts API (both of which are closely linked to the user themselves), I don't think that much can actually be done as far as using service accounts with clasp.

However, if all you really need is for your pipeline to work, there is a fairly simple solution, which I'll explain in my next comment.

[campionfellin]

So this is how I would solve your CI problem specifically, though we use Travis instead of BitBucket, it should be simple to translate.

On your local machine with some real user account (yours), use clasp login like normal and click to allow clasp access. That should save the ~/.clasprc.json file on your machine. What we did (but no longer do, for other reasons) is encrypt that file into ~/.clasprc.json.enc and then use the pipeline to decrypt it before running anything. (Here is the instructions for Travis: https://docs.travis-ci.com/user/encrypting-files/)

If you find that BitBucket doesn't allow that with files, but rather environmental variables, it would be a pretty simple change here:

https://github.com/google/clasp/blob/d807a6c72886a3cadf06bda53227fcb106f6b858/src/auth.ts#L61

To either read from a file or from ENV. However, Service Accounts and JWT will still not work.

Let me know if this at least unblocks you, or if you have further questions or can help me understand your situation better.

[marcosscriven]

@campionfellin Thanks for looking into this - as it happens, I'm not looking for impersonation in my case. There's two flavours of that in OAuth - there's the 3LO (Three-legged Oauth), which allows impersonation with a pre-shared key, but still requires user interaction for the user to accept. There's a much lesser known 2LOi (Two-legged Oauth with Impersonation) - but I don't see that mentioned anywhere in Google's docs.

Anyway - for me, I do have an account I setup just for services, and I give that account the rights to run scripts and access drives that way.

All I need here is for the OAuth client clasp uses to be setup with the service account json key (as in your penultimate post), and that'll work for me.

Regardless though - the method you specify for working around it (if one needed to) is what I considered to start with, but noted the token there has about a week's validity. At which point clasp can refresh the token with the URL it has in .clasprc - but that gets written to disk.

So this can't be one way in CI - it too would have to store (securely) any changes to the token that clasp made right?

[marcosscriven]

@campionfellin - I just wasted some hours on using the service account (which should work). The crucial step (even for a service account with 'Domain-wide delegation'), was that I still had to go to the script project and 'share' it with the service user email (of the form serviceuser@project-id-123456789.iam.gserviceaccount.com).

I'm pretty sure that last step is not meant to be necessary. Maybe related to https://issuetracker.google.com/issues/36763096?

@grant - As you're a member of the Google team on Github, is there any chance you can investigate this please? There's a lot of confusion around service accounts and the App Scripts API.

[marcosscriven]

To clarify, using the Python Google OAuth 2 client, this works - so long as I've 'shared' the script with the service account email:

from google.oauth2 import service_account
import googleapiclient.discovery
import json
import os

SCOPES = ['https://www.googleapis.com/auth/script.projects']
SERVICE_KEY = json.loads(os.environ['SERVICE_KEY'])

credentials = service_account.Credentials.from_service_account_info(SERVICE_KEY , scopes=SCOPES)

script = googleapiclient.discovery.build('script', 'v1', credentials=credentials)
response = script.projects().get(scriptId='myscriptid').execute()
print response

EDIT - So while this works, trying:

response = script.projects().updateContent(scriptId='myscriptid', body=body).execute()

Suddenly gives me a 403:

<HttpError 403 when requesting https://script.googleapis.com/v1/projects/myscriptid/content?alt=json returned "User has not enabled the Apps Script API. Enable it by visiting https://script.google.com/home/usersettings then retry.

Which is very peculiar given the the API is clearly being used during get operation...

[marcosscriven]

Trying to use delegation-wide service account the right way (E.g. without the 'sharing' hack I mentioned), even just reading the project fails:

SCOPES = ['https://www.googleapis.com/auth/script.projects', 'https://www.googleapis.com/auth/drive']
SERVICE_KEY = json.loads(os.environ['SERVICE_KEY'])

credentials = service_account.Credentials.from_service_account_info(SERVICE_KEY, scopes=SCOPES)
delegated_credentials = credentials.with_subject('<email>)
script = googleapiclient.discovery.build('script', 'v1', credentials=delegated_credentials)

response = script.projects().get(scriptId='<scriptId>').execute()

Fails with:

google.auth.exceptions.RefreshError: ('unauthorized_client: Client is unauthorized to retrieve access tokens using this method.', u'{\n  "error" : "unauthorized_client",\n  "error_description" : "Client is unauthorized to retrieve access tokens using this method."\n}')

Despite ensuring those API scopes have been authorized for that client ID as per https://developers.google.com/api-client-library/python/auth/service-accounts.

[marcosscriven]

Note I asked about this on Stack Overflow too https://stackoverflow.com/questions/51049548/how-can-i-publish-a-google-app-script-using-a-domain-wide-delegation-service-acc

[grant]

Hey @marcosscriven, thanks for all the investigation. I too want this feature and to reduce friction around this and a bunch of other setup.

Please upvote the linked bug and this issue so I can ask the Apps Script team to prioritize this. If there's a workaround that clasp can promote in the meantime, perhaps we can make that setup easy.

[marcosscriven]

@grant - I don't see any voting options there, but I've commented on it.

It says it's 'blocked by' https://issuetracker.google.com/issues/26400743, but I don't have view permissions on that.

[grant]

@marcosscriven It looks like this issue is being triaged by the Apps Script team. I've asked the team for an update on the issue. Unfortunately, it's a lot easier to change features in clasp than modify anything with the Apps Script tool/product.

[marcosscriven]

@grant good news! Let’s hope it gets fixed. Thanks for following for following up.

[grant]

To be honest, I don't expect this to be fixed by the team in the next 3 months, but we will see.

[aandis]

Any update on this? Is it possible to authenticate clasp using a service account credential?

[aandis]

for reference, this is how it works in gcloud

gcloud auth activate-service-account --key-file service-account-credentials.json

[aandis]

Out of curiosity, if I were to copy the .clasprc.json from my local to ci, what would happen after expiry_date? My understanding is that although access_token expires, refresh_token doesn't expire. So would clasp be able to fetch new tokens successfully?

[aandis]

What I'm basically asking is, once a user does a clasp login, will clasp ever ask the user to login again?

[grant]

I pinged again. You should only need to login once. Then run forever. We auto-refresh to access token with the Node client (googleapis).

You can read more about the token here: https://developers.google.com/identity/protocols/OAuth2#expiration I believe you can just login once and clasp will work for your service.

For example, clasp itself uses an encrypted token that is used for CI tests. I only had to login once.

[alexellis]

What is the summary of the workaround or fix for this?

[grant]

• Workaround: Create a new G Suite account and create scripts using that as a "service account"
• Action: Please upvote this bug: https://issuetracker.google.com/issues/36763096
  • This will allow me to tell the team that this is important

[andreacab]

Affected too. Upvoted the bug.

[ericanastas]

I pinged again. You should only need to login once. Then run forever. We auto-refresh to access token with the Node client (googleapis).

You can read more about the token here: https://developers.google.com/identity/protocols/OAuth2#expiration I believe you can just login once and clasp will work for your service.

For example, clasp itself uses an encrypted token that is used for CI tests. I only had to login on

Say I store a token and refresh token as part of a CI pipeline. This works for a while, but eventually the refresh token is used to create a new token. But the CI pipeline is ephemeral. This new refresh token is not persisted to the next CI run. So what happens when the CI process runs again and tries to use the original token/request token?

[sativ01]

Hi guys, I don't want to start a new thread, so I'll ask in this one

I want to use Github Actions to do clasp push on every push, and so far unsuccessful:

• I have a regular account in GCP and have downloaded creds.json and logged in locally, with clasp login --creds creds.json this resulted in .clasprc.json file that I have added to Github sercets and using it as environment variable
• While running CI I'm creating a .clasprc.json from that environment variable
• I verified that this file contains same data I have locally
• When it runs clasp push - it says I am not logged in. When it runs clasp login it's asked to use a URL, so build gets stuck

how do I make it login without a need to click on URL?

[ErikVanDenHoorn]

Hi all,

I am also trying to set this up, but then by using CloudBuild. After running clasp login --creds creds.json it still opens the browser. Where creds.json is a file with an OAuth 2.0 credential. I also created a service account and tried to sign in using that credential. But this credential has the wrong format. Any suggestions on how to set this up are appreciated.

[serrg]

@sativ01 I get this working by putting .clasprc.json into home directory, in my GitHub Actions I have:

echo $CLASPRC_CREDENTIALS > $HOME/.clasprc.json

where $CLASPRC_CREDENTIALS is content of .clasprc.json generated by clasp login --creds creds.json

[serrg]

Action: Please upvote this bug: https://issuetracker.google.com/issues/36763096

• This will allow me to tell the team that this is important

@grant Any news on service account support within App Script API (and later in clasp)?

[grant]

No news here ☹️

[ricardosllm]

+1 Looking to setup a CI pipeline that is able to do clasp push without sharing a (g suite) user's .clasprc.json with the CI pipeline, I find this pretty standard eg: AWS, and after 2 years this issue is still open...

[mattPiratt]

I also am waiting to have a solution for CI

[pregoli]

Any update here? Any recommendation to spin up a CI pipeline within Azure with following steps:

• Authenticate through ServiceAccount headless (clasp login)
• Pushing changes (clasp push)

Following this looks like an endless journey..

[ericanastas]

FYI

I've developed a CI/CD process for Google Apps Script using GitHub Actions.

See my comment here: https://github.com/google/clasp/issues/707#issuecomment-844364570

[pkit]

@ericanastas This one can be hardly called a "process" as .clasprc.json token will expire in 6 month.

[ericanastas]

@ericanastas This one can be hardly called a "process" as .clasprc.json token will expire in 6 month.

Did you look at the script? It's run by a cron trigger every week and stores.classprc.json if it is updated.

[bernardo-martinez]

is there a proper way to setup auth for service account yet? one more team impacted here :/

[pkit]

@ericanastas This one can be hardly called a "process" as .clasprc.json token will expire in 6 month.

Did you look at the script? It's run by a cron trigger every week and stores. classprc.json if it is updated.

Admin token required? No thanks, lol