Unable to deploy gmailr in production

[pythiantech]

I have a {golem} based {shiny} app where I am composing and sending emails using {gmailr}. I have no difficulty while using it locally, however when I dockerize the shiny app and deploy it on the production server, every time I try to send an email, I get a must create an app and register it with gm_auth_configure() error. This is my configuration for gmailr:

#GmailR configuration
gmailr::gm_auth_configure(path = path_to_credentials)
gmailr::gm_auth(email = TRUE, cache = gmail_secret)

I have copied my credentials.json file as well as the secret token on the production server and provided the correct path too. But this doesn't seem to be working.

[sanjmeh]

Was this fixed?

[pythiantech]

No :(

[jennybc]

It's hard to say what's going wrong, except that your locally-obtained token (what I hope / assume you mean by "the secret token") does not appear to be successfully found / used / refreshed. It appears gmailr is trying to do the OAuth dance on the server, which is not what you want. Without seeing the app code, I can't say any more.

[pythiantech]

@jennybc thanks for your comment. Since this is a gigantic app with multiple modules, I will try and explain the relevant parts. The app has been designed and developed in the {golem} framework as a package. This is what the directory structure looks like locally: Within the data folder, amongst other folders and files, I have my credentials.json and a folder called .secret. Within the inst folder I have a file called golem-config.yml file with different locations defined for gmailr related configuration for local. and production versions of the app. Relevant lines are:

default:
  golem_name: TMSGolem
  golem_version: 0.0.0.9000
  app_prod: no
  dbpath: data/ctms.db
  gmailr_config: data/credentials.json
  gmail_secret: data/.secret
production:
  app_prod: yes
  dbpath: /home/data/ctms.db
  gmailr_config: /home/data/credentials.json
  gmail_secret: /home/data/.secret
dev:
  golem_wd: /Users/dhirajkhanna/Documents/shiny_experiments/TMSGolem

To use the application in production I use a Dockerfile with the relevant code as below:

FROM rocker/r-ver:4.0.2
RUN apt-get update && apt-get install -y  git-core imagemagick libcairo2-dev libcurl4-openssl-dev libfftw3-dev libgit2-dev libglpk-dev libgmp-dev libicu-dev libpng-dev libssl-dev libtiff-dev libxml2-dev make pandoc pandoc-citeproc zlib1g-dev && rm -rf /var/lib/apt/lists/*
RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl', Ncpus = 4)" >> /usr/local/lib/R/etc/Rprofile.site
RUN R -e 'install.packages("remotes")'

RUN mkdir /home/data
COPY data /home/data
RUN mkdir /build_zone
ADD . /build_zone
WORKDIR /build_zone
RUN R -e 'remotes::install_local(upgrade="never")'
RUN rm -rf /build_zone
EXPOSE 80
CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0');Sys.setenv('GOLEM_CONFIG_ACTIVE' = 'production');TMSGolem::run_app()"

As you can see, I create a folder /home/data and copy all the contents of my local data folder into it while dockerizing the application.

Then in my golem_utils_server.R file which is in the R folder, I have the following configuration for gmailr:

gmailr::gm_auth_configure(path = gmailr_config)
gmailr::gm_auth(email = TRUE, cache = gmail_secret)

In one of the modules of the app, when a user clicks on an action button, the following gmailr related action is executed:

vetting_mail <- gmailr::gm_mime() %>% 
        gmailr::gm_to(mail_to) %>% 
        gmailr::gm_subject(paste('Vetting of MoM -', input$mom_ser_no)) %>% 
        gmailr::gm_html_body(paste(
          'Dear', paste0(input$mom_vetter, ','),
          br(), br(),
          'You are requested to vett MoM -', 
          input$mom_ser_no, 'titled', '<b>',
          input$mom_title, '</b>',
          'held on', input$mom_dttm,
          br(), br(),
          'The following members were present:',
          br(),
          '<b>', mom_members, '</b>',
          br(), br(),

          'The agenda of the meeting was as follows:', br(), br(),
          '<i>', input$mom_agenda, '</i>', br(), br(),
          'Discussion points are tabulated below:',
          br(), br(),
          tableHTML::tableHTML(df)
        ))
      gmailr::gm_send_message(vetting_mail)

The above works without any problems while running locally by using the run_app() function from golem. However in production it fails with the error message must create an app and register it with gm_auth_configure(). On the production server I have checked the existence of /home/data folder inside the Docker container and it houses both the credentials.json and .secret. Please do let me know if you need any further details.

[jennybc]

I'm staying rather zoomed out here, but it's clear your app has a lot of moving parts and my hunch is that this is ultimately a path / file finding problem.

Here are some troubleshooting ideas.

How sure are you that your local app is using the token you've embedded in the app vs. one stored in the default, user-level location? I would test this explicitly.

Look hard at any ignore files that could influence which files get copied / deployed. Is the embedded token cache somehow being omitted?

Create the very simplest app you can that does the gmailr task, then start adding other complexities. And/or build some "print debugging" into your current app to reveal which files exist. Maybe you can use some general gargle debugging ideas to get very detailed info on why the existing token is not being used.

https://gargle.r-lib.org/articles/troubleshooting.html

[pythiantech]

@jennybc thanks again for your insights. So here's what I did. Set the gargle verbosity to "debug".

options(
  gargle_verbosity = "debug"
)

I also ran the gargle::token_fetch() function which generated a new token in my ~/Library/Caches/gargle Now when I run the app locally, everytime I get this:

ℹ Loading TMSGolem
trying `token_fetch()`
trying `credentials_service_account()`
Error caught by `token_fetch()`:
Argument 'txt' must be a JSON string, URL or file.
trying `credentials_external_account()`
aws.ec2metadata not installed; can't detect whether running on EC2 instance
trying `credentials_app_default()`
trying `credentials_gce()`
trying `credentials_byo_oauth()`
Error caught by `token_fetch()`:
inherits(token, "Token2.0") is not TRUE
trying `credentials_user_oauth2()`
Gargle2.0 initialize
adding "userinfo.email" scope
loading token from the cache
no matching token in the cache
initiating new token
Waiting for authentication in browser...
Press Esc/Ctrl + C to abort

I then need to authenticate in the browser. Post authentication, I get this:

Authentication complete.
putting token into the cache:
data/.secret
Writing NAMESPACE
Writing NAMESPACE
ℹ Loading TMSGolem
trying `token_fetch()`
trying `credentials_service_account()`
Error caught by `token_fetch()`:
Argument 'txt' must be a JSON string, URL or file.
trying `credentials_external_account()`
aws.ec2metadata not installed; can't detect whether running on EC2 instance
trying `credentials_app_default()`
trying `credentials_gce()`
trying `credentials_byo_oauth()`
Error caught by `token_fetch()`:
inherits(token, "Token2.0") is not TRUE
trying `credentials_user_oauth2()`
Gargle2.0 initialize
adding "userinfo.email" scope
loading token from the cache
ℹ The gmailr package is using a cached token for tmsdahraglobal@gmail.com.
matching token found in the cache

And I am now able to send an email. I am not sure I follow completely, but I think a new token is being generated for every session. So even if I copied this token to my production server and stored in the path that I have specified, it won't work, would it? Is there some way that I could avoid this authentication per session and just use a permanent token?

I will go through the gargle articles in the meantime.

[pythiantech]

@jennybc just not able to figure this out. I have even tried getting into the Docker container and trying out these steps from there, but no joy.

[careercoachme]

I had a similar problem - could run gmailr locally but when I was trying to run it non-interactively on AWS it was failing. I realised I had multiple files in the .secret folder and when I kept only the most recent file in that .secret folder I was able to run it remotely (once I had copied across both the JSON file and the .secret folder)

[pythiantech]

I had a similar problem - could run gmailr locally but when I was trying to run it non-interactively on AWS it was failing. I realised I had multiple files in the .secret folder and when I kept only the most recent file in that .secret folder I was able to run it remotely (once I had copied across both the JSON file and the .secret folder)

I have only the most recent file in .secret folder but unfortunately this still doesn't work. Any other suggestions?

[jennybc]

And I am now able to send an email. I am not sure I follow completely, but I think a new token is being generated for every session. So even if I copied this token to my production server and stored in the path that I have specified, it won't work, would it?

No, this approach is viable. What you saw was this:

trying `credentials_user_oauth2()`
Gargle2.0 initialize
adding "userinfo.email" scope
loading token from the cache
ℹ The gmailr package is using a cached token for tmsdahraglobal@gmail.com.
matching token found in the cache

which shows a cached token being found and used, not a new token being generated. It is true that such a token might need to be refreshed but this does not require user interaction.

Is there some way that I could avoid this authentication per session and just use a permanent token?

The easiest way to do that would be to use a service account token. Then, however, the emails would be sent from that service account, by default, which might be undesirable.

My leading hypothesis for why you're not successfully using a cached token in the deployed app is that the OAuth client (a.k.a "app") is different.

There's been quite a bit of change in gargle since this story started, so I think it's best to close this issue. And if you're still having trouble, open a new issue with results based on current gargle.