Using Goth on GAE with no extra configuration - Config.get(:client_email) errors

[hubertlepicki]

I am trying to use Goth + Waffle + waffle_gcs on Google App Engine using the default service account, and without providing the credentials in any way through custom environmental variable or JSON file. I am using Goth 1.2, i.e. the version before redesign at the moment.

Things seem to work just fine, including uploading attachments to cloud storage, but waffle / waffle_gcs seem to fail on generating signed URL to display / download uploaded files.

I have pin pointed the error to this line over here:

https://github.com/tyler-eon/waffle_gcs/blob/master/lib/waffle/storage/google/url_v2.ex#L82

And indeed, when I try to manually execute Goth.Config.get(:client_email) I am getting :error instead of OK tuple.

I am not entirely sure where the problem is, and if that's a Goth or waffle_gcs issue so please excuse me if this is a wrong place to ask. Do you have any ideas what's wrong with my set up and if I do have to do a custom set up and give up using default service accoutnt?

[hubertlepicki]

So it looks like goth is not figuring out client_email from Google's Default Application Credentials using internal API, similar way that it does with project_id. This is kind of inconsistent, but easy to fix.

I don't know if this should land in Goth, having in mind the redesign thing ? but code that works for me is something liike:

https://github.com/amberbit/waffle_gcs/blob/support_gae_default_application_credentials/lib/waffle/storage/google/url_v2.ex#L165

[hubertlepicki]

@wojtekmach I'd like to make a proper fix to this at some point and was just wondering if you think client_email should be resolved by Goth the same way the project_id currently is, i.e. using instance metadata when on GAE / GCP, and thus the functionality to do so should be added to Goth, or should I continue doing that outside of Goth?

[wojtekmach]

regarding adding functionality to Goth, if it's for the "new" api that's more likely to be accepted, if for the "old", less so as we want to move away from that.

Sorry, I need more time to dig into this, I'll try to have a better answer soon. A pretty uninformed answer at this moment is this. I believe Waffle uses Goth as "credentials store", not just for token but also the email and stuff. In my opinion Goth is supposed to be just for generating tokens. For example, if the email is part of your credentials (e.g. the service account json thing), don't use Goth to later retrieve it. Store it yourself and retrieve it yourself. Similarly, if you're getting email from the google metadata service, don't use Goth for it, do it yourself. Does that make sense? Then I think the library is easier to understand and maintain. But maybe it poses unreasonable burden on the user, instead of letting users worry about all that, it should solve it once. I don't have an answer yet, any feedback is appreciated. :)

cc @peburrows

[hubertlepicki]

right, ok that's what I was thinking too about future of Goth and I was wondering if that even makes sense to add that functionality to the legacy modules if the intended usage is to not have them at some point. I think this makes sense @wojtekmach , but would be good for @peburrows to confirm

[peburrows]

In general, I agree with @wojtekmach on this — Goth should be pretty specifically focused on generating tokens, not used as a general purpose config retrieval or storage mechanism. The ability to retrieve the email from Goth's config is more an artifact / byproduct of the way the service account credentials were passed than an intentional interface for doing such things.