search

guardian / grid

The Guardian’s image management system
https://www.theguardian.com/info/developer-blog/2015/aug/12/open-sourcing-grid-image-service
Apache License 2.0
1.44k stars 120 forks source link

AWS access denied (S3 403) #2360

Closed aafonsobbc closed 5 years ago

aafonsobbc commented 5 years ago

Dear Grid Team,

Following your instructions from here https://github.com/guardian/grid/tree/master/docs, all went almost fine (probably still not fully configured domains and nginx though)

Anyway, after running ./dev_start.sh, all seems to start nicely but, when testing with the_pinger.sh, 500 is in response for all modules and, hitting directly using a browser, I got the below error.

AWS configuration (credentials file under .aws) seems to work out for configuration scripts and other steps, just wondering if I have to add key/secret values somewhere else (the pandomainauth thing is something I havent filled anywhere)

Any tip about this issue??

Thanks!!

Alfonso Afonso

AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 9ECEAE6D34814AAF; S3 Extended Request ID: YX9i/9EbNr1dwia+fX3CpGfyQxvh+FTTSoOpm3uB13LbSko1mWSWRkJyA/zVmZYyTSSwOjlUqEk=)

No source available, here is the exception stack trace:
->com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 9ECEAE6D34814AAF; S3 Extended Request ID: YX9i/9EbNr1dwia+fX3CpGfyQxvh+FTTSoOpm3uB13LbSko1mWSWRkJyA/zVmZYyTSSwOjlUqEk=)
     com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1630)
     com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1302)
     com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
     com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
     com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
     com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
     com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
     com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
     com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
     com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4330)
     com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4277)
     com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1410)
     com.gu.pandomainauth.service.S3Bucket.readDomainSettings(S3Bucket.scala:22)
     com.gu.pandomainauth.PanDomainAuthSettingsRefresher.settingsMap$lzycompute(PanDomainAuthSettingsRefresher.scala:37)
     com.gu.pandomainauth.PanDomainAuthSettingsRefresher.settingsMap(PanDomainAuthSettingsRefresher.scala:37)
     com.gu.pandomainauth.PanDomainAuthSettingsRefresher.authSettings$lzycompute(PanDomainAuthSettingsRefresher.scala:38)
     com.gu.pandomainauth.PanDomainAuthSettingsRefresher.authSettings(PanDomainAuthSettingsRefresher.scala:38)
     com.gu.pandomainauth.PanDomainAuthSettingsRefresher.domainSettingsRefreshActor$lzycompute(PanDomainAuthSettingsRefresher.scala:40)
     com.gu.pandomainauth.PanDomainAuthSettingsRefresher.domainSettingsRefreshActor(PanDomainAuthSettingsRefresher.scala:40)
     com.gu.pandomainauth.PanDomainAuthSettingsRefresher.<init>(PanDomainAuthSettingsRefresher.scala:42)
     com.gu.mediaservice.lib.auth.Authentication.buildPandaSettings(Authentication.scala:75)
     com.gu.mediaservice.lib.auth.Authentication.panDomainSettings$lzycompute(Authentication.scala:46)
     com.gu.mediaservice.lib.auth.Authentication.panDomainSettings(Authentication.scala:46)
     com.gu.pandomainauth.action.AuthActions.settings(Actions.scala:26)
     com.gu.pandomainauth.action.AuthActions.$init$(Actions.scala:71)
     com.gu.mediaservice.lib.auth.Authentication.<init>(Authentication.scala:24)
     com.gu.mediaservice.lib.play.GridComponents.<init>(GridComponents.scala:33)
     ThrallComponents.<init>(ThrallComponents.scala:7)
     AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
     AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
     com.gu.mediaservice.lib.play.GridAppLoader.load(GridAppLoader.scala:11)
     play.core.server.DevServerStart$$anon$1.$anonfun$reload$3(DevServerStart.scala:174)
     play.utils.Threads$.withContextClassLoader(Threads.scala:21)
     play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:171)
     play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
     play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:202)
     play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:117)
     akka.stream.impl.fusing.MapAsync$$anon$25.onPush(Ops.scala:1194)
     akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519)
     akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:482)
     akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:378)
     akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:585)
     akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:469)
     akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:560)
     akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:742)
     akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:757)
     akka.actor.Actor.aroundReceive(Actor.scala:517)
     akka.actor.Actor.aroundReceive$(Actor.scala:515)
     akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:667)
     akka.actor.ActorCell.receiveMessage(ActorCell.scala:590)
     akka.actor.ActorCell.invoke(ActorCell.scala:559)
     akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
     akka.dispatch.Mailbox.run(Mailbox.scala:224)
     akka.dispatch.Mailbox.exec(Mailbox.scala:234)
     akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
     akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
     akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
     akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
akash1810 commented 5 years ago

Firstly, hi @aafonsobbc thanks for using Grid. Thanks also for the stack trace as its shown an omission in the documentation.

Grid uses the pan-domain-authentication (Panda) library for auth. Panda's configuration lives in an S3 bucket which can be configured else will default to the one the Guardian uses. You are getting a 403 because you don't have access to the Guardian's bucket.

aafonsobbc commented 5 years ago

Hi Akash

Thanks so much for the answer and links, I will configure it then to a "local" bucket in my AWS account following your instructions.

Have a good one!!

Alfonso Afonso

aafonsobbc commented 5 years ago

Dear Akash / Guardian Grid Team,

Following your advice, I have read the documentation and created the S3 bucket. Now, it is expected a config file with a Google API Credentials config, and I am wondering if there is any other way to configure it or maybe it would be a good idea, just for testing / evaluating purposes, to provide some documentation or examples.

As we are using also AWS, maybe could be useful adding a template to generate IAM roles/credentials?

Thanks for your help and support.

Alfonso Afonso

aafonsobbc commented 5 years ago

Hi again @akash1810 and team... just to add that I do believe that would be great and worth the time a bit more detailed doc about what Google API credentials we need and the creation process... I am currently trying to figure out that part and would appreciate some more detail too.

mbarton commented 5 years ago

Hi @aafonsobbc, thanks again for trying the Grid!

The Google Credentials can be generated from the Google Cloud Developer Console. Once you've created a project, you can enable the Google+ API and generate service account credentials. The instructions to follow are here.

Feel free to chat us directly on gitter as well. There's still a lot of Guardian-specific defaults in the Grid that I think we could work on.

In an ideal world, what auth mechanism would you be looking to use?

aafonsobbc commented 5 years ago

Hi @mbarton, thanks for your promptly response!! I will follow the instructions to generate the OAUTH service with Google, and thanks for the freedom to contact back or chat to follow up.

About auth mechanism, I was just wondering if, as far as the product is using Dynamo, S3, SQS, etc on AWS, adding Cognito (https://aws.amazon.com/cognito/) to manage credentials and access wouldn't be beneficial (and thus, having a stack to generate it for this DEV Grid tool)

Thanks again!!

Alfonso Afonso

mbarton commented 5 years ago

Great. Agreed about Cognito, I'll take a look into it over the next few days

aafonsobbc commented 5 years ago

Hi again @mbarton ... after manually adding origin.crops and the Google API id, p12, etc in the PAN bucket, now logs said (kahuna logs) what I am pasting below.

Can be a missing configuration or issue with NGINX or is still related with Google API access??

Thanks

Alfonso Afonso

2018-11-30 13:34:01,540 - [ERROR] - from application in play-dev-mode-akka.actor.default-dispatcher-6 markers= 

! @7a4k8jaj4 - Internal server error, for (GET) [/] ->

play.api.UnexpectedException: Unexpected exception[AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 742B17DEC73F8C55; S3 Extended Request ID: FH1M9/MAoV5DFqah/dqJQEale9U1B1GySnq3gMQ7hyqQ5jQu5KPaU32dFrBbB1k8PTj4kWvfJhI=)]
    at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:186)
    at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
    at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:202)
    at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:117)
    at akka.stream.impl.fusing.MapAsync$$anon$25.onPush(Ops.scala:1194)
    at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519)
    at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:482)
    at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:378)
    at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:585)
    at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:469)
    at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:560)
    at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:742)
    at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:757)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:667)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590)
    at akka.actor.ActorCell.invoke(ActorCell.scala:559)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 742B17DEC73F8C55; S3 Extended Request ID: FH1M9/MAoV5DFqah/dqJQEale9U1B1GySnq3gMQ7hyqQ5jQu5KPaU32dFrBbB1k8PTj4kWvfJhI=)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1630)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1302)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4330)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4277)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1410)
    at com.gu.pandomainauth.service.S3Bucket.readDomainSettings(S3Bucket.scala:22)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.settingsMap$lzycompute(PanDomainAuthSettingsRefresher.scala:37)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.settingsMap(PanDomainAuthSettingsRefresher.scala:37)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.authSettings$lzycompute(PanDomainAuthSettingsRefresher.scala:38)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.authSettings(PanDomainAuthSettingsRefresher.scala:38)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.domainSettingsRefreshActor$lzycompute(PanDomainAuthSettingsRefresher.scala:40)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.domainSettingsRefreshActor(PanDomainAuthSettingsRefresher.scala:40)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.<init>(PanDomainAuthSettingsRefresher.scala:42)
    at com.gu.mediaservice.lib.auth.Authentication.buildPandaSettings(Authentication.scala:75)
    at com.gu.mediaservice.lib.auth.Authentication.panDomainSettings$lzycompute(Authentication.scala:46)
    at com.gu.mediaservice.lib.auth.Authentication.panDomainSettings(Authentication.scala:46)
    at com.gu.pandomainauth.action.AuthActions.settings(Actions.scala:26)
    at com.gu.pandomainauth.action.AuthActions.$init$(Actions.scala:71)
    at com.gu.mediaservice.lib.auth.Authentication.<init>(Authentication.scala:24)
    at com.gu.mediaservice.lib.play.GridComponents.<init>(GridComponents.scala:33)
    at KahunaComponents.<init>(KahunaComponents.scala:9)
    at AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
    at AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
    at com.gu.mediaservice.lib.play.GridAppLoader.load(GridAppLoader.scala:11)
    at play.core.server.DevServerStart$$anon$1.$anonfun$reload$3(DevServerStart.scala:174)
    at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
    at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:171)
    ... 24 common frames omitted

2018-11-30 13:34:01,825 - [ERROR] - from application in play-dev-mode-akka.actor.default-dispatcher-7 markers= 

! @7a4k8jaj4 - Internal server error, for (GET) [/favicon.ico] ->

play.api.UnexpectedException: Unexpected exception[AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 742B17DEC73F8C55; S3 Extended Request ID: FH1M9/MAoV5DFqah/dqJQEale9U1B1GySnq3gMQ7hyqQ5jQu5KPaU32dFrBbB1k8PTj4kWvfJhI=)]
    at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:186)
    at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
    at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:202)
    at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:117)
    at akka.stream.impl.fusing.MapAsync$$anon$25.onPush(Ops.scala:1194)
    at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519)
    at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:482)
    at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:378)
    at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:585)
    at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:469)
    at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:560)
    at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:742)
    at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:757)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:667)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590)
    at akka.actor.ActorCell.invoke(ActorCell.scala:559)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 742B17DEC73F8C55; S3 Extended Request ID: FH1M9/MAoV5DFqah/dqJQEale9U1B1GySnq3gMQ7hyqQ5jQu5KPaU32dFrBbB1k8PTj4kWvfJhI=)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1630)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1302)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4330)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4277)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1410)
    at com.gu.pandomainauth.service.S3Bucket.readDomainSettings(S3Bucket.scala:22)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.settingsMap$lzycompute(PanDomainAuthSettingsRefresher.scala:37)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.settingsMap(PanDomainAuthSettingsRefresher.scala:37)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.authSettings$lzycompute(PanDomainAuthSettingsRefresher.scala:38)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.authSettings(PanDomainAuthSettingsRefresher.scala:38)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.domainSettingsRefreshActor$lzycompute(PanDomainAuthSettingsRefresher.scala:40)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.domainSettingsRefreshActor(PanDomainAuthSettingsRefresher.scala:40)
    at com.gu.pandomainauth.PanDomainAuthSettingsRefresher.<init>(PanDomainAuthSettingsRefresher.scala:42)
    at com.gu.mediaservice.lib.auth.Authentication.buildPandaSettings(Authentication.scala:75)
    at com.gu.mediaservice.lib.auth.Authentication.panDomainSettings$lzycompute(Authentication.scala:46)
    at com.gu.mediaservice.lib.auth.Authentication.panDomainSettings(Authentication.scala:46)
    at com.gu.pandomainauth.action.AuthActions.settings(Actions.scala:26)
    at com.gu.pandomainauth.action.AuthActions.$init$(Actions.scala:71)
    at com.gu.mediaservice.lib.auth.Authentication.<init>(Authentication.scala:24)
    at com.gu.mediaservice.lib.play.GridComponents.<init>(GridComponents.scala:33)
    at KahunaComponents.<init>(KahunaComponents.scala:9)
    at AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
    at AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
    at com.gu.mediaservice.lib.play.GridAppLoader.load(GridAppLoader.scala:11)
    at play.core.server.DevServerStart$$anon$1.$anonfun$reload$3(DevServerStart.scala:174)
    at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
    at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:171)
    ... 24 common frames omitted
mbarton commented 5 years ago

Thanks for persevering! I'm making a note of all the things we're finding here so we can hopefully streamline this in the future.

It looks like the pan-domain-authentication library the Grid uses internally cannot load settings because the settings file it expects to find in S3 doesn't exist.

You can configure the S3 bucket where this settings file lives through the PANDA_BUCKET_NAME environment variable. The format of the file is documented here.

It will try to download this settings file from the root of the S3 bucket based on the value of domain.root in your auth.properties config file.

s3://${PANDA_BUCKET_NAME}/${domain.root}.settings
aafonsobbc commented 5 years ago

Thanks @mbarton !!! I am also writing down a step by step log, happy to share it with you when all is done.

About the issue, well spotted, I had added a file but with the full domain nane instead of the root domain, my fault... after that, it asked for a secret key not present in the file.

I have added a secret key with a random value and it has started, now what I can finally "something" in the browser and the following message:

Error loading results: {"uri":"https://api.media.example.com","body":null,"status":-1,"headers":{"Location":null,"Content-Type":null}}

Thanks again for your support!!

Alfonso Afonso

aafonsobbc commented 5 years ago

Maybe related to permissions and users (above error)... here logs 

java.lang.IllegalArgumentException: Unexpected CSV headers []. Expected [CproName, Id]
    at lib.UsageStore$.csvParser(UsageStore.scala:117)
    at lib.UsageStore.fetchUsage(UsageStore.scala:152)
    at lib.UsageStore.update(UsageStore.scala:143)
    at com.gu.mediaservice.lib.BaseStore.$anonfun$scheduleUpdates$1(BaseStore.scala:60)
    at akka.actor.Scheduler$$anon$2.run(Scheduler.scala:83)
    at akka.actor.LightArrayRevolverScheduler$$anon$2$$anon$1.run(LightArrayRevolverScheduler.scala:101)
    at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
[ERROR] [11/30/2018 15:08:06.158] [default-akka.actor.default-dispatcher-3] [akka://default/user/$a] Could not refresh permissions from S3
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: B0F40C1A07629FC1; S3 Extended Request ID: jATBEUH1e+FgG5VFYYlMm86voh+L1Sc0+jmga7B+cIgf+3Mdbp4Vy1ImBsRr99FCqNFDiyZbmWs=), S3 Extended Request ID: jATBEUH1e+FgG5VFYYlMm86voh+L1Sc0+jmga7B+cIgf+3Mdbp4Vy1ImBsRr99FCqNFDiyZbmWs=
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1630)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1302)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4330)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4277)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1410)
    at com.gu.editorial.permissions.client.AmazonS3.getObject(PermissionsStore.scala:200)
    at com.gu.editorial.permissions.client.AmazonS3.getContentsAndLastModified(PermissionsStore.scala:213)
    at com.gu.editorial.permissions.client.PermissionsStoreFromS3.get(PermissionsStore.scala:92)
    at com.gu.editorial.permissions.client.PermissionsStoreFromS3.$anonfun$refreshActor$1(PermissionsStore.scala:84)
    at com.gu.editorial.permissions.client.PermissionsStoreRefreshActor$$anonfun$receive$1.applyOrElse(PermissionsStore.scala:138)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at com.gu.editorial.permissions.client.PermissionsStoreRefreshActor.aroundReceive(PermissionsStore.scala:127)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590)
    at akka.actor.ActorCell.invoke(ActorCell.scala:559)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
aafonsobbc commented 5 years ago

Also, after this page, the browser is redirected and got this error:

Action Not Found
For request 'GET /login?redirectUri=https%3A%2F%2Fmedia.example.com%2Fsearch'

These routes have been tried, in this order:
1GET/controllers.MediaApi.index
2GET/images/metadata/$field<[^/]+>controllers.SuggestionController.metadataSearch(field:String, q:Option[String])
3GET/images/edits/$field<[^/]+>controllers.SuggestionController.editsSearch(field:String, q:Option[String])
4GET/images/aggregations/date/$field<[^/]+>controllers.AggregationController.dateHistogram(field:String, q:Option[String])
5GET/images/$id<[^/]+>controllers.MediaApi.getImage(id:String)
6GET/images/$id<[^/]+>/fileMetadatacontrollers.MediaApi.getImageFileMetadata(id:String)
7GET/images/$imageId<[^/]+>/export/$exportId<[^/]+>controllers.MediaApi.getImageExport(imageId:String, exportId:String)
8GET/images/$imageId<[^/]+>/exportcontrollers.MediaApi.getImageExports(imageId:String)
9GET/images/$imageId<[^/]+>/downloadcontrollers.MediaApi.downloadOriginalImage(imageId:String)
10POST/images/$id<[^/]+>/reindexcontrollers.MediaApi.reindexImage(id:String)
11DELETE/images/$id<[^/]+>controllers.MediaApi.deleteImage(id:String)
12GET/imagescontrollers.MediaApi.imageSearch
13GET/suggest/metadata/creditcontrollers.SuggestionController.suggestMetadataCredit(q:Option[String], size:Option[Int])
14GET/suggest/metadata/photoshootcontrollers.SuggestionController.suggestPhotoshoot(q:Option[String], size:Option[Int])
15GET/usage/supplierscontrollers.UsageController.bySupplier
16GET/usage/suppliers/$id<[^/]+>controllers.UsageController.forSupplier(id:String)
17GET/usage/quotascontrollers.UsageController.quotas
18GET/usage/quotas/$id<[^/]+>controllers.UsageController.quotaForImage(id:String)
19GET/management/healthcheckcontrollers.HealthCheck.healthCheck
20GET/management/manifestcom.gu.mediaservice.lib.management.Management.manifest
21GET/robots.txtcom.gu.mediaservice.lib.management.Management.disallowRobots
aafonsobbc commented 5 years ago

Just in case it can help, here the results from the_pingler.sh

Pingling!

200 http://localhost:9010/management/healthcheck

200 http://localhost:9003/management/healthcheck

200 http://localhost:9006/management/healthcheck

200 http://localhost:9007/management/healthcheck

200 http://localhost:9002/

200 http://localhost:9005/management/healthcheck

200 http://localhost:9001/management/healthcheck

500 http://localhost:9009/management/healthcheck

200 http://localhost:9011/management/healthcheck

200 http://localhost:9012/management/healthcheck

mbarton commented 5 years ago

Great! Glad you've got something in the browser now.

The Grid has a separate microservice for authentication (the auth project). I think the browser is trying to authenticate against the main media-api service instead.

It should be redirecting to media-auth.{{YOUR_DOMAIN}}. The browser gets the authentication link injected to follow. It's defined on the server in Services.scala.

If you curl the main page of the Grid, what value do you have in the HTML in the <link rel="reauth-uri" tag?

mbarton commented 5 years ago

Re the S3 permissions errors, I think you will be able to run the Grid fine with the default permissions. We should look at how external users could grant users permission however.

The second S3 error (Unexpected CSV headers) is for the usage service. It's very Guardian-specific, relying on our content API, so you're fine to ignore it or avoid running the usage project altogether.

aafonsobbc commented 5 years ago

Hi @mbarton , thanks for coming back!!!

Redirection for auth is going to

https://media-auth.example.com/login?redirectUri=https%3A%2F%2Fmedia.example.com%2Fsearch

I can't run the grid right now within the proxy config we have here, I will try to find out either the right config or work without proxies, anyway, answering your question (the auth redirection) is the above and it is resolved to the same host (localhost)... any idea why the GET could be failing then?

Thanks so much for your help (also really welcome suggestions to configure proxies within The Grid)

Alfonso Afonso

aafonsobbc commented 5 years ago

No worries about proxies, added config in .sbtopts 👍

aafonsobbc commented 5 years ago

Hi @mbarton I have managed to stop the redirection and also got this screenshoot... anyway and answering your previous question, redirection link is set the above, not sure where or what to look at now :(

screen shot 2018-12-03 at 15 12 32

Thanks Alfonso Afonso

aafonsobbc commented 5 years ago

Hi again, @mbarton

Confirmed auth module is working, see below the error I got from auth/logs/application.log... not sure if this is related to the GoogleAPI key (maybe I am missing some permission??) or about AWS... or maybe is just another thing.

Any thoughts?? Thanks!!

Alfonso Afonso

2018-12-03 16:29:41,744 - [WARN] - from application in play-dev-mode-akka.actor.default-dispatcher-6 markers=
[unauthorized] Responding with error status 401, Not authenticated

2018-12-03 16:29:41,806 - [WARN] - from application in play-dev-mode-akka.actor.default-dispatcher-6 markers=
[session-expired] Responding with error status 419, Session expired, required to log in again

2018-12-03 16:29:41,807 - [WARN] - from application in play-dev-mode-akka.actor.default-dispatcher-6 markers=
[invalid-api-key] Responding with error status 401, Invalid API key provided

2018-12-03 16:29:41,900 - [INFO] - from ROOT in play-dev-mode-akka.actor.default-dispatcher-6 markers=
Kinesis logging disabled in DEV

2018-12-03 16:29:41,946 - [INFO] - from play.api.Play in play-dev-mode-akka.actor.default-dispatcher-6 markers=
Application started (Dev)

2018-12-03 16:29:42,097 - [WARN] - from application in application-akka.actor.default-dispatcher-2 markers=
[forbidden] Responding with error status 403, Unauthorized - the API key is not allowed to perform this operation

2018-12-03 16:29:42,117 - [INFO] - from request in application-akka.actor.default-dispatcher-5 markers={duration=34, method=GET, origin=0:0:0:0:0:0:0:1, status=200, referrer=}
0:0:0:0:0:0:0:1 - "GET /management/healthcheck HTTP/1.1" 200 0 "" 34ms

2018-12-03 16:29:46,968 - [INFO] - from request in application-akka.actor.default-dispatcher-5 markers={duration=2, method=GET, origin=0:0:0:0:0:0:0:1, status=200, referrer=}
0:0:0:0:0:0:0:1 - "GET /management/healthcheck HTTP/1.1" 200 0 "" 2ms
mbarton commented 5 years ago

Hmm. I don't think the auth service logs anything when someone logs in succesfully. The [unauthorized] and [session-expired] logs I would expect as normal when logging in for the first time (or after a short period).

Would you be able to post the HTTP requests you see in the browser network inspector? That'll help diagnose which bit of the Grid is not working.

For the API key error, various parts of the Grid use them to communicate with other parts (eg the Cropper, S3 Watcher, Reaper.

It looks like the quick-start documentation doesn't mention setting up these API keys, which could explain the error. You can set them up following the instructions here. I think you'll need at a minumum a key where the filename is cropper-${RANDOM_STRING} so that you can crop images.

aafonsobbc commented 5 years ago

Hi @mbarton

Key tested and added to bucket (if I hit the api.media.example.com url with the key I got a 200 with the response)

However, not sure now how to proceed as error when hitting the media.example.com from a browser is the same, I have checked auth/logs folder and there are no files now (probably the log I copied were there before the GoogleAPI config)

The HTTP requests is https://media-auth.example.com/login?redirectUri=https%3A%2F%2Fmedia.example.com%2Fsearch

Request headers:

GET /login?redirectUri=https%3A%2F%2Fmedia.example.com%2Fsearch HTTP/1.1
Host: media-auth.example.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://media.example.com/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,es;q=0.8,cy;q=0.7
Cookie: _ga=GA1.2.1489982402.1543589962; PLAY_SESSION=eyJhbGciOiJIUzI1NiJ9.eyJkYXRhIjp7ImNzcmZUb2tlbiI6ImFkZGQ3NGUxYTU3YTg4Yjg1MjhkYjdlNmYxYWJkNWY5MDNmNDczZTItMTU0MzU5NDUzMDc4MC00YmYxMzQ0OTgyNTdmNzVjYjY0Mzk0ODUifSwibmJmIjoxNTQzNTk0NTMwLCJpYXQiOjE1NDM1OTQ1MzB9.pbJDl92t_IWz2WwK9jnazHXviL9BZPGeT3GRUbfRov8; _gid=GA1.2.1939343828.1543846804; _gat=1

Response Headers:

HTTP/1.1 404 Not Found
Server: nginx/1.15.6
Date: Tue, 04 Dec 2018 16:00:01 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 1433
Connection: keep-alive
Vary: Accept-Encoding,Origin
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
X-Frame-Options: DENY
Content-Encoding: gzip
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self'
X-Permitted-Cross-Domain-Policies: master-only

And the response body is:

Action Not Found
For request 'GET /login?redirectUri=https%3A%2F%2Fmedia.example.com%2Fsearch'

These routes have been tried, in this order:
1GET/controllers.MediaApi.index
2GET/images/metadata/$field<[^/]+>controllers.SuggestionController.metadataSearch(field:String, q:Option[String])
3GET/images/edits/$field<[^/]+>controllers.SuggestionController.editsSearch(field:String, q:Option[String])
4GET/images/aggregations/date/$field<[^/]+>controllers.AggregationController.dateHistogram(field:String, q:Option[String])
5GET/images/$id<[^/]+>controllers.MediaApi.getImage(id:String)
6GET/images/$id<[^/]+>/fileMetadatacontrollers.MediaApi.getImageFileMetadata(id:String)
7GET/images/$imageId<[^/]+>/export/$exportId<[^/]+>controllers.MediaApi.getImageExport(imageId:String, exportId:String)
8GET/images/$imageId<[^/]+>/exportcontrollers.MediaApi.getImageExports(imageId:String)
9GET/images/$imageId<[^/]+>/downloadcontrollers.MediaApi.downloadOriginalImage(imageId:String)
10POST/images/$id<[^/]+>/reindexcontrollers.MediaApi.reindexImage(id:String)
11DELETE/images/$id<[^/]+>controllers.MediaApi.deleteImage(id:String)
12GET/imagescontrollers.MediaApi.imageSearch
13GET/suggest/metadata/creditcontrollers.SuggestionController.suggestMetadataCredit(q:Option[String], size:Option[Int])
14GET/suggest/metadata/photoshootcontrollers.SuggestionController.suggestPhotoshoot(q:Option[String], size:Option[Int])
15GET/usage/supplierscontrollers.UsageController.bySupplier
16GET/usage/suppliers/$id<[^/]+>controllers.UsageController.forSupplier(id:String)
17GET/usage/quotascontrollers.UsageController.quotas
18GET/usage/quotas/$id<[^/]+>controllers.UsageController.quotaForImage(id:String)
19GET/management/healthcheckcontrollers.HealthCheck.healthCheck
20GET/management/manifestcom.gu.mediaservice.lib.management.Management.manifest
21GET/robots.txtcom.gu.mediaservice.lib.management.Management.disallowRobots

I hope it helps to find out what I am missing here.

Thanks!!

Alfonso Afonso

mbarton commented 5 years ago

Yep, looks like an nginx configuration issue. The request to media-auth.example.com is being served from the media-api service.

This is my nginx config I'm using locally: https://gist.github.com/mbarton/2120c3234a3e16058f591a719ebc080b

api.media.local.dev-gutools.co.uk is proxying to http://localhost:9001 and media-auth.local.dev-gutools.co.uk proxies to http://localhost:9011.

I suspect for your config media-auth.local.dev-gutools.co.uk is proxying to http://localhost:9001 instead?

aafonsobbc commented 5 years ago

Hi @mbarton

I was writing another comment just saying something similar to above, a nginx config error (port error actually).

Thanks for your help!!

I am having a 403 now, not sure if I have to add users somewhere or is part of the Google API config... will keep you posted (and if you already have experienced something similar, happy to get any input from you)

Alfonso Afonso

aafonsobbc commented 5 years ago

Hi @mbarton

I have made it work so far but still have the

Error loading results: {"uri":"https://api.media.example.com","body":null,"status":-1,"headers":{"Location":null,"Content-Type":null}}

In the browser (Safari and Firefox). I am wondering if I should add the header X-Gu-Media-Key somewhere in the configuration or in the browser?? to make the api-media request (without this header it won't work, if I understood it right)

Thanks

Alfonso Afonso

mbarton commented 5 years ago

Hmm strange. The browser doesn't need an X-Gu-Media-Key header, that's just for service to service communication on the server side.

Could you post the network requests the browser is making? I'm wondering if it makes it as far as the oauthCallback or is failing before? It could also be setting the cookie fine but then failing to read it afterwards.

Sorry this has been such a protracted process!

aafonsobbc commented 5 years ago

Hi @mbarton morning,

Yep, is about the OAUTH, I was redirected now to sign in Google and after using my own account, got a 403 ... I am not fluent on this Google API, I have probably missed the user config (who can use the tool) somewhere, could it be? Reading the doc I see that there is a user class that can be used to test the Auth module, just wondering it that's the root cause and where should I add the emails allowed to work with the tool

Thanks!!

Alfonso Afonso

mbarton commented 5 years ago

Aaaah I think I've found the code causing the issue. There's a function in Authentication called validateUser and it is currently hard-coded to guardian.co.uk!

final override def validateUser(authedUser: AuthenticatedUser): Boolean = {
    authedUser.user.email.endsWith("@guardian.co.uk") && authedUser.multiFactor
}

If you edit that to change @guardian.co.uk to the email domain you are using and remove the multi-factor check you should be able to log in.

I am preparing a PR that will allow the Grid to use a Cognito pool rather than through the Google OAuth so I can address this problem as part of that. In the meantime feel free to raise a PR to make this a configuration option or similar to avoid blocking your PoC.