NoSuchElementException: key not found: domain.root

TRDRakesh commented 5 years ago

When I install all dependencies and run Grid I get this error in browser: Please Help.

Unexpected exception
NoSuchElementException: key not found: domain.root

No source available, here is the exception stack trace:
->java.util.NoSuchElementException: key not found: domain.root
     scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:101)
     scala.collection.immutable.Map$EmptyMap$.apply(Map.scala:99)
     com.gu.mediaservice.lib.config.CommonConfig.domainRoot(CommonConfig.scala:54)
     com.gu.mediaservice.lib.config.CommonConfig.domainRoot$(CommonConfig.scala:54)
     lib.ThrallConfig.domainRoot$lzycompute(ThrallConfig.scala:8)
     lib.ThrallConfig.domainRoot(ThrallConfig.scala:8)
     com.gu.mediaservice.lib.config.CommonConfig.services(CommonConfig.scala:66)
     com.gu.mediaservice.lib.config.CommonConfig.services$(CommonConfig.scala:66)
     lib.ThrallConfig.services$lzycompute(ThrallConfig.scala:8)
     lib.ThrallConfig.services(ThrallConfig.scala:8)
     com.gu.mediaservice.lib.auth.Authentication.buildPandaSettings(Authentication.scala:86)
     com.gu.mediaservice.lib.auth.Authentication.panDomainSettings$lzycompute(Authentication.scala:43)
     com.gu.mediaservice.lib.auth.Authentication.panDomainSettings(Authentication.scala:43)
     com.gu.pandomainauth.action.AuthActions.settings(Actions.scala:26)
     com.gu.pandomainauth.action.AuthActions.$init$(Actions.scala:71)
     com.gu.mediaservice.lib.auth.Authentication.<init>(Authentication.scala:26)
     com.gu.mediaservice.lib.play.GridComponents.<init>(GridComponents.scala:35)
     ThrallComponents.<init>(ThrallComponents.scala:8)
     AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
     AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
     com.gu.mediaservice.lib.play.GridAppLoader.load(GridAppLoader.scala:11)
     play.core.server.DevServerStart$$anon$1.$anonfun$reload$3(DevServerStart.scala:174)
     play.utils.Threads$.withContextClassLoader(Threads.scala:21)
     play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:171)
     play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
     play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:241)
     play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:138)
     akka.stream.impl.fusing.MapAsyncUnordered$$anon$26.onPush(Ops.scala:1304)
     akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519)
     akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:482)
     akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:378)
     akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:588)
     akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:472)
     akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:563)
     akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:745)
     akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:760)
     akka.actor.Actor.aroundReceive(Actor.scala:517)
     akka.actor.Actor.aroundReceive$(Actor.scala:515)
     akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:670)
     akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
     akka.actor.ActorCell.invoke(ActorCell.scala:557)
     akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
     akka.dispatch.Mailbox.run(Mailbox.scala:225)
     akka.dispatch.Mailbox.exec(Mailbox.scala:235)
     akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
     akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
     akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
     akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

cstrd commented 5 years ago

I get this error in terminal:

Exception thrown when trying to create FileInputStream: java.io.FileNotFoundException: /etc/gu/thrall.properties (No such file or directory)

cstrd commented 5 years ago

@akash1810 Can you please look into the issue

akash1810 commented 5 years ago

Hello,

Thank you for trying Grid and apologies for the delayed response.

The stack trace provided is very helpful! This section:

com.gu.mediaservice.lib.config.CommonConfig.domainRoot(CommonConfig.scala:54)
com.gu.mediaservice.lib.config.CommonConfig.domainRoot$(CommonConfig.scala:54)
lib.ThrallConfig.domainRoot$lzycompute(ThrallConfig.scala:8)
lib.ThrallConfig.domainRoot(ThrallConfig.scala:8)

suggests config for the Thrall app is missing the domain.root value.

Could you check the contents of the file /etc/gu/thrall.properties? It should match this (customised with your values). If not, please follow the instructions to generate config again and retry?

cstrd commented 5 years ago

My service-config.js file:

function getThrallConfig(config) {
    return stripMargin`
        |domain.root=${config.domainRoot}
        |aws.region=${config.aws.region}
        |auth.keystore.bucket=${config.stackProps.KeyBucket}
        |s3.image.bucket=${config.stackProps.ImageBucket}
        |s3.thumb.bucket=${config.stackProps.ThumbBucket}
        |persistence.identifier=picdarUrn
        |es.index.aliases.write=writeAlias
        |es.index.aliases.read=readAlias
        |indexed.image.sns.topic.arn=${config.stackProps.IndexedImageTopicArn}
        |es6.url=${config.es6.url}
        |es6.cluster=${config.es6.cluster}
        |es6.shards=${config.es6.shards}
        |es6.replicas=${config.es6.replicas}
        |thrall.kinesis.stream.name=${config.stackProps.ThrallMessageQueue}
        |`;
}

cstrd commented 5 years ago

The directory /etc/gu/ is empty

akash1810 commented 5 years ago

Hi,

service-config.js only defines the shape of the resulting file; it is used here to write files to /etc/gu. /etc/gu being empty suggests that you may not have run the generate-dot-properties script, please follow the instructions and let us know how you get on.

cstrd commented 5 years ago

Hey @akash1810 I followed your steps, now I am getting this error:

Unexpected exception
IllegalStateException: Unable to download public key

No source available, here is the exception stack trace:
->java.lang.IllegalStateException: Unable to download public key
     com.gu.pandomainauth.Settings$.errorToThrowable(Settings.scala:55)
     com.gu.pandomainauth.PanDomainAuthSettingsRefresher.<init>(PanDomainAuthSettingsRefresher.scala:35)
     com.gu.mediaservice.lib.auth.Authentication.buildPandaSettings(Authentication.scala:85)
     com.gu.mediaservice.lib.auth.Authentication.panDomainSettings$lzycompute(Authentication.scala:43)
     com.gu.mediaservice.lib.auth.Authentication.panDomainSettings(Authentication.scala:43)
     com.gu.pandomainauth.action.AuthActions.settings(Actions.scala:26)
     com.gu.pandomainauth.action.AuthActions.$init$(Actions.scala:71)
     com.gu.mediaservice.lib.auth.Authentication.<init>(Authentication.scala:26)
     com.gu.mediaservice.lib.play.GridComponents.<init>(GridComponents.scala:35)
     ThrallComponents.<init>(ThrallComponents.scala:8)
     AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
     AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3)
     com.gu.mediaservice.lib.play.GridAppLoader.load(GridAppLoader.scala:11)
     play.core.server.DevServerStart$$anon$1.$anonfun$reload$3(DevServerStart.scala:174)
     play.utils.Threads$.withContextClassLoader(Threads.scala:21)
     play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:171)
     play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
     play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:241)
     play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:138)
     akka.stream.impl.fusing.MapAsyncUnordered$$anon$26.onPush(Ops.scala:1304)
     akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519)
     akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:482)
     akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:378)
     akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:588)
     akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:472)
     akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:563)
     akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:745)
     akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:760)
     akka.actor.Actor.aroundReceive(Actor.scala:517)
     akka.actor.Actor.aroundReceive$(Actor.scala:515)
     akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:670)
     akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
     akka.actor.ActorCell.invoke(ActorCell.scala:557)
     akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
     akka.dispatch.Mailbox.run(Mailbox.scala:225)
     akka.dispatch.Mailbox.exec(Mailbox.scala:235)
     akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
     akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
     akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
     akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

cstrd commented 5 years ago

thrall.properties file contents:

domain.root=
aws.region=us-east-1
auth.keystore.bucket=media-service-dev-keybucket-q8srzkai5zqw
s3.image.bucket=media-service-dev-imagebucket-k5g2o96bh52l
s3.thumb.bucket=media-service-dev-thumbbucket-1kpybr3hm4jfs
persistence.identifier=picdarUrn
es.index.aliases.write=writeAlias
es.index.aliases.read=readAlias
indexed.image.sns.topic.arn=arn:aws:sns:us-east-1:762368696241:media-service-DEV-IndexedImageTopic-1PA1HKLQORF7H
es6.url=http://localhost:9002
es6.cluster=media-service
es6.shards=1
es6.replicas=0
thrall.kinesis.stream.name=media-service-DEV-ThrallMessageQueue-1JWXBSM0BHVVV

cstrd commented 5 years ago

@akash1810 Also, I have installed nvm correctly, but when I run GRID. In terminal it shows:

nvm not found  NVM is required to run this project
Install it from https://github.com/creationix/nvm#installation

But it runs after these lines

akash1810 commented 5 years ago

Looks like https://github.com/guardian/grid/issues/2586 is addressing https://github.com/guardian/grid/issues/2582#issuecomment-500355278.

Your thrall.properties file is still missing the domain.root value, which suggests the initial config is incorrect.

Regarding NVM: it seems the check for nvm use quite simple, in that it checks for a file on disk.

Could you run nvm use before executing dev-start.sh? nvm use will instruct nvm to switch the version of node that it's using.

Regarding:

But it runs after these lines

Grid is doing the wrong thing here! https://github.com/guardian/grid/pull/2589 should resolve that.

cstrd commented 5 years ago

I am running GRID in my local machine, what should I put in domain.root value,Should I put "localhost:9002"?

akash1810 commented 5 years ago

I am running GRID in my local machine, what should I put in domain.root value,Should I put "localhost:9002"?

Ultimately, the domain root is used here. That is, it is used to craft the endpoints for the various micro services and its value determines how you access Grid in the browser.

For example, a domain root of grid.local means the Kahuna service will run on https://media.grid.local.

If the domain does not exist in DNS, you'll need to edit your hosts file to fake a DNS entry. For example, if you use grid.local you'll need add the following to your hosts file (one entry per micro-service).

127.0.0.1               media.grid.local
127.0.0.1               api.media.grid.local
127.0.0.1               loader.media.grid.local
127.0.0.1               cropper.media.grid.local
127.0.0.1               media-metadata.grid.local
127.0.0.1               media-imgops.grid.local
127.0.0.1               media-usage.grid.local
127.0.0.1               media-collections.grid.local
127.0.0.1               media-auth.grid.local
127.0.0.1               media-leases.grid.local

You will also want to add a domain-root value to nginx-mappings.yaml with the same value to ensure dev-nginx generates the correct nginx config.

We use the value local.dev-gutools.co.uk which has got a wildcard DNS resolution in place, meaning we don't edit the hosts file. This may be an option for you too - you can change it at a later date if you want a more personal domain.

cstrd commented 5 years ago

Here is my nginx-mappings.yml file:

domain-root: grid.local
name: media-service
mappings:
  - prefix: api.media
    port: 9001
  - prefix: loader.media
    port: 9003
    client_max_body_size: 20m
  - prefix: media
    port: 9005
  - prefix: cropper.media
    port: 9006
  - prefix: media-metadata
    port: 9007
  - prefix: media-imgops
    port: 9008
  - prefix: media-usage
    port: 9009
  - prefix: media-collections
    port: 9010
  - prefix: media-auth
    port: 9011
  - prefix: media-leases
    port: 9012
  - prefix: es.media
    port: 9200

I am still getting same error, I am running dev-start.sh and going to localhost:9002

akash1810 commented 5 years ago

Can I confirm what we've done so far:

Installed all dependencies
Created the config file, setting all values, including domain.root
Ran generate-dot-properties to write Grid's config to the /etc/gu directory
- Each file in /etc/gu should also now have a domain.root value
Updated nginx-mappings.yaml, adding domain-root that matches the one specified in the config file above
Ran dev-nginx setup-app nginx-mappings.yaml to use the new nginx-mappings.yaml file
Setup panda
Updated the hosts file if necessary

Could you confirm if that's correct?

If yes, then running ./dev-start.sh afterwards gives an error. Could you share a stacktrace please?

cstrd commented 5 years ago

Hello @akash1810 , I really appreciate you for giving your time into our issue. I have done all the steps, except setting up panda. I am running GRID in my local machine, for the Panda configuration which domain do I need to use and what callback urls do I need to add. Thanks!

cstrd commented 5 years ago

Hello @akash1810 , For setting up Panda, how to integrate it to grid, there is an example code given in the repository but where do I add the files in grid folder. Thanks!

cstrd commented 5 years ago

@akash1810 I have followed your instructions and here is what my /etc/gu and config.json5 contains. and cognito-settingsbucket-sycde3ha71md bucket in s3 was generated by cognito stack and here are the files in it file1 file2 These files were generated by cognito stack in s3 bucket using this command ./cognito/generate-settings.sh ${CLOUDFORMATION_STACK} ${REGION} Also I have added, libraryDependencies += "com.gu" %% "pan-domain-auth-verification" % "0.2.13"; line to build.sbt in grid. and in the Cognito config, I put CALLBACK URL as http://localhost I have done till 3rd step from the docs. Here is what I get when I run ./dev-start.sh and go to localhost:9002

`IllegalStateException: Unable to download public key

No source available, here is the exception stack trace: ->java.lang.IllegalStateException: Unable to download public key com.gu.pandomainauth.Settings$.errorToThrowable(Settings.scala:55) com.gu.pandomainauth.PanDomainAuthSettingsRefresher.(PanDomainAuthSettingsRefresher.scala:35) com.gu.mediaservice.lib.auth.Authentication.buildPandaSettings(Authentication.scala:85) com.gu.mediaservice.lib.auth.Authentication.panDomainSettings$lzycompute(Authentication.scala:43) com.gu.mediaservice.lib.auth.Authentication.panDomainSettings(Authentication.scala:43) com.gu.pandomainauth.action.AuthActions.settings(Actions.scala:26) com.gu.pandomainauth.action.AuthActions.$init$(Actions.scala:71) com.gu.mediaservice.lib.auth.Authentication.(Authentication.scala:26) com.gu.mediaservice.lib.play.GridComponents.(GridComponents.scala:35) ThrallComponents.(ThrallComponents.scala:8) AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3) AppLoader$$anonfun$$lessinit$greater$1.apply(AppLoader.scala:3) com.gu.mediaservice.lib.play.GridAppLoader.load(GridAppLoader.scala:11) play.core.server.DevServerStart$$anon$1.$anonfun$reload$3(DevServerStart.scala:174) play.utils.Threads$.withContextClassLoader(Threads.scala:21) play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:171) play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124) play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:241) play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:138) akka.stream.impl.fusing.MapAsyncUnordered$$anon$26.onPush(Ops.scala:1304) akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519) akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:482) akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:378) akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:588) akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:472) akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:563) akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:745) akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:760) akka.actor.Actor.aroundReceive(Actor.scala:517) akka.actor.Actor.aroundReceive$(Actor.scala:515) akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:670) akka.actor.ActorCell.receiveMessage(ActorCell.scala:588) akka.actor.ActorCell.invoke(ActorCell.scala:557) akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) akka.dispatch.Mailbox.run(Mailbox.scala:225) akka.dispatch.Mailbox.exec(Mailbox.scala:235) akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)`

Can you look into it, what am I missing. Thanks

cstrd commented 5 years ago

@akash1810 Sir can you please have a look at it! Thanks

akash1810 commented 5 years ago

Closing as the original issue (missing domain.root) has been resolved and pan-domain setup issue is being discussed on https://github.com/guardian/grid/issues/2586.

akash1810 commented 4 years ago

Quite a late response... #2927 looks like it'll help here. Would be interested to get your feedback on it.

guardian / grid

NoSuchElementException: key not found: domain.root #2582