search

Teevity / ice

AWS Usage Tool
2.86k stars 435 forks source link

readme lacks installation details #4

Closed ralph-tice closed 11 years ago

ralph-tice commented 11 years ago

I've installed tomcat, grails, and ivy via brew on MacOSX. I copied build-template.xml to build.xml when I run: GRAILS_HOME=/usr/local ant I get:

Buildfile: /Users/ralphtice/personal/ice/build.xml
     [echo] /usr/local/bin/grails

war:
     [exec] Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/groovy/grails/cli/support/GrailsStarter
     [exec] Caused by: java.lang.ClassNotFoundException: org.codehaus.groovy.grails.cli.support.GrailsStarter
     [exec]     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
     [exec]     at java.security.AccessController.doPrivileged(Native Method)
     [exec]     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
     [exec]     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
     [exec]     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
     [exec]     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

BUILD FAILED
/Users/ralphtice/personal/ice/build.xml:46:

I invoke ivy and get:

Caused by: java.io.FileNotFoundException: JAR entry org/apache/ivy/core/settings/ivy-configurations.xml not found in /Users/ralphtice/homebrew/Cellar/ivy/2.3.0/libexec/ivy-2.3.0.jar

When I remove this reference in ivy.xml everything breaks:

Exception in thread "main" java.text.ParseException: Problem occurred while parsing ivy file: Cannot add dependency 'javax.mail#mail;1.4.1' to configuration 'compile' of module netflix#ice;working@Ralphs-MacBook-Air.local because this configuration doesn't exist! in file:/Users/ralphtice/personal/ice/ivy.xml
rayrod2030 commented 11 years ago

I agree. I managed to get the application running via grails 1.3.7 on ubuntu 12.04 but I don't even want to post the steps because there were so many hacks. I'm sure there is a better way to just have grails create a deployable war that you can just drop into tomcat but I'm not familiar with grails deployments and configuration. I agree the installation/setup instructions in the README could use some help.

Jithinxavi commented 11 years ago

I am new to Grails framework, Can you please post details installation steps in README file , I am using Ubuntu12.04LTS with grails-2.1.1.

Please let me know how to run this step 4.Get all libraries listed in ivy.xml

ralph-tice commented 11 years ago

i found these two commands very helpful grails war and grails run-app Alternatively, importing the project into IntelliJ's IDE made it very easy to build/run the app. Currently, the problem I have is that I don't know how to get the app to find my ice.properties file

rayrod2030 commented 11 years ago

@ralph-tice Place the ice.properties file in grails-app/conf

fangji commented 11 years ago

I will get the build cleaned up today.

Thanks -Fang

rayrod2030 commented 11 years ago

Also here is the list of jars I had to cobble together from various sources and place under lib/ to get the app to fire up.

ant-1.7.0.jar guava-14.0.1.jar httpcore-4.1.jar jcommon-1.0.16.jar joda-time-2.0.jar org.json-chargebee-1.0.jar wstx-asl-3.2.9.jar aws-java-sdk-1.4.4.1.jar httpclient-4.1.jar javacsv-2.0.jar jfreechart-1.0.13.jar mail-1.4.1.jar poi-3.7.jar

fangji commented 11 years ago

I update the ice project to be built on grails 2.2.1. Now to start running ice, you can following the following steps (copied form readme):

4.1 Pull the project

4.2 Run grails wrapper from the project root directory. This step will pull all necessary jar from maven central.

4.3 Make sure ice.properties is added to directory src/java

4.4 From project root directory, run ./grailsw run-app. Note you may need to add system properties like ./grailsw -Dice.s3AccessKeyId= -Dice.s3SecretKey= run-app.

unnisathyarajan commented 11 years ago

hi fangi, @fangji : I attempted to make an installation of the application yesterday 19-JUN-13 but couldnt complete it...Today ie 20-JUN-13 I see that some steps are being removed like 1.No tomcat installation 2.Get all library files in ivy.xml etc wat does this exactly mean......It would be great if you could publish a more descriptive steps on installation of ICE either here or on any blog post.....

@rayrod2030 : u got the app running thats great......wish u change ur mind and post the steps :-)

ralph-tice commented 11 years ago

@cottonmouth00 Steps 4.1 through 4.4 cover bootstrapping the project in the readme as well as Fang's last comment.

@fangji Thank you for your help and for bumping to grails 2.2.1! I stepped through every step much more easily this time. I can't use grailsw because of these errors:

$ ./grailsw
Error opening zip file or JAR manifest missing : wrapper/springloaded-core-1.1.1.jar
Error occurred during initialization of VM
agent library failed to init: instrument

However, when I invoke grails directly everything appears to work. When I try to hit the application at localhost:8080/ice however I get this stacktrace:

| Server running. Browse to http://localhost:8080/ice
2013-06-20 08:39:41,810 [http-bio-8080-exec-4] INFO  [localhost].[/ice]  - Initializing Spring FrameworkServlet 'gsp'
2013-06-20 08:39:41,811 [http-bio-8080-exec-4] INFO  [localhost].[/ice]  - GSP servlet initialized
| Error 2013-06-20 08:39:42,256 [http-bio-8080-exec-4] ERROR errors.GrailsExceptionResolver  - IOException occurred when processing request: [GET] /ice/
Stream closed. Stacktrace follows:
Message: Stream closed
   Line | Method
->> 187 | doCall  in com.netflix.ice.DashboardController$_closure8
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|    68 | doCall  in com.netflix.ice.DashboardController$_closure1
|   895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run     in     ''
^   680 | run . . in java.lang.Thread
| Error 2013-06-20 08:39:42,328 [http-bio-8080-exec-4] ERROR [/ice].[jsp]  - Servlet.service() for servlet jsp threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread
| Error 2013-06-20 08:39:42,336 [http-bio-8080-exec-4] ERROR errors.GrailsExceptionResolver  - ServletException occurred when processing request: [GET] /ice/
File "/WEB-INF/grails-app/views/error.jsp" not found. Stacktrace follows:
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread
| Error 2013-06-20 08:39:42,367 [http-bio-8080-exec-4] ERROR [/ice].[jsp]  - Servlet.service() for servlet jsp threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread
| Error 2013-06-20 08:39:42,379 [http-bio-8080-exec-4] ERROR [/ice].[grails]  - Servlet.service() for servlet grails threw exception
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread
| Error 2013-06-20 08:39:42,382 [http-bio-8080-exec-4] ERROR [/ice].[gsp]  - Servlet.service() for servlet [gsp] in context with path [/ice] threw exception [File "/WEB-INF/grails-app/views/error.jsp" not found] with root cause
Message: File "/WEB-INF/grails-app/views/error.jsp" not found
   Line | Method
->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   379 | serviceJspFile        in     ''
|   334 | service . . . . . . . in     ''
|   895 | runTask               in java.util.concurrent.ThreadPoolExecutor$Worker
|   918 | run . . . . . . . . . in     ''
^   680 | run                   in java.lang.Thread

I tried to look in the commit history to see what changed but it looks like you might have rebased some commits because I only see one commit now in github.

fangji commented 11 years ago

@cottonmouth00 As @ralph-tice mentioned, please follow steps 4.1 through 4.4 cover bootstrapping the project in the readme. Please also make sure you have Grails 2.2.1 installed and have GRAILS_HOME and JAVA_HOME set.

@ralph-tice Fromt the running grailsw error, it looks like grails wrapper wasn't set up correctly. Did you run grails wrapper and see a wrapper directory was added to your project root?

ralph-tice commented 11 years ago

@fangji aha!! I didn't realize grails wrapper was different from grailsw. ./grailsw -Dice.s3AccessKeyId=$AWS_ACCESS_KEY_ID -Dice.s3SecretKey=$AWS_SECRET_KEY run-app works now, but I still get the same runtime exceptions trying to load http://localhost:8080/ice as in my previous comment.

fangji commented 11 years ago

mm...

Try http://localhost:8080/ice/dashboard/summary instead.

ralph-tice commented 11 years ago

OK, that works. I guess the index route is missing?

Jithinxavi commented 11 years ago

@fangji Thanks for the README update,

I have followed your readme and installed 

1. Grails version: 2.1.1
2. Java - java version "1.6.0_27"
3. Uninstall running tomcat server.

Pull the latest code from repository and ran

grails wrapper

Wrapper installed successfully

Created ice-properties file like this and copied to grails-app/conf

ice.billing_s3bucketname= ice.billing_s3bucketprefix=ice ice.processor=true ice.processor.localDir=/opt/ice/ice_processor ice.work_s3bucketname= ice.work_s3bucketprefix=iceworker ice.s3AccessKeyId= ice.s3SecretKey= ice.startmillis=1370044800 ice.account1= ice.account2= ice.account3= ice.reader=true ice.processor.localDir=/opt/ice/ice_reader ice.s3AccessKeyId= ice.s3SecretKey= ice.startmillis=1370044800 ice.account1= ice.companyName=

Then I ran this command - ./grailsw -Dice.s3AccessKeyId=$AWS_ACCESS_KEY_ID -Dice.s3SecretKey=$AWS_SECRET_KEY run-app

I am getting this error

| Running Grails application 2013-06-20 21:47:19,401 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"] 2013-06-20 21:47:19,519 [main] INFO core.StandardService - Starting service Tomcat 2013-06-20 21:47:19,519 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30 2013-06-20 21:47:19,828 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found 2013-06-20 21:47:20,280 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext 2013-06-20 21:47:27,191 [localhost-startStop-1] WARN config.ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/root/.grails/ivy-cache/net.sf.ehcache/ehcache-core/jars/ehcache-core-2.4.6.jar!/ehcache-failsafe.xml 2013-06-20 21:47:27,406 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.UpdateTimestampsCache]; using defaults. 2013-06-20 21:47:27,434 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.StandardQueryCache]; using defaults. 2013-06-20 21:47:28,981 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@27f33aef: startup date [Thu Jun 20 21:47:26 UTC 2013]; parent: Root WebApplicationContext 2013-06-20 21:47:32,403 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded. 2013-06-20 21:47:32,651 [localhost-startStop-1] INFO BootStrap - Starting ice... 2013-06-20 21:47:34,883 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up... 2013-06-20 21:47:34,895 [localhost-startStop-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started... 2013-06-20 21:47:34,895 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting... | Error 2013-06-20 21:47:34,933 [localhost-startStop-1] ERROR BootStrap - Startup failed Message: null Line | Method ->> 411 | put in java.util.Hashtable

| 160 | setProperty in java.util.Properties | 145 | doCall . . . . . . . . . . . . . in BootStrap$_closure1 | 301 | evaluateEnvironmentSpecificBlock in grails.util.Environment | 294 | executeForEnvironment . . . . . in '' | 270 | executeForCurrentEnvironment in '' | 334 | innerRun . . . . . . . . . . . . in java.util.concurrent.FutureTask$Sync | 166 | run in java.util.concurrent.FutureTask | 1146 | runWorker . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor | 615 | run in java.util.concurrent.ThreadPoolExecutor$Worker ^ 679 | run . . . . . . . . . . . . . . in java.lang.Thread

Please advise me to solve this problem

fangji commented 11 years ago

@Jithinxavi you have a lot empty properties in your ice.properties file. did you remove them? You have reader enabled, but didn't specify ice.reader.localDir

For the first time, I strongly suggest you only enable processor first.

fangji commented 11 years ago

@Jithinxavi Please note the following 3 things:

  1. you don't need to set ice.s3AccessKeyId and ice.s3SecretKey in ice.properties file
  2. please don't include the real access key and secret key in the issue comment any more.
  3. The error happens at BootStrap$_closure1.doCall(BootStrap.groovy:145): properties.setProperty(IceOptions.LOCAL_DIR, prop.getProperty("ice.reader.localDir")); that means you do have ice.reader=true in your ice.properties file. please double check.
unnisathyarajan commented 11 years ago

Hi fangji, i have made a Clean installation from start and this I got this ----

@fangji : command not working for me : ./grailsw -Dice.s3AccessKeyId=A11111111111 -Dice.s3SecretKey=A22222222222 run-app

Script 'Dice.s3AccessKeyId=A11111111111111111' not found, did you mean: 1) CreateService 2) Package 3) GenerateController 4) Interactive 5) CreateMultiProjectBuild_

Please make a selection or enter Q to quit: q

And on using the following command i got the server up but when reached on browser it shows error

./grailsw run-app

Output on Browser : HTTP Status 404 - /ice/WEB-INF/grails-app/views/error.jsp)

Output on Server Terminal : | Error 2013-06-21 07:02:34,353 [http-bio-8080-exec-2] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [GET] /ice/ Cannot get property 'accountService' on null object. Stacktrace follows: Message: Cannot get property 'accountService' on null object Line | Method ->> 80 | doCall in com.netflix.ice.DashboardController$_closure3

| 68 | doCall in com.netflix.ice.DashboardController$_closure1 | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run in '' ^ 662 | run . . in java.lang.Thread

Also please take a look at my ice.properites file . i have not enabled reader as you mentioned earilier.

contents ice.properities file ice.processor=true ice.reader=false ice.reservationCapacityPoller=false ice.reservationPeriod=threeyear ice.reservationUtilization=HEAVY ice.ondemandCostAlertThreshold=250 ice.startmillis=1364774400000 ice.companyName=ABC ice.billing_s3bucketname=billing-test-ice ice.billing_s3bucketprefix=ice/billing/ ice.work_s3bucketname=work_abcice ice.work_s3bucketprefix=work_billing/ ice.processor.localDir=/mnt/ice_processor ice.reader.localDir=/mnt/ice_reader ice.monthlycachesize=12 ice.account.account1=0123456789

fangji commented 11 years ago

@cottonmouth00 You ice.properties file look fine to me. Did you miss '-' in the command? I tried the following and got the same error: ./grailsw Dice.s3AccessKeyId=A11111111111 -Dice.s3SecretKey=A22222222222 run-app | Script 'Dice.s3AccessKeyId=A11111111111' not found, did you mean: 1) CreateService 2) Package 3) RefreshDependencies 4) Interactive 5) CreateScript

Please make a selection or enter Q to quit: q

unnisathyarajan commented 11 years ago

@fangji - Thanks fangi i realised that and executed it already and now stuck here ! ... please take a look -

./grailsw run-app -Dice.s3AccessKeyId=A11111111111 -Dice.s3SecretKey=A22222222222 run-app

Output on Browser : HTTP Status 404 - /ice/WEB-INF/grails-app/views/error.jsp)

Output on Server Terminal : | Error 2013-06-21 07:02:34,353 [http-bio-8080-exec-2] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [GET] /ice/ Cannot get property 'accountService' on null object. Stacktrace follows: Message: Cannot get property 'accountService' on null object Line | Method ->> 80 | doCall in com.netflix.ice.DashboardController$_closure3

| 68 | doCall in com.netflix.ice.DashboardController$_closure1 | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run in '' ^ 662 | run . . in java.lang.Thread

fangji commented 11 years ago

@cottonmouth00 It looks like you are trying to access the UI with reader disabled.

unnisathyarajan commented 11 years ago

Hi @fangji : ,

yes the reader wasnt enabled and now its been enabled, and got a new error

| Error 2013-06-23 17:57:49,455 [http-bio-8080-exec-1] ERROR errors.GrailsExceptionResolver - IOException occurred when processing request: [GET] /ice/ Stream closed. Stacktrace follows: Message: Stream closed Line | Method ->> 187 | doCall in com.netflix.ice.DashboardController$_closure8

| 68 | doCall in com.netflix.ice.DashboardController$_closure1 | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run in '' ^ 662 | run . . in java.lang.Thread | Error 2013-06-23 17:57:49,595 [http-bio-8080-exec-1] ERROR [/ice].[jsp] - Servlet.service() for servlet jsp threw exception Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread | Error 2013-06-23 17:57:49,620 [http-bio-8080-exec-1] ERROR errors.GrailsExceptionResolver - ServletException occurred when processing request: [GET] /ice/ File "/WEB-INF/grails-app/views/error.jsp" not found. Stacktrace follows: Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread | Error 2013-06-23 17:57:49,651 [http-bio-8080-exec-1] ERROR [/ice].[jsp] - Servlet.service() for servlet jsp threw exception Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread | Error 2013-06-23 17:57:49,690 [http-bio-8080-exec-1] ERROR [/ice].[grails] - Servlet.service() for servlet grails threw exception Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread | Error 2013-06-23 17:57:49,698 [http-bio-8080-exec-1] ERROR [/ice].[gsp] - Servlet.service() for servlet [gsp] in context with path [/ice] threw exception [File "/WEB-INF/grails-app/views/error.jsp" not found] with root cause Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread 2013-06-23 17:57:50,268 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers... | Error 2013-06-23 17:58:12,412 [http-bio-8080-exec-5] ERROR errors.GrailsExceptionResolver - IOException occurred when processing request: [GET] /ice/ Stream closed. Stacktrace follows: Message: Stream closed Line | Method ->> 187 | doCall in com.netflix.ice.DashboardController$_closure8

| 68 | doCall in com.netflix.ice.DashboardController$_closure1 | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run in '' ^ 662 | run . . in java.lang.Thread | Error 2013-06-23 17:58:12,436 [http-bio-8080-exec-5] ERROR [/ice].[jsp] - Servlet.service() for servlet jsp threw exception Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread | Error 2013-06-23 17:58:12,442 [http-bio-8080-exec-5] ERROR errors.GrailsExceptionResolver - ServletException occurred when processing request: [GET] /ice/ File "/WEB-INF/grails-app/views/error.jsp" not found. Stacktrace follows: Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread | Error 2013-06-23 17:58:12,462 [http-bio-8080-exec-5] ERROR [/ice].[jsp] - Servlet.service() for servlet jsp threw exception Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread | Error 2013-06-23 17:58:12,474 [http-bio-8080-exec-5] ERROR [/ice].[grails] - Servlet.service() for servlet grails threw exception Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread | Error 2013-06-23 17:58:12,484 [http-bio-8080-exec-5] ERROR [/ice].[gsp] - Servlet.service() for servlet [gsp] in context with path [/ice] threw exception [File "/WEB-INF/grails-app/views/error.jsp" not found] with root cause Message: File "/WEB-INF/grails-app/views/error.jsp" not found Line | Method ->> 412 | handleMissingResource in org.apache.jasper.servlet.JspServlet

| 379 | serviceJspFile in '' | 334 | service . . . . . . . in '' | 895 | runTask in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run . . . . . . . . . in '' ^ 662 | run in java.lang.Thread

I have installed ICE on Micro EC2 instance. on accessing the link : http://ec2-107-22-116-234.compute-1.amazonaws.com:8080/ice/

I get this :- {"status":200,"data":[]}

fangji commented 11 years ago

@cottonmouth00 Try this url for now: http://ec2-107-22-116-234.compute-1.amazonaws.com:8080/ice/dashboard/summary

I will fix the index redirecting in a little later.

unnisathyarajan commented 11 years ago

Hi @fangji - finally i am happy to see something on my browser. http://ec2-107-22-116-234.compute-1.amazonaws.com:8080/ice/dashboard/detail

My boxes seems to be empty for account, region, product, operation etc. is this due to permission/accessbility issue if so where can i correct this.

Thanks

fangji commented 11 years ago

@cottonmouth00 @Jithinxavi To verify ice runs correctly, I suggest you do the following:

  1. Make sure you do receive billing files in billing s3 bucket. Make sure the billing files do contain hourly usage line items.
  2. Re-run ice from a clean slate by following instruction in instructions 4.a-4.e in readme.
  3. Make sure Ice writer finishes correctly. If you see errors, post your logs before log "AWS usage processed.".
  4. Make sure you have non-empty output files in your working s3 bucket.
Jithinxavi commented 11 years ago

Hi @fangji ,

Followed your above instructions and still I am facing the same issues, I could see some files are written in my prefix, billing bucket and worker bucket, but unfortunately there is no files in /mnt/ice_processor & /mnt/ice_reader both these directory owned by root, ist any permission related issues? Please advise .

| Compiling 58 source files Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. | Running Grails application 2013-06-24 16:40:17,409 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"] 2013-06-24 16:40:17,443 [main] INFO core.StandardService - Starting service Tomcat 2013-06-24 16:40:17,443 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30 2013-06-24 16:40:17,623 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found 2013-06-24 16:40:18,003 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext 2013-06-24 16:40:23,644 [localhost-startStop-1] WARN config.ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/root/.grails/ivy-cache/net.sf.ehcache/ehcache-core/jars/ehcache-core-2.4.6.jar!/ehcache-failsafe.xml 2013-06-24 16:40:23,956 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.UpdateTimestampsCache]; using defaults. 2013-06-24 16:40:23,985 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.StandardQueryCache]; using defaults. 2013-06-24 16:40:25,497 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@6781a7dc: startup date [Mon Jun 24 16:40:22 UTC 2013]; parent: Root WebApplicationContext 2013-06-24 16:40:29,440 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded. 2013-06-24 16:40:29,705 [localhost-startStop-1] INFO BootStrap - Starting ice... 2013-06-24 16:40:31,629 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up... 2013-06-24 16:40:31,643 [localhost-startStop-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started... 2013-06-24 16:40:31,643 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting... 2013-06-24 16:40:31,881 [localhost-startStop-1] INFO basic.BasicManagers - trying to find new tag group and data managers... 2013-06-24 16:40:32,715 [localhost-startStop-1] INFO basic.BasicManagers - poller thread for com.netflix.ice.basic.BasicManagers started... 2013-06-24 16:40:32,715 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - poller starting... 2013-06-24 16:40:32,724 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers... 2013-06-24 16:40:32,716 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - AWS usage processed. 2013-06-24 16:40:47,306 [localhost-startStop-1] INFO util.SessionIdGenerator - Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [14,532] milliseconds. 2013-06-24 16:40:47,354 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring FrameworkServlet 'grails' 2013-06-24 16:40:47,515 [main] INFO http11.Http11Protocol - Starting ProtocolHandler ["http-bio-8080"] | Server running. Browse to http://localhost:8080/ice 1372092165688 1372092194028 1372092203220 2013-06-24 16:45:32,769 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers...

And How can I run this process in background, if my ssh is closed application is not working , I tried with nohup but no luck.

Thanks, Jithin Xavier

fangji commented 11 years ago

@Jithinxavi from your logs, it looks like BillingFileProcessor did not find any billing files to process at all. Can you confirm what billing files you have in your billing s3 bucket? What values did you set for the following properties? ice.billing_s3bucketname ice.billing_s3bucketprefix

Jithinxavi commented 11 years ago

@fangji , Please find the details below.

ice.billing_s3bucketname= ice.billing_s3bucketprefix=ice/billing/

Once I ran the grails app its automatically created one directory inside my billing bucket called ice and inside that created couple of files starting with billing(prefix) - Any issues here? or it should be inside the billing bucket itself?

fangji commented 11 years ago

@Jithinxavi ice.billing_s3bucketname is the bucket where you detailed billing files are. ice.work_s3bucketname is the bucket Ice writer will use to store output files. It looks like you have the same values for both properties.

Jithinxavi commented 11 years ago

@fangji , No , Both have different values,

ice.billing_s3bucketname=dsb-billingdata ice.billing_s3bucketprefix=ice/billing/ ice.work_s3bucketname=dsb-iceworker ice.work_s3bucketprefix=work_billing/

Inside the dsb-billingdata bucket have all the billing data files(.czv,gz etc) and inside the ice directory having the app generated data(attached screenshot)

And I have one separate bucket for ice write - dsb-iceworker ,inside of this having one folder and have output files.

billing-data work-billing

fangji commented 11 years ago

@Jithinxavi the billing files in your billing s3 bucket don't look right. As mentioned in prerequisite in readme, the billing files should look like: -aws-billing-detailed-line-items--.csv.zip or -aws-billing-detailed-line-items-with-resources-and-tags--.csv.zip. Please double check with Amazon that you receive the correct billing files.

fangji commented 11 years ago

@Jithinxavi it looks like your ice.billing_s3bucketprefix should be empty since those billing files are directly under the s3 bucket. Make sure you have read permission to the billing s3 bucket and read&write permission to working s3 bucket. But since we didn't see any permission errors in your Ice writer log, you probably have the right permission already.

Please change ice.billing_s3bucketprefix to following and try again: ice.billing_s3bucketprefix=

Jithinxavi commented 11 years ago

Thanks @fangji for your valuable notes, It's looks awesome.

I have couple of concerns- 1. how can I run this apps in background, now when my ssh session is interrupting, application as well shutting down. I tried with nohup but no look.

  1. In screenshot I can see one option "Breakdown" this tab is missing in my dashboard. How can I configure that?
  2. Compare to screenshot in Readme , application style has little different, that is because of "highstock"?
fangji commented 11 years ago

@Jithinxavi

  1. One option is to create a war file and deploy it to tomcat
  2. To enable breakdown feature, you will need to do step 6 in advanced options. Please see readme.
  3. The ui style should be the same. In what way is yours different from the screenshots in readme?
Jithinxavi commented 11 years ago

@fangji -

  1. We have created one war file and follow the below steps. war file in - /ice/target/ice.war

    ubuntu@Netflix-ICE-Monitoring:~/ice$ sudo ./grailsw run-war | Done creating WAR target/ice.war | Running Grails application Tomcat Server running WAR (output written to: /home/ubuntu/ice/target/tomcat-out.txt) | Server running. Browse to http://localhost:8080/ice

ice.properties file I have passed,

ice.s3AccessKeyId=Access Key ice.s3SecretKey=Secret Key

But unfortunately, I'm getting this problem ,looks like some credentials file issues.

2013-06-25 07:32:52,347 [localhost-startStop-1] WARN config.ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/home/ubuntu/ice/work/war/WEB-INF/lib/ehcache-core-2.4.6.jar!/ehcache-failsafe.xml 2013-06-25 07:32:52,897 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.UpdateTimestampsCache]; using defaults. 2013-06-25 07:32:53,025 [localhost-startStop-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.StandardQueryCache]; using defaults. 2013-06-25 07:32:54,953 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@314382c6: startup date [Tue Jun 25 07:32:46 UTC 2013]; parent: Root WebApplicationContext 2013-06-25 07:32:58,059 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded. 2013-06-25 07:32:58,513 [localhost-startStop-1] INFO BootStrap - Starting ice... 2013-06-25 07:32:59,230 [localhost-startStop-1] INFO processor.ProcessorConfig - starting up... 2013-06-25 07:32:59,237 [localhost-startStop-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started... 2013-06-25 07:32:59,249 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting... 2013-06-25 07:32:59,329 [localhost-startStop-1] INFO basic.BasicManagers - trying to find new tag group and data managers... 2013-06-25 07:33:00,526 [localhost-startStop-1] ERROR basic.BasicTagGroupManager - cannot poll data AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 914BBD25162C7BAD, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: hfBpaymvraL7JBTIzpGfcOfaX8FoRdoFtV9BghyM+iob2SCRbb6xY9SpOg757W6H at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:793) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:773) at com.netflix.ice.common.AwsUtils.downloadFileIfChanged(AwsUtils.java:191) at com.netflix.ice.basic.BasicTagGroupManager.poll(BasicTagGroupManager.java:64) at com.netflix.ice.basic.BasicTagGroupManager.(BasicTagGroupManager.java:54) at com.netflix.ice.basic.BasicManagers.doWork(BasicManagers.java:111) at com.netflix.ice.basic.BasicManagers.init(BasicManagers.java:59) at com.netflix.ice.reader.ReaderConfig.(ReaderConfig.java:98) at BootStrap$_closure1.doCall(BootStrap.groovy:163) at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:301) at grails.util.Environment.executeForEnvironment(Environment.java:294) at grails.util.Environment.executeForCurrentEnvironment(Environment.java:270) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-06-25 07:33:00,535 [localhost-startStop-1] INFO basic.BasicTagGroupManager - poller thread for tagdb_all started... 2013-06-25 07:33:00,536 [tagdb_all] INFO basic.BasicTagGroupManager - poller starting... 2013-06-25 07:33:00,546 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: D5A7D61ACF577CAA, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: Ggt46yIaVWaMnhlWmyvfeesp1IC8nN7iZWZV+QaWx8kjLCgk5wKkCM5PnrJz6Mx5 at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2950) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478) at com.netflix.ice.common.AwsUtils.listAllObjects(AwsUtils.java:102) at com.netflix.ice.processor.BillingFileProcessor.poll(BillingFileProcessor.java:75) at com.netflix.ice.common.Poller.doWork(Poller.java:50) at com.netflix.ice.common.Poller.access$000(Poller.java:28) at com.netflix.ice.common.Poller$1.run(Poller.java:88) at java.lang.Thread.run(Thread.java:679) 2013-06-25 07:33:00,552 [tagdb_all] ERROR basic.BasicTagGroupManager - Error polling AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 61291A1A92D9DC79, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: nIbn8SrGymPPT8oPl+LXuq35x/m287bpuAFR63JVcA7x//Ep3T4alBsaAOv/or6a at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:793) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:773) at com.netflix.ice.common.AwsUtils.downloadFileIfChanged(AwsUtils.java:191) at com.netflix.ice.basic.BasicTagGroupManager.poll(BasicTagGroupManager.java:64) at com.netflix.ice.common.Poller.doWork(Poller.java:50) at com.netflix.ice.common.Poller.access$000(Poller.java:28) at com.netflix.ice.common.Poller$1.run(Poller.java:88) at java.lang.Thread.run(Thread.java:679) 2013-06-25 07:33:00,603 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for cost_hourly_all started... 2013-06-25 07:33:00,604 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for usage_hourly_all started... 2013-06-25 07:33:00,604 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for cost_daily_all started... 2013-06-25 07:33:00,604 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for usage_daily_all started... 2013-06-25 07:33:00,605 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for cost_weekly_all started... 2013-06-25 07:33:00,605 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for usage_weekly_all started... 2013-06-25 07:33:00,605 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for cost_monthly_all started... 2013-06-25 07:33:00,605 [localhost-startStop-1] INFO basic.BasicDataManager - poller thread for usage_monthly_all started... 2013-06-25 07:33:00,606 [localhost-startStop-1] INFO basic.BasicManagers - poller thread for com.netflix.ice.basic.BasicManagers started... 2013-06-25 07:33:00,603 [cost_hourly_all] INFO basic.BasicDataManager - poller starting... 2013-06-25 07:33:00,606 [usage_monthly_all] INFO basic.BasicDataManager - poller starting... 2013-06-25 07:33:00,605 [cost_monthly_all] INFO basic.BasicDataManager - poller starting... 2013-06-25 07:33:00,605 [usage_weekly_all] INFO basic.BasicDataManager - poller starting... 2013-06-25 07:33:00,605 [cost_weekly_all] INFO basic.BasicDataManager - poller starting... 2013-06-25 07:33:00,604 [usage_daily_all] INFO basic.BasicDataManager - poller starting... 2013-06-25 07:33:00,604 [cost_daily_all] INFO basic.BasicDataManager - poller starting... 2013-06-25 07:33:00,604 [usage_hourly_all] INFO basic.BasicDataManager - poller starting... 2013-06-25 07:33:00,616 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - poller starting... 2013-06-25 07:33:00,617 [com.netflix.ice.basic.BasicManagers] INFO basic.BasicManagers - trying to find new tag group and data managers... 2013-06-25 07:33:00,722 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring FrameworkServlet 'grails' 2013-06-25 07:33:00,785 [main] INFO http11.Http11Protocol - Starting ProtocolHandler ["http-bio-8080"] Server running. Browse to http://localhost:8080/ice 2013-06-25 07:33:56,645 [http-bio-8080-exec-21] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [GET] /ice/dashboard/getAccounts Stacktrace follows: java.lang.NullPointerException at com.netflix.ice.basic.BasicTagGroupManager.getMonthMillis(BasicTagGroupManager.java:126) at com.netflix.ice.basic.BasicTagGroupManager.getAccounts(BasicTagGroupManager.java:137) at com.netflix.ice.basic.BasicTagGroupManager.getAccounts(BasicTagGroupManager.java:223) at com.netflix.ice.DashboardController$_closure2.doCall(DashboardController.groovy:73) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-06-25 07:33:56,871 [http-bio-8080-exec-22] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [POST] /ice/dashboard/getData Cannot invoke method getStart() on null object. Stacktrace follows: java.lang.NullPointerException: Cannot invoke method getStart() on null object at com.netflix.ice.DashboardController.roundInterval(DashboardController.groovy:621) at com.netflix.ice.DashboardController.doGetData(DashboardController.groovy:397) at com.netflix.ice.DashboardController$_closure11.doCall(DashboardController.groovy:314) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-06-25 07:34:31,131 [http-bio-8080-exec-14] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [GET] /ice/dashboard/getAccounts Stacktrace follows: java.lang.NullPointerException at com.netflix.ice.basic.BasicTagGroupManager.getMonthMillis(BasicTagGroupManager.java:126) at com.netflix.ice.basic.BasicTagGroupManager.getAccounts(BasicTagGroupManager.java:137) at com.netflix.ice.basic.BasicTagGroupManager.getAccounts(BasicTagGroupManager.java:223) at com.netflix.ice.DashboardController$_closure2.doCall(DashboardController.groovy:73) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-06-25 07:34:31,136 [http-bio-8080-exec-12] ERROR errors.GrailsExceptionResolver - NullPointerException occurred when processing request: [POST] /ice/dashboard/getData

Please let me know if anything went wrong.

Thanks, Jithin Xavier

fangji commented 11 years ago

@Jithinxavi You don't need to set properties ice.s3AccessKeyId and ice.s3SecretKey in ice.properties file. They are system properties, you can set them through JAVA_OPTS.

Jithinxavi commented 11 years ago

@fangji - I am newbie to tomcat and java, Can you please advise me to how to set the JAVA_OPTS. file with credentials?

And any restriction is there to add AWS account in ICE? (I have added 5 account butonly I can see 4)

Thanks, Jithin Xavier

fangji commented 11 years ago

@Jithinxavi If you are using tomcat, you can probably set it as env variable. Here is what I found on google when searching "tomcat how to set JAVA_OPTS": http://tomcat.10.x6.nabble.com/How-to-set-JAVA-OPTS-variable-in-tomcat6-0-16-td2119946.html For the account issue, I'd open up the billing file and check if the account id is really there.

ma78 commented 11 years ago

Any chance of having ice use the instance metadata for credentials? (Ie. Instance has been started with an IBM role) On 26/06/2013 5:06 AM, "fangji" notifications@github.com wrote:

@Jithinxavi https://github.com/Jithinxavi You don't need to set properties ice.s3AccessKeyId and ice.s3SecretKey in ice.properties file. They are system properties, you can set them through JAVA_OPTS.

— Reply to this email directly or view it on GitHubhttps://github.com/Netflix/ice/issues/4#issuecomment-19999450 .

Jithinxavi commented 11 years ago

@fangji - 1. Installed and configured tomcat7 with ice.s3AccessKeyId and ice.s3SecretKey in /etc/init.d/tomcat

JAVA_OPTS="ice.s3AccessKeyId=Accesskey ice.s3SecretKey=SecretKey" - correct?

  1. Added my ice.war file in /var/lib/tomcat7/webapps and restart tomcat, and application I can access from http://url:8080/ice/dashboard/summary ,but still getting the access key error in tomcat catalina.out

Please have look on my catalina.out

13-06-26 11:39:27,706 [pool-2-thread-1] WARN config.ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml found in the classpath: jar:file:/var/lib/tomcat7/webapps/ice/WEB-INF/lib/ehcache-core-2.4.6.jar!/ehcache-failsafe.xml 2013-06-26 11:39:27,974 [pool-2-thread-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.UpdateTimestampsCache]; using defaults. 2013-06-26 11:39:28,015 [pool-2-thread-1] WARN hibernate.AbstractEhcacheRegionFactory - Couldn't find a specific ehcache configuration for cache named [org.hibernate.cache.StandardQueryCache]; using defaults. 2013-06-26 11:39:29,737 [pool-2-thread-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@2f473a2e: startup date [Wed Jun 26 11:39:23 UTC 2013]; parent: Root WebApplicationContext 2013-06-26 11:39:33,803 [pool-2-thread-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded. 2013-06-26 11:39:34,079 [pool-2-thread-1] INFO BootStrap - Starting ice... 2013-06-26 11:39:35,283 [pool-2-thread-1] INFO processor.ProcessorConfig - starting up... 2013-06-26 11:39:35,289 [pool-2-thread-1] INFO processor.BillingFileProcessor - poller thread for com.netflix.ice.processor.BillingFileProcessor started... 2013-06-26 11:39:35,293 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - poller starting... 2013-06-26 11:39:35,394 [pool-2-thread-1] INFO basic.BasicManagers - trying to find new tag group and data managers... 2013-06-26 11:39:36,420 [pool-2-thread-1] ERROR basic.BasicTagGroupManager - cannot poll data AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 9E2DF6F4826EC604, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: 6siKtaIQMJetW/v/ojJXu/H0w8IqfYbaZi7Qp51Bm0egEHboPyI9gY5G8otRKnlt at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:793) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:773) at com.netflix.ice.common.AwsUtils.downloadFileIfChanged(AwsUtils.java:191) at com.netflix.ice.basic.BasicTagGroupManager.poll(BasicTagGroupManager.java:64) at com.netflix.ice.basic.BasicTagGroupManager.(BasicTagGroupManager.java:54) at com.netflix.ice.basic.BasicManagers.doWork(BasicManagers.java:111) at com.netflix.ice.basic.BasicManagers.init(BasicManagers.java:59) at com.netflix.ice.reader.ReaderConfig.(ReaderConfig.java:98) at BootStrap$_closure1.doCall(BootStrap.groovy:163) at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:301) at grails.util.Environment.executeForEnvironment(Environment.java:294) at grails.util.Environment.executeForCurrentEnvironment(Environment.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-06-26 11:39:36,429 [pool-2-thread-1] INFO basic.BasicTagGroupManager - poller thread for tagdb_all started... 2013-06-26 11:39:36,432 [tagdb_all] INFO basic.BasicTagGroupManager - poller starting... 2013-06-26 11:39:36,469 [tagdb_all] ERROR basic.BasicTagGroupManager - Error polling AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 50F18CAE514C2A2E, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: xa7BsuK/N/tfU/2u6vJZZqKifyjbP6AKtogVwkgbfMhllk3iKgTNVnx00e9zDe33 2013-06-26 11:39:36,429 [pool-2-thread-1] INFO basic.BasicTagGroupManager - poller thread for tagdb_all started... 2013-06-26 11:39:36,432 [tagdb_all] INFO basic.BasicTagGroupManager - poller starting... 2013-06-26 11:39:36,469 [tagdb_all] ERROR basic.BasicTagGroupManager - Error polling AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 50F18CAE514C2A2E, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: xa7BsuK/N/tfU/2u6vJZZqKifyjbP6AKtogVwkgbfMhllk3iKgTNVnx00e9zDe33 at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:793) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:773) at com.netflix.ice.common.AwsUtils.downloadFileIfChanged(AwsUtils.java:191) at com.netflix.ice.basic.BasicTagGroupManager.poll(BasicTagGroupManager.java:64) at com.netflix.ice.common.Poller.doWork(Poller.java:50) at com.netflix.ice.common.Poller.access$000(Poller.java:28) at com.netflix.ice.common.Poller$1.run(Poller.java:88) at java.lang.Thread.run(Thread.java:679) 2013-06-26 11:39:36,469 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: F537BB4073C04D3F, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: StbH8zIxtNvh8C6P2YHM8bs86ukp1XPSq97vgq7LzBgN4Qcp+OzXp7sqoBgd53pY at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2979) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2950) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478) at com.netflix.ice.common.AwsUtils.listAllObjects(AwsUtils.java:102) at com.netflix.ice.processor.BillingFileProcessor.poll(BillingFileProcessor.java:75) at com.netflix.ice.common.Poller.doWork(Poller.java:50) at com.netflix.ice.common.Poller.access$000(Poller.java:28) at com.netflix.ice.common.Poller$1.run(Poller.java:88) at java.lang.Thread.run(Thread.java:679) 2013-06-26 11:39:36,534 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for cost_hourly_all started... 2013-06-26 11:39:36,535 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for usage_hourly_all started... 2013-06-26 11:39:36,535 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for cost_daily_all started... 2013-06-26 11:39:36,536 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for usage_daily_all started... 2013-06-26 11:39:36,536 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for cost_weekly_all started... 2013-06-26 11:39:36,536 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for usage_weekly_all started... 2013-06-26 11:39:36,537 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for cost_monthly_all started... 2013-06-26 11:39:36,537 [pool-2-thread-1] INFO basic.BasicDataManager - poller thread for usage_monthly_all started... 2013-06-26 11:39:36,537 [pool-2-thread-1] INFO basic.BasicManagers - poller thread for com.netflix.ice.basic.BasicManagers started... 2013-06-26 11:39:36,535 [cost_hourly_all] INFO basic.BasicDataManager - poller starting... 2013-06-26 11:39:36,537 [usage_monthly_all] INFO basic.BasicDataManager - poller starting... 2013-06-26 11:39:36,537 [cost_monthly_all] INFO basic.BasicDataManager - poller starting... 2013-06-26 11:39:36,537 [usage_weekly_all] INFO basic.BasicDataManager - poller starting...

fangji commented 11 years ago

@Jithinxavi JAVA_OPTS should be "-Dice.s3AccessKeyId=Accesskey -Dice.s3SecretKey=SecretKey" I will check in the code to get credentials from instance metadata today. Hopefully it will make things a little easier for you.

Jithinxavi commented 11 years ago

Oh! That's great @fangji - I have added the new entry as well but no luck. once you made the metadata change, Please let me know.

ma78 commented 11 years ago

@fangji will you able to use the token from the instance userdata also?

fangji commented 11 years ago

@Jithinxavi @ma78 Now, if access keys are not set through System properties, Bootstrap.groovy will use credentials of instance metadata by using InstanceProfileCredentialsProvider.

Jithinxavi commented 11 years ago

@fangji Thanks for the update. I have added credentials as userdata in instance, like below.

ubuntu@Netflix-ICE-Monitoring:~$ GET http://169.254.169.254/latest/user-data ice.s3AccessKeyId=AccessKeyId ice.s3SecretKey=s3SecretKey@Netflix-ICE-Monitoring:~$

Is this correct?

ebuychance commented 11 years ago

I have been fighting this all day, not sure why I am hitting a null exception...? Any insight would be greatly appreciated. I am not a Java developer and fairly new to AWS.

sudo ./grailsw -Dice.s3AccessKeyId=theS3AccessKeyId -Dice.s3SecretKey=theS3SecretKey run-app

| Running Grails application 2013-07-02 11:49:00,603 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"] 2013-07-02 11:49:00,625 [main] INFO core.StandardService - Starting service Tomcat 2013-07-02 11:49:00,625 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30 2013-07-02 11:49:00,693 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found 2013-07-02 11:49:00,746 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext 2013-07-02 11:49:02,602 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@326d234c: startup date [Tue Jul 02 11:49:02 PDT 2013]; parent: Root WebApplicationContext 2013-07-02 11:49:03,193 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded. 2013-07-02 11:49:03,214 [localhost-startStop-1] INFO BootStrap - Starting ice... | Error java.lang.NullPointerException | Error at java.util.Hashtable.put(Hashtable.java:394) | Error at java.util.Properties.setProperty(Properties.java:143) | Error at java_util_Properties$setProperty.call(Unknown Source) | Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) | Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) | Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) | Error at BootStrap$_closure1.doCall(BootStrap.groovy:117) | Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) | Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | Error at java.lang.reflect.Method.invoke(Method.java:597) | Error at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1243) | Error at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) | Error at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) | Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082) | Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106) | Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906) | Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:956) | Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106) | Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906) | Error at groovy.lang.Closure.call(Closure.java:412) | Error at groovy.lang.Closure.call(Closure.java:406) | Error at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:308) | Error at grails.util.Environment.executeForEnvironment(Environment.java:301) | Error at grails.util.Environment.executeForCurrentEnvironment(Environment.java:277) | Error at org.codehaus.groovy.grails.commons.DefaultGrailsBootstrapClass.callInit(DefaultGrailsBootstrapClass.java:60) | Error at org.codehaus.groovy.grails.web.context.GrailsConfigUtils.executeGrailsBootstraps(GrailsConfigUtils.java:74) | Error at org.codehaus.groovy.grails.web.context.GrailsContextLoader.initWebApplicationContext(GrailsContextLoader.java:108) | Error at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111) | Error at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791) | Error at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285) | Error at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) | Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) | Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) | Error at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) | Error at java.util.concurrent.FutureTask.run(FutureTask.java:138) | Error at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) | Error at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) | Error at java.lang.Thread.run(Thread.java:680) | Error 2013-07-02 11:49:03,282 [localhost-startStop-1] ERROR BootStrap - Startup failed Message: null Line | Method ->> 394 | put in java.util.Hashtable

| 143 | setProperty in java.util.Properties | 117 | doCall . . . . . . . . . . . . . in BootStrap$_closure1 | 308 | evaluateEnvironmentSpecificBlock in grails.util.Environment | 301 | executeForEnvironment . . . . . in '' | 277 | executeForCurrentEnvironment in '' | 303 | innerRun . . . . . . . . . . . . in java.util.concurrent.FutureTask$Sync | 138 | run in java.util.concurrent.FutureTask | 895 | runTask . . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker | 918 | run in '' ^ 680 | run . . . . . . . . . . . . . . in java.lang.Thread

ice.properties:

ice.processor=true ice.reader=false ice.startmillis=1370070000000 ice.billing_s3bucketname=abucket ice.billing_s3bucketprefix= ice.work_s3bucketname=work_abucket ice.work_s3bucketprefix= ice.processor.localDir=/mnt/ice_processor ice.account.account1=00123456789

fangji commented 11 years ago

@ebuychance It looks like you didn't specify ice.companyName in your ice.properties file. The readme says it's only needed for running reader. But actually it's also required for running processor in the code. I will fix the code in a little later. For now you can try adding ice.companyName to your ice.properties file.

ebuychance commented 11 years ago

Wow that was a fast response, work around and a code fix! I am very impressed and thankful. Thankful both for the open source project and bug support.

Yep that did the trick, loved seeing this message: "AWS usage processed"

ralph-tice commented 11 years ago

I think the README is in much better shape now, thanks for all the awesome effort @fangji and very nice project !

roinsharma commented 10 years ago

@fangji - i am a new bie here , i have been working on Netflix since two weeks and unable to proceed further because of some errors . I am in my local environment and got the ice /netflix . I have my ice properties file in right location .

the issue is when i run this command : grailsw -Dice.s3AccessKeyId=ABCDEFADHAHAH -Dice.s3SecretKey=ox737271dhwhhdgqg737hsh run-app

I get the following error :+1:

Extracting C:\Users\roin.grails\wrapper\grails-2.2.1-download.zip to C:\Users\r oin.grails\wrapper\2.2.1 Exception in thread "main" java.util.zip.ZipException: error in opening zip file

    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:215)
    at java.util.zip.ZipFile.<init>(ZipFile.java:145)
    at java.util.zip.ZipFile.<init>(ZipFile.java:159)
    at org.grails.wrapper.GrailsWrapper.extract(GrailsWrapper.java:147)
    at org.grails.wrapper.GrailsWrapper.configureGrailsInstallation(GrailsWr

apper.java:138) at org.grails.wrapper.GrailsWrapper.main(GrailsWrapper.java:58)

I have already installed the following files in LIB ant-1.7.0.jar | guava-14.0.1.jar | httpcore-4.1.jar jcommon-1.0.16.jar joda-time-2.0.jar org.json-chargebee-1.0.jar wstx-asl-3.2.9.jar aws-java-sdk-1.4.4.1.jar httpclient-4.1.jar javacsv-2.0.jar jfreechart-1.0.13.jar mail-1.4.1.jar poi-3.7.jar

Please Help.