linkedin / dr-elephant

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark
Apache License 2.0
1.36k stars 859 forks source link

Database 'default' is in an inconsistent state! #328

Open jibran123 opened 6 years ago

jibran123 commented 6 years ago

Hello All

I have run the updated 1.sql ( including the drop table ) still i am facing the issue, can you all please help me

I have also run the following - ( based on FAQ link )

export jvm_args="-Devolutionplugin=enabled -DapplyEvolutions.default=true -mem 1024 -J-Xloggc:$project_root../logs/elephant/dr-gc.date +'%Y%m%d%H%M' -J-XX:+PrintGCDetails" export jvm_props="-Devolutionplugin=enabled and -DapplyEvolutions.default=true"

nohup /apps/dr_elephant/target/universal/dr-elephant-2.0.13/bin/dr-elephant -Djava.library.path=/lib/native -Dhttp.port=8080 -Ddb.default.url="jdbc:mysql://it00512:3306/drelephant?characterEncoding=UTF-8"" -Ddb.default.user=root -Ddb.default.password=***** > /apps/dr_elephant/dr.log 2>&1

Error stack trace -

SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/apps/dr_elephant/target/universal/dr-elephant-2.0.13/lib/ch.qos.logback.logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/apps/dr_elephant/target/universal/dr-elephant-2.0.13/lib/org.slf4j.slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/apps/dr_elephant/target/universal/dr-elephant-2.0.13/lib/org.slf4j.slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/apps/opt/cloudera/parcels/CDH-5.11.2-1.cdh5.11.2.p2699.2946/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] [info] play - database [default] connected at jdbc:mysql://it00512:3306/drelephant?characterEncoding=UTF-8 [error] play - Oops, cannot start the server. @76pacppo4: Database 'default' is in an inconsistent state! at play.api.db.evolutions.Evolutions$.checkEvolutionsState(Evolutions.scala:193) at play.api.db.evolutions.Evolutions$.databaseEvolutions(Evolutions.scala:348) at play.api.db.evolutions.Evolutions$$anonfun$evolutionScript$2.apply(Evolutions.scala:313) at play.api.db.evolutions.Evolutions$$anonfun$evolutionScript$2.apply(Evolutions.scala:311) at scala.Option.map(Option.scala:145) at play.api.db.evolutions.Evolutions$.evolutionScript(Evolutions.scala:311) at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1$$anonfun$apply$1.apply$mcV$sp(Evolutions.scala:462) at play.api.db.evolutions.EvolutionsPlugin.withLock(Evolutions.scala:507) at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.scala:461) at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.scala:459) at scala.collection.immutable.List.foreach(List.scala:318) at play.api.db.evolutions.EvolutionsPlugin.onStart(Evolutions.scala:459) at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88) at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88) at scala.collection.immutable.List.foreach(List.scala:318) at play.api.Play$$anonfun$start$1.apply$mcV$sp(Play.scala:88) at play.api.Play$$anonfun$start$1.apply(Play.scala:88) at play.api.Play$$anonfun$start$1.apply(Play.scala:88) at play.utils.Threads$.withContextClassLoader(Threads.scala:18) at play.api.Play$.start(Play.scala:87) at play.core.StaticApplication.(ApplicationProvider.scala:52) at play.core.server.NettyServer$.createServer(NettyServer.scala:280) at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:316) at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:311) at scala.Option.map(Option.scala:145) at play.core.server.NettyServer$.main(NettyServer.scala:311) at play.core.server.NettyServer.main(NettyServer.scala)

ricdong commented 6 years ago

I met the same error, Database 'default' is in an inconsistent **state

but I found two tables were created in my database.

play_evolutions 
yarn_app_result 
jibran123 commented 6 years ago

Were u able to resolve the issue Are we supposed to keep all the tables as part of 1.sql or remove them Can anyone please suggest or help us on this.

ricdong commented 6 years ago

hi jibran123:

have you seen any error information in logs/application.log ?

which version of mysql are you use? upgrade your mysql to 5.7 if you use the lower versions because of mysql 5.6 (includes prior versions) InnoDB max index length is 767 bytes, mysql 5.7.7 is up to 3072 bytes.

jibran123 commented 6 years ago

hello

i do not see that error in application.log

following is the o/p of application.log

Starting Application...

2018-01-29 07:35:27,303 - [INFO] - from application in pool-1-thread-1 Stopping application...

2018-01-29 07:35:27,376 - [INFO] - from application in pool-1-thread-1 Starting Application...

2018-01-29 07:35:27,381 - [INFO] - from application in pool-1-thread-1 Stopping application...

2018-01-29 07:35:27,455 - [INFO] - from application in pool-1-thread-1 Starting Application...

2018-01-29 07:35:27,484 - [INFO] - from application in pool-1-thread-1 Stopping application...

2018-01-29 07:38:31,988 - [INFO] - from application in pool-1-thread-1 Starting Application...

2018-01-29 07:38:32,014 - [INFO] - from application in pool-1-thread-1 Stopping application...

2018-01-29 07:38:33,437 - [INFO] - from application in pool-1-thread-1 Starting Application...

2018-01-29 07:38:33,438 - [INFO] - from application in pool-1-thread-1 Stopping application...

2018-01-29 07:38:33,567 - [INFO] - from application in pool-1-thread-1 Starting Application...

2018-01-29 07:38:33,580 - [INFO] - from application in pool-1-thread-1 Stopping application...

is there any more troubleshooting i can do i am sure i am missing something very cosmetic, but i have this urgent client requirement to fix this set up.

Thanks in advance.

ricdong commented 6 years ago

have you configured the database connector in app-conf/elephant.conf ?

jibran123 commented 6 years ago

yes, thats done

ambauma commented 6 years ago

Have you tried running each statement manually that is in the sql files?

jibran123 commented 6 years ago

yeah i did that i have even updated the mysql version

i also updated the 1.sql and now i am getting this error inspite of following all the steps

[info] play - database [default] connected at jdbc:mysql://it0645:3306/drelephant [error] play - Specified key was too long; max key length is 767 bytes [ERROR:1071, SQLSTATE:42000] Oops, cannot start the server. @7755npnf2: Database 'default' is in an inconsistent state!

chetnachaudhari commented 6 years ago

Any update on this? I am facing the same issue?

chetnachaudhari commented 6 years ago

@jibran123 : you can refer to https://github.com/linkedin/dr-elephant/wiki/FAQ#2-specified-key-was-too-long-max-key-length-is-767-bytes-error1071-sqlstate42000 ti get rid of max length error. It worked for me.

loupipalien commented 6 years ago

@jibran123 you can refer to dr-elephant/issues#230

songgane-zz commented 6 years ago

To fix this problem, you must install mysql 5.7+ or mariadb 10.2+.