search

linkedin / dr-elephant

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark
Apache License 2.0
1.35k stars 859 forks source link

Cannot start Dr Elephant as Specified key was too long. max key length is 767 bytes [ERROR:1071, SQLSTATE:42000] #374

Open clairvoyant-rohit opened 6 years ago

clairvoyant-rohit commented 6 years ago

Facing this known issue with max key length while starting Dr Elephant. Did follow the steps in below url to fix it but still facing same issue . https://github.com/linkedin/dr-elephant/wiki/FAQ#2-specified-key-was-too-long-max-key-length-is-767-bytes-error1071-sqlstate42000

Is there anything else that needs to be done to fix the max key issue ? Anything on mysql to be changed ? I am getting blocked with this at the moment . Any help would be great.

Error: SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] [info] play - database [default] connected at jdbc:mysql:///drelephant?characterEncoding=UTF-8 [error] play - Specified key was too long; max key length is 767 bytes [ERROR:1071, SQLSTATE:42000] Oops, cannot start the server. @77l6c33ba: Database 'default' is in an inconsistent state! at play.api.db.evolutions.Evolutions$.checkEvolutionsState(Evolutions.scala:193) at play.api.db.evolutions.Evolutions$.databaseEvolutions(Evolutions.scala:348) at play.api.db.evolutions.Evolutions$$anonfun$evolutionScript$2.apply(Evolutions.scala:313) at play.api.db.evolutions.Evolutions$$anonfun$evolutionScript$2.apply(Evolutions.scala:311) at scala.Option.map(Option.scala:145) at play.api.db.evolutions.Evolutions$.evolutionScript(Evolutions.scala:311) at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1$$anonfun$apply$1.apply$mcV$sp(Evolutions.scala:462) at play.api.db.evolutions.EvolutionsPlugin.withLock(Evolutions.scala:507) at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.scala:461) at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.scala:459) at scala.collection.immutable.List.foreach(List.scala:318) at play.api.db.evolutions.EvolutionsPlugin.onStart(Evolutions.scala:459) at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88) at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88) at scala.collection.immutable.List.foreach(List.scala:318) at play.api.Play$$anonfun$start$1.apply$mcV$sp(Play.scala:88) at play.api.Play$$anonfun$start$1.apply(Play.scala:88) at play.api.Play$$anonfun$start$1.apply(Play.scala:88) at play.utils.Threads$.withContextClassLoader(Threads.scala:18) at play.api.Play$.start(Play.scala:87) at play.core.StaticApplication.(ApplicationProvider.scala:52) at play.core.server.NettyServer$.createServer(NettyServer.scala:243) at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:279) at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:274) at scala.Option.map(Option.scala:145) at play.core.server.NettyServer$.main(NettyServer.scala:274) at play.core.server.NettyServer.main(NettyServer.scala)

shkhrgpt commented 6 years ago

After making this change in 1.sql, try wiping out all the tables and restart Dr Elephant. It worked for me.

clairvoyant-rohit commented 6 years ago

After cleaning up the tables and restart Dr Elephant , ended up with same error . It seems that there might be other sql which will need similar changes. If anyone can help with it would be great.

This time it was able to create all the tables required in 1-5.sql as show below . +-----------------------------------+ | Tables_in_drelephant | +-----------------------------------+ | play_evolutions | | tuning_algorithm | | tuning_parameter | | yarn_app_heuristic_result | | yarn_app_heuristic_result_details | | yarn_app_result | +-----------------------------------+ 6 rows in set (0.00 sec)

Missing tables are flow_definition , job_definition, tuning_job_definition, flow_execution, job_execution, tuning_job_execution, job_saved_state, job_suggested_param_value. So looks like we have another problem with 5.sql script.

cnfree0355 commented 6 years ago

My mysql version is 5.6, when i change these sql index lengh ,It works !

evolutions/default/1.sql

create index yarn_app_result_i4 on yarn_app_result (flow_exec_id(100)); create index yarn_app_result_i5 on yarn_app_result (job_def_id(100)); create index yarn_app_result_i6 on yarn_app_result (flow_def_id(100));

evolutions/default/5.sql

-- flow_definition table change to UNIQUE KEY flow_def_id (flow_def_id(100)) -- job_definition table change to UNIQUE KEY job_def_id (job_def_id(100)) -- job_execution table change the index length like below: create index index_je_job_exec_id on job_execution (job_exec_id(100)); create index index_je_job_exec_url on job_execution (job_exec_url(100));

JOSEPH-HU commented 5 years ago

Why has this problem been not resolved, and the error is not detailed?