search

twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Memcache timeout #704

Open leesf opened 7 years ago

leesf commented 7 years ago

When using memcache(https://github.com/twitter/storehaus) in online layer to store summingbird processed data. timeout of memcache is set to 15 seconds. However, when writing speed to memcache exceeds 5000/s, there will be a individualRequestTimeoutException of Memecache, the details are following

ERROR backtype.storm.daemon.executor - com.twitter.finagle.IndividualRequestTimeoutException: exceeded 15.seconds to memcache while waiting for a response for an individual request, excluding retries at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]

Who knows why? Thanks for replying.

pankajroark commented 7 years ago

15 seconds timeout is very high, timeouts at that indicate serious problems. May be check the state of memcache cluster, and the client. You'd want to make sure you're using a high value of MaxWaitingFutures to make sure you're getting sufficient parallelism on the client. Try different loads and observe how p999 latency varies. I'm afraid there's no silver bullet. These kinds of issues require careful debugging. Taking yourkit profile has been very helpful to me in the past for debugging such issues.

On Jan 1, 2017, 9:31 PM -0800, leesf notifications@github.com, wrote:

When using memcache(https://github.com/twitter/storehaus) in online layer to store summingbird processed data. timeout of memcache is set to 15 seconds. However, when writing speed to memcache exceeds 5000/s, there will be a individualRequestTimeoutException of Memecache, the details are following

ERROR backtype.storm.daemon.executor - com.twitter.finagle.IndividualRequestTimeoutException: exceeded 15.seconds to memcache while waiting for a response for an individual request, excluding retries at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]

Who knows why? Thanks for replying.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub (https://github.com/twitter/summingbird/issues/704), or mute the thread (https://github.com/notifications/unsubscribe-auth/AAojhljzaCnNQdB1ayHPmYwd5FNIGyuUks5rOIu-gaJpZM4LY1-2).

leesf commented 7 years ago

@pankajroark thanks for replying, i did not use memcache in cluster, but in local machine, and i checked the state of memcache in machine, it seems all goes well, its really odd.

leesf commented 7 years ago

@pankajroark

When i use memcache in storehaus alone, there is an error. I changed the memcache port to 12000. And the code is below

import com.twitter.storehaus.memcache.MemcacheStore import com.twitter.conversions.time._ import com.twitter.finagle.redis.util.{CBToString, StringToChannelBuffer} import com.twitter.util.Await

object MemcacheTest { def main(args: Array[String]): Unit = { val client = MemcacheStore. defaultClient("memcache", "localhost:12000", 2, 10.seconds) val store = MemcacheStore(client)

new Thread(new Runnable { override def run(): Unit = try { for (i <- 1 to 100000) store.put(i.toString, Some(StringToChannelBuffer(i.toString))) } catch { case e: Exception => e.printStackTrace } }).start

new Thread(new Runnable { override def run(): Unit = { val start = System.currentTimeMillis() val num = "100" try { for (i <- 1 to 10) Await.result(store.get(num)) match { case Some(value) => println(CBToString(value)) case None => { println("None") } } } catch { case e: Exception => e.printStackTrace } val end = System.currentTimeMillis() println((end - start) + "") }}).start } }

error info is below

com.twitter.finagle.ServiceTimeoutException: exceeded 10.seconds to memcache while creating a service/connection or reserving a service/connection from the service/connection pool memcache at com.twitter.finagle.client.DefaultClient$$anonfun$14.apply(DefaultClient.scala:155) at com.twitter.finagle.client.DefaultClient$$anonfun$14.apply(DefaultClient.scala:153) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at com.twitter.finagle.factory.Refinery.(Refinery.scala:20) at com.twitter.finagle.client.DefaultClient$$anonfun$22.apply(DefaultClient.scala:220) at com.twitter.finagle.client.DefaultClient$$anonfun$22.apply(DefaultClient.scala:220) at com.twitter.finagle.builder.ClientBuilder.buildFactory(ClientBuilder.scala:849) at com.twitter.finagle.builder.ClientBuilder.build(ClientBuilder.scala:882) at com.twitter.finagle.memcached.KetamaClientBuilder.com$twitter$finagle$memcached$KetamaClientBuilder$$legacyFAClientBuilder$1(Client.scala:890) at com.twitter.finagle.memcached.KetamaClientBuilder$$anonfun$build$1.apply(Client.scala:901) at com.twitter.finagle.memcached.KetamaClientBuilder$$anonfun$build$1.apply(Client.scala:901) at com.twitter.finagle.memcached.KetamaClient$$anonfun$9$$anonfun$10.apply(Client.scala:722) at com.twitter.finagle.memcached.KetamaClient$$anonfun$9$$anonfun$10.apply(Client.scala:721) at scala.Option.map(Option.scala:145) at com.twitter.finagle.memcached.KetamaClient$$anonfun$9.apply(Client.scala:721) at com.twitter.finagle.memcached.KetamaClient$$anonfun$9.apply(Client.scala:716) at com.twitter.finagle.Group$$anonfun$map$1.applyOrElse(Group.scala:58) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) at com.twitter.finagle.Group$$anon$2$$anonfun$1$$anonfun$apply$1.applyOrElse(Group.scala:71) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) at scala.collection.TraversableLike$$anonfun$collect$1.apply(TraversableLike.scala:278) at scala.collection.immutable.Set$Set1.foreach(Set.scala:74) at scala.collection.TraversableLike$class.collect(TraversableLike.scala:278) at scala.collection.AbstractTraversable.collect(Traversable.scala:105) at com.twitter.finagle.Group$$anon$2$$anonfun$1.apply(Group.scala:70) at com.twitter.finagle.Group$$anon$2$$anonfun$1.apply(Group.scala:68) at com.twitter.util.Var$$anonfun$map$1.apply(Var.scala:53) at com.twitter.util.Var$$anonfun$map$1.apply(Var.scala:53) at com.twitter.util.Var$$anon$2$$anonfun$1.apply(Var.scala:66) at com.twitter.util.Var$$anon$2$$anonfun$1.apply(Var.scala:65) at com.twitter.util.Var$Observer.publish(Var.scala:153) at com.twitter.util.UpdatableVar.observe(Var.scala:371) at com.twitter.util.Var$$anon$2.observe(Var.scala:65) at com.twitter.util.Var$class.observe(Var.scala:31) at com.twitter.util.Var$$anon$2.observe(Var.scala:62) at com.twitter.util.Var$class.observeTo(Var.scala:82) at com.twitter.util.Var$$anon$2.observeTo(Var.scala:62) at com.twitter.finagle.Group$class.ref(Group.scala:49) at com.twitter.finagle.Group$$anon$2.ref$lzycompute(Group.scala:65) at com.twitter.finagle.Group$$anon$2.ref(Group.scala:65) at com.twitter.finagle.Group$class.members(Group.scala:87) at com.twitter.finagle.Group$$anon$2.members(Group.scala:65) at com.twitter.finagle.Group$class.apply(Group.scala:88) at com.twitter.finagle.Group$$anon$2.apply(Group.scala:65) at com.twitter.finagle.memcached.KetamaClient.(Client.scala:730) at com.twitter.finagle.memcached.KetamaClientBuilder.build(Client.scala:896) at com.twitter.storehaus.memcache.MemcacheStore$.defaultClient(MemcacheStore.scala:78) at com.hust.grid.leesf.WrappedMemcacheStore.MemcacheTest$.main(MemcacheTest.scala:15) at com.hust.grid.leesf.WrappedMemcacheStore.MemcacheTest.main(MemcacheTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

It seems the high concurrency read and write operation to memcache causes the problem. Any suggestions to solve it? Thanks for replying.

pankajroark commented 7 years ago

This error is different, afaict it indicates timeout in establishing a connection to memcache.

On Mon, Jan 2, 2017 at 9:09 AM Pankaj Gupta pankaj.roark@gmail.com wrote:

This

On Mon, Jan 2, 2017 at 3:32 AM leesf notifications@github.com wrote:

@pankajroark https://github.com/pankajroark

When i use memcache in storehaus alone, there is an error. I changed the memcache port to 12000. And the code is below

import com.twitter.storehaus.memcache.MemcacheStore import com.twitter.conversions.time._ import com.twitter.finagle.redis.util.{CBToString, StringToChannelBuffer} import com.twitter.util.Await

object MemcacheTest { def main(args: Array[String]): Unit = { val client = MemcacheStore. defaultClient("memcache", "localhost:12000", 2, 10.seconds) val store = MemcacheStore(client)

new Thread(new Runnable { override def run(): Unit = try { for (i <- 1 to 100000) store.put(i.toString, Some(StringToChannelBuffer(i.toString))) } catch { case e: Exception => e.printStackTrace } }).start

new Thread(new Runnable { override def run(): Unit = { val start = System.currentTimeMillis() val num = "100" try { for (i <- 1 to 10) Await.result(store.get(num)) match { case Some(value) => println(CBToString(value)) case None => { println("None") } } } catch { case e: Exception => e.printStackTrace } val end = System.currentTimeMillis() println((end - start) + "") }}).start } }

error info is below

com.twitter.finagle.ServiceTimeoutException: exceeded 10.seconds to memcache while creating a service/connection or reserving a service/connection from the service/connection pool memcache at com.twitter.finagle.client.DefaultClient$$anonfun$14.apply(DefaultClient.scala:155) at com.twitter.finagle.client.DefaultClient$$anonfun$14.apply(DefaultClient.scala:153) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at com.twitter.finagle.factory.Refinery.(Refinery.scala:20) at com.twitter.finagle.client.DefaultClient$$anonfun$22.apply(DefaultClient.scala:220) at com.twitter.finagle.client.DefaultClient$$anonfun$22.apply(DefaultClient.scala:220) at com.twitter.finagle.builder.ClientBuilder.buildFactory(ClientBuilder.scala:849) at com.twitter.finagle.builder.ClientBuilder.build(ClientBuilder.scala:882) at com.twitter.finagle.memcached.KetamaClientBuilder.com $twitter$finagle$memcached$KetamaClientBuilder$$legacyFAClientBuilder$1(Client.scala:890) at com.twitter.finagle.memcached.KetamaClientBuilder$$anonfun$build$1.apply(Client.scala:901) at com.twitter.finagle.memcached.KetamaClientBuilder$$anonfun$build$1.apply(Client.scala:901) at com.twitter.finagle.memcached.KetamaClient$$anonfun$9$$anonfun$10.apply(Client.scala:722) at com.twitter.finagle.memcached.KetamaClient$$anonfun$9$$anonfun$10.apply(Client.scala:721) at scala.Option.map(Option.scala:145) at com.twitter.finagle.memcached.KetamaClient$$anonfun$9.apply(Client.scala:721) at com.twitter.finagle.memcached.KetamaClient$$anonfun$9.apply(Client.scala:716) at com.twitter.finagle.Group$$anonfun$map$1.applyOrElse(Group.scala:58) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) at com.twitter.finagle.Group$$anon$2$$anonfun$1$$anonfun$apply$1.applyOrElse(Group.scala:71) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) at scala.collection.TraversableLike$$anonfun$collect$1.apply(TraversableLike.scala:278) at scala.collection.immutable.Set$Set1.foreach(Set.scala:74) at scala.collection.TraversableLike$class.collect(TraversableLike.scala:278) at scala.collection.AbstractTraversable.collect(Traversable.scala:105) at com.twitter.finagle.Group$$anon$2$$anonfun$1.apply(Group.scala:70) at com.twitter.finagle.Group$$anon$2$$anonfun$1.apply(Group.scala:68) at com.twitter.util.Var$$anonfun$map$1.apply(Var.scala:53) at com.twitter.util.Var$$anonfun$map$1.apply(Var.scala:53) at com.twitter.util.Var$$anon$2$$anonfun$1.apply(Var.scala:66) at com.twitter.util.Var$$anon$2$$anonfun$1.apply(Var.scala:65) at com.twitter.util.Var$Observer.publish(Var.scala:153) at com.twitter.util.UpdatableVar.observe(Var.scala:371) at com.twitter.util.Var$$anon$2.observe(Var.scala:65) at com.twitter.util.Var$class.observe(Var.scala:31) at com.twitter.util.Var$$anon$2.observe(Var.scala:62) at com.twitter.util.Var$class.observeTo(Var.scala:82) at com.twitter.util.Var$$anon$2.observeTo(Var.scala:62) at com.twitter.finagle.Group$class.ref(Group.scala:49) at com.twitter.finagle.Group$$anon$2.ref$lzycompute(Group.scala:65) at com.twitter.finagle.Group$$anon$2.ref(Group.scala:65) at com.twitter.finagle.Group$class.members(Group.scala:87) at com.twitter.finagle.Group$$anon$2.members(Group.scala:65) at com.twitter.finagle.Group$class.apply(Group.scala:88) at com.twitter.finagle.Group$$anon$2.apply(Group.scala:65) at com.twitter.finagle.memcached.KetamaClient.(Client.scala:730) at com.twitter.finagle.memcached.KetamaClientBuilder.build(Client.scala:896) at com.twitter.storehaus.memcache.MemcacheStore$.defaultClient(MemcacheStore.scala:78) at com.hust.grid.leesf.WrappedMemcacheStore.MemcacheTest$.main(MemcacheTest.scala:15) at com.hust.grid.leesf.WrappedMemcacheStore.MemcacheTest.main(MemcacheTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

It seems the high concurrency read and write operation to memcache causes the problem. Any suggestions to solve it? Thanks for replying.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/twitter/summingbird/issues/704#issuecomment-269961203, or mute the thread https://github.com/notifications/unsubscribe-auth/AAojhuqoOWFNkEDUI7CcAjGRGZ2b23JOks5rOOBggaJpZM4LY1-2 .

johnynek commented 7 years ago

are you sure you have memcache running? Can you access it any other way? Are you sure you don't have some firewall issues?

leesf commented 7 years ago

@johnynek Thanks for replying. Memcache is running well and i can use telnet command to access to memcache in my machine.