search

alibaba / jstorm

Enterprise Stream Process Engine
http://jstorm.io
Apache License 2.0
3.91k stars 1.8k forks source link

Drpc #216

Open shijiyu opened 8 years ago

shijiyu commented 8 years ago

你好 我先问下使用Jstorm的dprc 在配置文件中修改 drpc.servers:

hustfxj commented 8 years ago

是的，至于你说没有反馈结果。建议你先试一下 这个例子 你先跑下BasicDRPCTopology 这个例子

hammertank commented 8 years ago

@shijiyu ，我碰到过类似的情况。我当时遇见信息已经在Drpc服务收到了,但是没有反馈结果的原因是没有在所有节点配置drpc servers，导致topology中的woker一直在连接默认的drpc servers地址localhost。

这个是storm官网的相关文档，希望对你有用：

Next, you need to configure your Storm cluster to know the locations of the DRPC server(s). 
This is how DRPCSpout knows from where to read function invocations. 
This can be done through the storm.yaml file or the topology configurations.

如果配置storm.yaml，你需要配置所有节点的storm.yaml。如果通过topology configuration，需要在submitTopology的参数conf中配置。“drpc.servers”的value类型是java.util.List。

shijiyu commented 8 years ago

恩谢谢,现在storm.yaml下我都配置了drpc.servers,运行了一段时间以后又出现了 没有反馈的情况,这个是nimbus的日志 能否帮我分析下: [ERROR 2016-03-15 12:47:37 c.a.j.d.n.TopologyMetricsRunnable:689 TopologyMetricsRunnable] exceeding maxPendingUploadMetrics, skip metrics data for topology:move-jstorm-1-1457942488 [INFO 2016-03-15 12:47:55 c.a.j.d.n.TopologyMetricsRunnable:175 pool-6-thread-1] cluster metrics force upload. [INFO 2016-03-15 12:47:55 c.a.j.d.n.TopologyMetricsRunnable:610 pool-6-thread-1] send update event for cluster metrics, size : 7 [ERROR 2016-03-15 12:47:56 c.a.j.d.n.TopologyMetricsRunnable:689 TopologyMetricsRunnable] exceeding maxPendingUploadMetrics, skip metrics data for topology:CLUSTER [INFO 2016-03-15 12:47:59 c.a.j.d.n.TopologyMetricsRunnable:552 TopologyMetricsRunnable] refresh topologies, cost:63 [INFO 2016-03-15 12:48:00 c.a.j.d.n.ServiceHandler:1256 pool-10-thread-34] Received topology metrics:NIMBUS [ERROR 2016-03-15 12:48:00 c.a.j.d.n.TopologyMetricsRunnable:689 TopologyMetricsRunnable] exceeding maxPendingUploadMetrics, skip metrics data for topology:NIMBUS [INFO 2016-03-15 12:48:06 c.a.j.d.n.ServiceHandler:1256 pool-10-thread-38] Received topology metrics:move-drpc-2-1457942534 [INFO 2016-03-15 12:48:06 c.a.j.d.n.TopologyMetricsRunnable:356 TopologyMetricsRunnable] register metrics, topology:CLUSTER, size:7, cost:1 [ERROR 2016-03-15 12:48:06 c.a.j.d.n.TopologyMetricsRunnable:689 TopologyMetricsRunnable] exceeding maxPendingUploadMetrics, skip metrics data for topology:move-drpc-2-1457942534 [INFO 2016-03-15 12:48:23 c.a.j.d.s.Httpserver:138 pool-7-thread-1] Receive command {cmd=showLog, log=nimbus.log, pos=2392064} [INFO 2016-03-15 12:48:23 c.a.j.d.s.Httpserver:249 pool-7-thread-1] logview /data/softwares/jstorm/logs/./nimbus.log, position=2392064, size=32768 [INFO 2016-03-15 12:48:37 c.a.j.d.n.ServiceHandler:1256 pool-10-thread-49] Received topology metrics:move-jstorm-1-1457942488 [INFO 2016-03-15 12:48:37 c.a.j.d.n.TopologyMetricsRunnable:356 TopologyMetricsRunnable] register metrics, topology:CLUSTER, size:7, cost:0 [ERROR 2016-03-15 12:48:37 c.a.j.d.n.TopologyMetricsRunnable:689 TopologyMetricsRunnable] exceeding maxPendingUploadMetrics, skip metrics data for topology:move-jstorm-1-1457942488 [INFO 2016-03-15 12:48:55 c.a.j.d.n.TopologyMetricsRunnable:175 pool-6-thread-1] cluster metrics force upload. [INFO 2016-03-15 12:48:55 c.a.j.d.n.TopologyMetricsRunnable:610 pool-6-thread-1] send update event for cluster metrics, size : 7 [ERROR 2016-03-15 12:48:56 c.a.j.d.n.TopologyMetricsRunnable:689 TopologyMetricsRunnable] exceeding maxPendingUploadMetrics, skip metrics data for topology:CLUSTER [INFO 2016-03-15 12:48:59 c.a.j.d.n.TopologyMetricsRunnable:552 TopologyMetricsRunnable] refresh topologies, cost:93 [INFO 2016-03-15 12:49:00 c.a.j.d.n.ServiceHandler:1256 pool-10-thread-55] Received topology metrics:NIMBUS [ERROR 2016-03-15 12:49:00 c.a.j.d.n.TopologyMetricsRunnable:689 TopologyMetricsRunnable] exceeding maxPendingUploadMetrics, skip metrics data for topology:NIMBUS [INFO 2016-03-15 12:49:06 c.a.j.d.n.ServiceHandler:1256 pool-10-thread-59] Received topology metrics:move-drpc-2-1457942534 [INFO 2016-03-15 12:49:06 c.a.j.d.n.TopologyMetricsRunnable:356 TopologyMetricsRunnable] register metrics, topology:CLUSTER, size:7, cost:0 [ERROR 2016-03-15 12:49:06 c.a.j.d.n.TopologyMetricsRunnable:689 TopologyMetricsRunnable] exceeding maxPendingUploadMetrics, skip metrics data for topology:move-drpc-2-1457942534

hammertank commented 8 years ago

@shijiyu ，不好意思，这个我没遇见过。需要阿里的大神们来回答了~

longdafeng commented 8 years ago

drpc的问题，一般

(1) 确定drpc server是否接受到消息， 如果没有接收到，则是你发送drpc请求时，没有设置好drpc 服务器 （2） 确定应用的spout 是否接收到消息， 2.1 在应用中打开日志输出， topology.debug/topology.debug.recv.tuple 2个选择都设置为true， 你检查spout的日志是否接收到日志。如果没有接收到， 则需要重启topology，确认spout启动前，drpc server已经启动， spout启动后可以连接上drpc 2.2 如果spout工作正常， 得看看是哪个bolt 消息不完整

（3） 你什么版本的jstorm， 2.1.1吗？建议用2.1.1 试试

另外建议不要用drpc， 因为太慢了， 推荐你自己实现springspout 就好了，提供web service 服务，快很多。

johnnian commented 7 years ago

@longdafeng 

另外建议不要用drpc， 因为太慢了， 推荐你自己实现springspout 就好了，提供web service 服务，快很多。

您好～ 刚接触 JStorm，想要实现让外部主动触发Task，看到Storm可以用DRPC实现，您说效率比较低，可以自己实现WebSerice，请问，有没有一些例子可以参考？谢谢