OryxProject / oryx

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
http://oryx.io
Apache License 2.0
1.79k stars 405 forks source link

./oryx-run.sh serving #284

Closed pinguo-xudianyang closed 8 years ago

pinguo-xudianyang commented 8 years ago

kafka-brokers = "hadoop-slave-1:9092,hadoop-slave-2:9092,phpboy:9092" zk-servers = "10.200.143.204:4180,10.47.100.215:4180,10.46.75.247:4180" hdfs-base = "hdfs://hadoop-slave-1:9000/Oryx"

oryx { id = "ALSExample" als { rescorer-provider-class = null } input-topic { broker = ${kafka-brokers} lock = { master = ${zk-servers} } } update-topic { broker = ${kafka-brokers} lock = { master = ${zk-servers} } } batch { streaming { generation-interval-sec = 300 num-executors = 4 executor-cores = 8 executor-memory = "4g" } update-class = "com.cloudera.oryx.app.batch.mllib.als.ALSUpdate" storage { data-dir = ${hdfs-base}"/data/" model-dir = ${hdfs-base}"/model/" } ui { port = 4040 } } speed { model-manager-class = "com.cloudera.oryx.app.speed.als.ALSSpeedModelManager" ui { port = 4041 } } serving { model-manager-class = "com.cloudera.oryx.app.serving.als.model.ALSServingModelManager" application-resources = "com.cloudera.oryx.app.serving,com.cloudera.oryx.app.serving.als" api { port = 8080 } } }

2016-04-11 11:17:54,194 INFO ConsumerFetcherManager:68 [ConsumerFetcherManager-1460344661151] Added fetcher for partitions ArrayBuffer() 2016-04-11 11:17:54,401 INFO VerifiableProperties:68 Verifying properties 2016-04-11 11:17:54,402 INFO VerifiableProperties:68 Property client.id is overridden to OryxGroup-ServingLayer-1460344660862 2016-04-11 11:17:54,402 INFO VerifiableProperties:68 Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092 2016-04-11 11:17:54,402 INFO VerifiableProperties:68 Property request.timeout.ms is overridden to 30000 2016-04-11 11:17:54,403 INFO ClientUtils$:68 Fetching metadata from broker id:1,host:localhost,port:9092 with correlation id 56 for 1 topic(s) Set(OryxUpdate) 2016-04-11 11:17:54,403 INFO SyncProducer:68 Connected to localhost:9092 for producing 2016-04-11 11:17:54,407 INFO SyncProducer:68 Disconnecting from localhost:9092 2016-04-11 11:17:54,407 INFO ConsumerFetcherManager:68 [ConsumerFetcherManager-1460344661151] Added fetcher for partitions ArrayBuffer() 2016-04-11 11:17:54,623 INFO VerifiableProperties:68 Verifying properties 2016-04-11 11:17:54,624 INFO VerifiableProperties:68 Property client.id is overridden to OryxGroup-ServingLayer-1460344660862 2016-04-11 11:17:54,624 INFO VerifiableProperties:68 Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092 2016-04-11 11:17:54,624 INFO VerifiableProperties:68 Property request.timeout.ms is overridden to 30000 2016-04-11 11:17:54,624 INFO ClientUtils$:68 Fetching metadata from broker id:2,host:localhost,port:9092 with correlation id 57 for 1 topic(s) Set(OryxUpdate) 2016-04-11 11:17:54,626 INFO SyncProducer:68 Connected to localhost:9092 for producing 2016-04-11 11:17:54,628 INFO SyncProducer:68 Disconnecting from localhost:9092 2016-04-11 11:17:54,629 INFO ConsumerFetcherManager:68 [ConsumerFetcherManager-1460344661151] Added fetcher for partitions ArrayBuffer() 2016-04-11 11:17:54,835 INFO VerifiableProperties:68 Verifying properties 2016-04-11 11:17:54,835 INFO VerifiableProperties:68 Property client.id is overridden to OryxGroup-ServingLayer-1460344660862 2016-04-11 11:17:54,835 INFO VerifiableProperties:68 Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092 2016-04-11 11:17:54,835 INFO VerifiableProperties:68 Property request.timeout.ms is overridden to 30000 2016-04-11 11:17:54,836 INFO ClientUtils$:68 Fetching metadata from broker id:0,host:localhost,port:9092 with correlation id 58 for 1 topic(s) Set(OryxUpdate) 2016-04-11 11:17:54,836 INFO SyncProducer:68 Connected to localhost:9092 for producing 2016-04-11 11:17:54,838 INFO SyncProducer:68 Disconnecting from localhost:9092 2016-04-11 11:17:54,839 INFO ConsumerFetcherManager:68 [ConsumerFetcherManager-1460344661151] Added fetcher for partitions ArrayBuffer() 2016-04-11 11:17:55,047 INFO VerifiableProperties:68 Verifying properties 2016-04-11 11:17:55,047 INFO VerifiableProperties:68 Property client.id is overridden to OryxGroup-ServingLayer-1460344660862 2016-04-11 11:17:55,047 INFO VerifiableProperties:68 Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092 2016-04-11 11:17:55,048 INFO VerifiableProperties:68 Property request.timeout.ms is overridden to 30000 2016-04-11 11:17:55,048 INFO ClientUtils$:68 Fetching metadata from broker id:1,host:localhost,port:9092 with correlation id 59 for 1 topic(s) Set(OryxUpdate) 2016-04-11 11:17:55,048 INFO SyncProducer:68 Connected to localhost:9092 for producing 2016-04-11 11:17:55,051 INFO SyncProducer:68 Disconnecting from localhost:9092

pinguo-xudianyang commented 8 years ago

many these INFO LOG for console output,what does this matter?

srowen commented 8 years ago

I'm not clear what your question is? what is the problem? This looks normal.

pinguo-xudianyang commented 8 years ago

ok,I setup all dependency in a manual way, such as hadoop,kafka,zookeeper,spark. I keep on.