python ray运行报错 - Githubissues

hecheng64 commented 3 years ago

按照新手指南那边配置，搭建好ray环境。自身创建ray相关python例子，运行正常的，但是跑我们例子跑好久还出错 Job aborted due to stage failure: Task 0 in stage 583.0 failed 1 times, most recent failure: Lost task 0.0 in stage 583.0 (TID 564, localhost, executor driver): java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.(Socket.java:434) at java.net.Socket.(Socket.java:211) at tech.mlsql.arrow.python.runner.SparkSocketRunner.readFromStreamWithArrow(SparkSocketRunner.scala:47) at tech.mlsql.ets.Ray$$anonfun$15.apply(Ray.scala:257) at tech.mlsql.ets.Ray$$anonfun$15.apply(Ray.scala:253) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
例子代码： set mockData=''' {"title":"第一","body":"内容1"} {"title":"第二","body":"内容2"} {"title":"第三","body":"内容3"} '''; load jsonStr.mockData as data; !python env "PYTHON_ENV=source /root/miniconda3/bin/activate dev"; !python conf "runIn=driver"; !python conf "schema=st(field(title,string),field(body,string))"; !python conf "dataMode=data";

!ray on data '''

import ray from pyjava.api.mlsql import RayContext import numpy as np ray.init(redis_address="192.168.6.180:17997", redis_password="123456") ray_context = RayContext.connect(globals(), None) def echo(row): row1 = {} row1["title"]="jackm" row1["body"]= row["body"] return row1

ray_context.foreach(echo)

''' named newdata;

select * from newdata as output;

hecheng64 commented 3 years ago

ray.init 这个要执行？我看有些例子要写，有些例子不用写。

hecheng64 commented 3 years ago

换成最新版本也是同样问题，可能是我ray需要环境有问题吧。目前mlsql-engine-2.4-2.1.0 部署local模式，ray部署集群模式。请大神帮忙协助下

allwefantasy commented 3 years ago

mlsql-engine 使用 2.1.0-SNAPSHOT版本
Ray 使用 1.3.0版本

将代码：

ray.init(redis_address="192.168.6.180:17997", redis_password="123456")
ray_context = RayContext.connect(globals(), None)

修改成

ray_context = RayContext.connect(globals(),"192.168.6.180:10001")

PS: ,如果你使用2.1.0版本，那么这个版本暂时锁定了 ray 0.8.0版本。版本需要配套。然后连接采用下面的方式：

ray_context = RayContext.connect(globals(),"192.168.6.180:17997")

并且Ray集群不能设置密码；

allwefantasy commented 3 years ago

driver 端必要的依赖：

pip install Cython
pip install ray==1.3.0
pip install aiohttp psutil setproctitle grpcio pandas xlsxwriter
pip install watchdog requests click uuid sfcli
pip install pyjava

allwefantasy commented 3 years ago

复杂的环境描述： http://blog.mlsql.tech/blog/play-python-ray.html

hecheng64 commented 3 years ago

非常感谢，哪里可以找到你们默认demo脚本吗？我看有博客文字介绍说，你们提供MLSQL Analysis 有默认体验引擎和示例脚本了但是目前好像都打不开。

allwefantasy commented 3 years ago

非常感谢，哪里可以找到你们默认demo脚本吗？我看有博客文字介绍说，你们提供MLSQL Analysis 有默认体验引擎和示例脚本了但是目前好像都打不开。

是的；可以看

官网： mlsql.ai
博客： http://blog.mlsql.tech/
文档： https://docs.mlsql.tech/mlsql-stack/
视频演示： https://space.bilibili.com/22610047/video

hecheng64 commented 3 years ago

你好，你提供那些链接我都看过，当我执行一些例子时候拷贝文章里面代码，执行有时候会出错，想知道你们有没有demo集麻烦共享，提供下。就是demo工程脚本

byzer-org / byzer-lang

python ray运行报错 #1456