Open nfangxu opened 3 months ago
nfangxu
commented
3 months ago 相关版本信息如下（spu 因为改动过，所以使用的是 0.8.0b0 版本）：
# pip3 list | grep secretflow
secretflow 1.5.0b0
secretflow-rayfed 0.2.1a1
secretflow-serving-lib 0.3.0.dev20240320
# pip3 list | grep spu
spu 0.8.0b0
ian-huu
commented
3 months ago 需要在脚本最后加上 sf.shutdown()，可能会看到报错 AttributeError: 'NoneType' object has no attribute 'get_job_name'，这个是已知问题，会尽快修复。
此外，为了保证在 shutdown 之前执行完任务，建议在 shutdown 之前加上 sf.wait(某个结果)，比如：
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--party", default="", help="party id")
args = parser.parse_args()
# Ray init
ray_init(args.party)
spu_device = spu_init()
pyus = [sf.PYU("client"), sf.PYU("server")]
print("=========================Start")
spu_objs = []
for pyu in pyus:
obj = pyu(get_data)(1).to(spu_device)
spu_objs.append(obj)
print("=========================Success")
sf.wait(spu_objs)
sf.shutdown()
magic-hya
commented
1 month ago 问题我也遇到了，加了shutdown也结束不了，还有rayfed没有结束，下面是我用ctrl+C才结束的，正常代码无法结束
[^C2024-08-02 03:19:57.864 WARNING api.py:60 [alice] -- [Anonymous_job] Stop signal received (e.g. via SIGINT/Ctrl+C), try to shutdown fed. Press CTRL+C (or send SIGINT/SIGKILL/SIGTERM) to skip.
2024-08-02 03:19:57.865 WARNING api.py:325 [alice] -- [Anonymous_job] Shutdowning rayfed unintendedly...
2024-08-02 03:19:57.865 INFO api.py:337 [alice] -- [Anonymous_job] Wait for data sending.
2024-08-02 03:19:57.867 INFO message_queue.py:70 [alice] -- [Anonymous_job] Notify message polling thread[DataSendingQueueThread] to exit.
2024-08-02 03:19:57.876 INFO message_queue.py:100 [alice] -- [Anonymous_job] The message polling thread[DataSendingQueueThread] was exited.
2024-08-02 03:19:57.876 INFO message_queue.py:70 [alice] -- [Anonymous_job] Notify message polling thread[ErrorSendingQueueThread] to exit.
2024-08-02 03:19:57.943 INFO message_queue.py:100 [alice] -- [Anonymous_job] The message polling thread[ErrorSendingQueueThread] was exited.
2024-08-02 03:19:57.944 INFO api.py:352 [alice] -- [Anonymous_job] Shutdowned rayfed.
2024-08-02 03:19:57.944 CRITICAL api.py:356 [alice] -- [Anonymous_job] Exit now due to the previous error.]
Issue Type
Bug
Source
binary
Secretflow Version
1.5.0b0
OS Platform and Distribution
Centos 7.9.2009
Python version
3.10.14
Bazel version
No response
GCC/Compiler version
No response
What happend and What you expected to happen.
两台机器分布启动 ray 集群：
分别执行：
执行完毕输出日志如下：
Reproduction code to reproduce the issue.
demo.py代码如下：