search

secretflow / secretpad

SecretPad is a privacy-preserving computing web platform based on the Kuscia framework, designed to provide easy access to privacy-preserving data intelligence and machine learning functions.
https://www.secretflow.org.cn
Apache License 2.0
41 stars 23 forks source link

二元操作组件使用问题 #80

Open wangzeyu135798 opened 5 months ago

wangzeyu135798 commented 5 months ago

Issue Type

Running

Have you searched for existing documents and issues?

Yes

OS Platform and Distribution

centos7

All_in_one Version

0.5

Kuscia Version

0.6

What happend and What you expected to happen.

使用了两个样本表组件，一个隐私求交组件，然后对隐私求交的结果进行二元操作，出现了错误；
样本表使用的是两组不同的数据，并且两组数据的列名也互相没有重复。

Log output.

2024-06-03 16:31:10 INFO the jobId=llkv, taskId=llkv-qevmexgz-node-4 start ...
2024-06-03 16:31:33 INFO the jobId=llkv, taskId=llkv-qevmexgz-node-4 failed: party testa failed msg: container[secretflow] terminated state reason "Error", message: "<class 'secretflow.data.core.agent.PartitionAgent'>, num_returns=1, args len: 1, kwargs len: 0.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:28.195 DEBUG link.py:93 [testa] -- [Anonymous_job] Getting data for 48 from 47#0 of testc
\x1b[2m\x1b[33m(raylet)\x1b[0m [2024-06-03 08:31:28,944 I 476 476] logging.cc:230: Set ray log level from environment variable RAY_BACKEND_LOG_LEVEL to -1
2024-06-03 08:31:30.734 DEBUG proxy.py:45 [testa] -- [Anonymous_job] Run method __len__ of actor <class 'secretflow.data.core.agent.PartitionAgent'>, num_returns=1, args len: 1, kwargs len: 0.
2024-06-03 08:31:30.734 DEBUG utils.py:63 [testa] -- [Anonymous_job] Insert fed object, arg.party=testa
2024-06-03 08:31:30.734 DEBUG fed_actor.py:104 [testa] -- [Anonymous_job] Actor method call: __len__, num_returns: 1
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.728 DEBUG link.py:114 [testa] -- [Anonymous_job] Received data for 48 from 47#0.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.728 DEBUG link.py:120 [testa] -- [Anonymous_job] Getted data for 48 from 47#0 of testc.
2024-06-03 08:31:30.880 DEBUG proxy.py:45 [testa] -- [Anonymous_job] Run method columns of actor <class 'secretflow.data.core.agent.PartitionAgent'>, num_returns=1, args len: 1, kwargs len: 0.
2024-06-03 08:31:30.892 DEBUG proxy.py:45 [testa] -- [Anonymous_job] Run method columns of actor <class 'secretflow.data.core.agent.PartitionAgent'>, num_returns=1, args len: 1, kwargs len: 0.
2024-06-03 08:31:30.892 DEBUG utils.py:63 [testa] -- [Anonymous_job] Insert fed object, arg.party=testa
2024-06-03 08:31:30.892 DEBUG fed_actor.py:104 [testa] -- [Anonymous_job] Actor method call: columns, num_returns: 1
2024-06-03 08:31:30.898 DEBUG proxy.py:45 [testa] -- [Anonymous_job] Run method shape of actor <class 'secretflow.data.core.agent.PartitionAgent'>, num_returns=1, args len: 1, kwargs len: 0.
2024-06-03 08:31:30.909 DEBUG proxy.py:45 [testa] -- [Anonymous_job] Run method shape of actor <class 'secretflow.data.core.agent.PartitionAgent'>, num_returns=1, args len: 1, kwargs len: 0.
2024-06-03 08:31:30.909 DEBUG utils.py:63 [testa] -- [Anonymous_job] Insert fed object, arg.party=testa
2024-06-03 08:31:30.909 DEBUG fed_actor.py:104 [testa] -- [Anonymous_job] Actor method call: shape, num_returns: 1
2024-06-03 08:31:30.911 INFO data_utils.py:367 [testa] -- [Anonymous_job] loaded VDataFrame, shape (400000, 4)
2024-06-03 08:31:30.911 ERROR component.py:1089 [testa] -- [Anonymous_job] eval on domain: \"preprocessing\"
name: \"binary_op\"
version: \"0.0.2\"
attr_paths: \"input/in_ds/f1\"
attr_paths: \"input/in_ds/f2\"
attr_paths: \"binary_op\"
attr_paths: \"new_feature_name\"
attr_paths: \"as_label\"
attrs {
  ss: \"Column2\"
}
attrs {
  ss: \"Column4\"
}
attrs {
  s: \"+\"
}
attrs {
  s: \"new_column\"
}
attrs {
  is_na: true
}
inputs {
  name: \"llkv-qevmexgz-node-6-output-0\"
  type: \"sf.table.vertical_table\"
  system_info {
  }
  meta {
    type_url: \"type.googleapis.com/secretflow.spec.v1.VerticalTable\"
    value: \"\
1\
\
Column5099\\022\\007Column4\\022\\007Column5\\\"\\003int*\\005float*\\005float\
1\
\
Column7099\\022\\007Column2\\022\\007Column3\\\"\\003int*\\005float*\\005float\\020\\200\\265\\030\"
  }
  data_refs {
    uri: \"llkv-qevmexgz-node-6-output-0\"
    party: \"testc\"
    format: \"csv\"
  }
  data_refs {
    uri: \"llkv-qevmexgz-node-6-output-0\"
    party: \"testa\"
    format: \"csv\"
  }
}
output_uris: \"llkv-qevmexgz-node-4-output-0\"
output_uris: \"llkv-qevmexgz-node-4-output-1\"
 failed, error <>
2024-06-03 08:31:30.911 INFO api.py:342 [testa] -- [Anonymous_job] Shutdowning rayfed intendedly...
2024-06-03 08:31:30.911 INFO api.py:356 [testa] -- [Anonymous_job] No wait for data sending.
2024-06-03 08:31:30.913 DEBUG message_queue.py:98 [testa] -- [Anonymous_job] Killing thread[DataSendingQueueThread].
2024-06-03 08:31:30.913 INFO message_queue.py:72 [testa] -- [Anonymous_job] Notify message polling thread[DataSendingQueueThread] to exit.
2024-06-03 08:31:30.913 DEBUG message_queue.py:98 [testa] -- [Anonymous_job] Killing thread[ErrorSendingQueueThread].
2024-06-03 08:31:30.913 INFO message_queue.py:72 [testa] -- [Anonymous_job] Notify message polling thread[ErrorSendingQueueThread] to exit.
2024-06-03 08:31:30.913 INFO api.py:384 [testa] -- [Anonymous_job] Shutdowned rayfed.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.881 DEBUG barriers.py:397 [testa] -- [Anonymous_job] Sending send data to seq_id 50 of testc from 49#0 without credentials.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.881 DEBUG barriers.py:408 [testa] -- [Anonymous_job] Succeeded to send data to seq_id 50 of testc from 49#0. Response is True
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.882 DEBUG link.py:93 [testa] -- [Anonymous_job] Getting data for 52 from 51#0 of testc
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.891 DEBUG link.py:114 [testa] -- [Anonymous_job] Received data for 52 from 51#0.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.891 DEBUG link.py:120 [testa] -- [Anonymous_job] Getted data for 52 from 51#0 of testc.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.899 DEBUG barriers.py:397 [testa] -- [Anonymous_job] Sending send data to seq_id 54 of testc from 53#0 without credentials.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.899 DEBUG barriers.py:408 [testa] -- [Anonymous_job] Succeeded to send data to seq_id 54 of testc from 53#0. Response is True
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.900 DEBUG link.py:93 [testa] -- [Anonymous_job] Getting data for 56 from 55#0 of testc
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.908 DEBUG link.py:114 [testa] -- [Anonymous_job] Received data for 56 from 55#0.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.908 DEBUG link.py:120 [testa] -- [Anonymous_job] Getted data for 56 from 55#0 of testc.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.913 DEBUG barriers.py:397 [testa] -- [Anonymous_job] Sending send data to seq_id 58 of testc from 57#0 without credentials.
\x1b[2m\x1b[36m(SenderReceiverProxyActor pid=313)\x1b[0m 2024-06-03 08:31:30.913 DEBUG barriers.py:408 [testa] -- [Anonymous_job] Succeeded to send data to seq_id 58 of testc from 57#0. Response is True
Traceback (most recent call last):
  File \"/usr/local/lib/python3.8/runpy.py\", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File \"/usr/local/lib/python3.8/runpy.py\", line 87, in _run_code
    exec(code, run_globals)
  File \"/usr/local/lib/python3.8/site-packages/secretflow/kuscia/entry.py\", line 530, in <module>
    main()
  File \"/usr/local/lib/python3.8/site-packages/click/core.py\", line 1157, in __call__
    return self.main(*args, **kwargs)
  File \"/usr/local/lib/python3.8/site-packages/click/core.py\", line 1078, in main
    rv = self.invoke(ctx)
  File \"/usr/local/lib/python3.8/site-packages/click/core.py\", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File \"/usr/local/lib/python3.8/site-packages/click/core.py\", line 783, in invoke
    return __callback(*args, **kwargs)
  File \"/usr/local/lib/python3.8/site-packages/secretflow/kuscia/entry.py\", line 510, in main
    res = comp_eval(sf_node_eval_param, storage_config, sf_cluster_config)
  File \"/usr/local/lib/python3.8/site-packages/secretflow/component/entry.py\", line 164, in comp_eval
    res = comp.eval(
  File \"/usr/local/lib/python3.8/site-packages/secretflow/component/component.py\", line 1091, in eval
    raise e from None
  File \"/usr/local/lib/python3.8/site-packages/secretflow/component/component.py\", line 1086, in eval
    ret = self.__eval_callback(**kwargs)
  File \"/usr/local/lib/python3.8/site-packages/secretflow/component/preprocessing/unified_single_party_ops/binary_op.py\", line 150, in binary_op_eval_fn
    (output_dd, model_dd, _) = v_preprocessing_transform(
  File \"/usr/local/lib/python3.8/site-packages/secretflow/component/preprocessing/core/table_utils.py\", line 69, in v_preprocessing_transform
    assert len(trans.partitions) == 1
AssertionError
"
haha-zwx-ooo commented 5 months ago

二元操作是单边组件，需要选同一方的特征列