search

FederatedAI / FATE

An Industrial Grade Federated Learning Framework
Apache License 2.0
5.71k stars 1.55k forks source link

an error occurred when using high dimensional data for HeteroPearson algorithm #4083

Closed timber-vine closed 4 months ago

timber-vine commented 2 years ago

Describe the bug Hello, an error occurred when using high dimensional data for HeteroPearson algorithm, The dimension of data is 300000 lines and 100 columns. Task is succeed when using low dimension data.

Task log

[INFO] [2022-07-06 10:22:14,644] [202207061020403060030] [142648:140652751726400] - [base_worker.run] [line:98]: log level: INFO
[INFO] [2022-07-06 10:22:14,644] [202207061020403060030] [142648:140652751726400] - [base_worker.run] [line:100]: VIRTUAL_ENV: /opt/app-root
[INFO] [2022-07-06 10:22:14,644] [202207061020403060030] [142648:140652751726400] - [base_worker.run] [line:100]: PROCESS_ROLE: worker
[INFO] [2022-07-06 10:22:14,644] [202207061020403060030] [142648:140652751726400] - [base_worker.run] [line:100]: FATE_DEPLOY_BASE: None
[INFO] [2022-07-06 10:22:14,644] [202207061020403060030] [142648:140652751726400] - [base_worker.run] [line:100]: SPARK_HOME: None
[INFO] [2022-07-06 10:22:14,644] [202207061020403060030] [142648:140652751726400] - [base_worker.run] [line:100]: FATE_JOB_ID: 202207061020403060030
[INFO] [2022-07-06 10:22:14,645] [202207061020403060030] [142648:140652751726400] - [base_worker.run] [line:100]: PYTHONPATH: /data/projects/fate/fate/python/federatedml:/opt/rh/rh-nodejs10/root/usr/lib/python2.7/site-packages:$PYTHONPATH:/data/projects/fate/fate/python:/data/projects/fate/eggroll/python:/data/projects/fate/fateflow/python:/data/projects/fate/fate/python/fate_client
[INFO] [2022-07-06 10:22:14,700] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: use_registry is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,701] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: use_deserialize_safe_module is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,701] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: dependent_distribution is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,701] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: encrypt_password is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,701] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: encrypt_module is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,702] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: private_key is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,702] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: fateflow is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,702] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: database is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,702] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: enable_model_store is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:14,702] [202207061020403060030] [142648:140652751726400] - [service_registry.load] [line:52]: zookeeper is fate flow server start config, pass load
[INFO] [2022-07-06 10:22:15,188] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update federation engine STANDALONE fate_on_standalone registration information
[INFO] [2022-07-06 10:22:15,194] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update federation engine EGGROLL fate_on_eggroll registration information
[INFO] [2022-07-06 10:22:15,200] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update federation engine RABBITMQ fate_on_spark registration information
[INFO] [2022-07-06 10:22:15,205] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update federation engine PULSAR fate_on_spark registration information
[INFO] [2022-07-06 10:22:15,212] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update computing engine STANDALONE fate_on_standalone registration information
[INFO] [2022-07-06 10:22:15,217] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update computing engine EGGROLL fate_on_eggroll registration information
[INFO] [2022-07-06 10:22:15,222] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update computing engine SPARK fate_on_spark registration information
[INFO] [2022-07-06 10:22:15,228] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update computing engine LINKIS_SPARK fate_on_spark registration information
[INFO] [2022-07-06 10:22:15,233] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update storage engine STANDALONE fate_on_standalone registration information
[INFO] [2022-07-06 10:22:15,239] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update storage engine EGGROLL fate_on_eggroll registration information
[INFO] [2022-07-06 10:22:15,244] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update storage engine HDFS fate_on_spark registration information
[INFO] [2022-07-06 10:22:15,250] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update storage engine HIVE fate_on_spark registration information
[INFO] [2022-07-06 10:22:15,256] [202207061020403060030] [142648:140652751726400] - [resource_manager.register_engine] [line:64]: update storage engine LINKIS_HIVE fate_on_spark registration information
[INFO] [2022-07-06 10:22:15,256] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:55]: run hetero_pearson_0 202207061020403060030_hetero_pearson_0 0 on guest 9998 task
[INFO] [2022-07-06 10:22:15,277] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:76]: user name:
[INFO] [2022-07-06 10:22:15,279] [202207061020403060030] [142648:140652751726400] - [task_base_worker.report_task_info_to_driver] [line:124]: report TaskExecutor 202207061020403060030_hetero_pearson_0 0 guest 9998 to driver:
{'job_id': '202207061020403060030', 'component_name': 'hetero_pearson_0', 'task_id': '202207061020403060030_hetero_pearson_0', 'task_version': '0', 'role': 'guest', 'party_id': '9998', 'run_ip': '10.244.1.56', 'run_pid': 142648, 'party_status': 'running'}
[INFO] [2022-07-06 10:22:15,279] [202207061020403060030] [142648:140652751726400] - [control_client.report_task] [line:41]: request update job 202207061020403060030 task 202207061020403060030_hetero_pearson_0 0 on guest 9998
[INFO] [2022-07-06 10:22:15,350] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:109]: previous_components_parameters:
{}
[INFO] [2022-07-06 10:22:15,779] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:117]: component parameters on party:
{
    "dsl_version": "2",
    "initiator": {
        "party_id": 9998,
        "role": "guest"
    },
    "job_parameters": {
        "common": {
            "adaptation_parameters": {
                "if_initiator_baseline": true,
                "request_task_cores": 8,
                "task_cores_per_node": 8,
                "task_memory_per_node": 0,
                "task_nodes": 1
            },
            "auto_retries": 0,
            "auto_retry_delay": 1,
            "computing_engine": "EGGROLL",
            "computing_partitions": 8,
            "eggroll_run": {},
            "engines_address": {},
            "federated_mode": "MULTIPLE",
            "federated_status_collect_type": "PUSH",
            "inheritance_info": {},
            "job_type": "train",
            "model_id": "arbiter-9999#guest-9998#host-9999#model",
            "model_version": "202207061020403060030",
            "pulsar_run": {},
            "rabbitmq_run": {},
            "spark_run": {},
            "task_cores": 8,
            "task_parallelism": 1
        },
        "role": {
            "guest": {
                "0": {}
            }
        }
    },
    "role": {
        "guest": [
            9998
        ],
        "host": [
            9999
        ]
    },
    "local": {
        "role": "guest",
        "party_id": 9998
    },
    "module": "HeteroPearson",
    "CodePath": "HeteroPearson",
    "ComponentParam": {
        "column_names": [
            "x1",
            "x2",
            "x3",
            "x4",
            "x5",
            "x6",
            "x7",
            "x8",
            "x9",
            "x10"
        ],
        "column_indexes": [],
        "cross_parties": true,
        "need_run": true,
        "use_mix_rand": false,
        "calc_local_vif": true,
        "_name": "HeteroPearson#hetero_pearson_0",
        "_feeded_deprecated_params": [],
        "_user_feeded_params": [
            "column_names",
            "use_mix_rand",
            "need_run",
            "calc_local_vif",
            "cross_parties"
        ],
        "_is_raw_conf": false
    }
}
[INFO] [2022-07-06 10:22:15,848] [202207061020403060030] [142648:140652751726400] - [_session.__init__] [line:53]: using engines: {'computing': 'EGGROLL', 'federation': 'EGGROLL', 'storage': 'EGGROLL', 'federated_mode': 'MULTIPLE'}
[INFO] [2022-07-06 10:22:15,848] [202207061020403060030] [142648:140652751726400] - [_session.__init__] [line:68]: create manager session 202207061020403060030_hetero_pearson_0_0_guest_9998
[INFO] [2022-07-06 10:22:15,852] [202207061020403060030] [142648:140652751726400] - [db_models.init_database_tables] [line:84]: start create table SessionRecord
[INFO] [2022-07-06 10:22:15,856] [202207061020403060030] [142648:140652751726400] - [db_models.init_database_tables] [line:87]: create table success: SessionRecord
[INFO] [2022-07-06 10:22:15,857] [202207061020403060030] [142648:140652751726400] - [db_models.init_database_tables] [line:84]: start create table StorageConnectorModel
[INFO] [2022-07-06 10:22:15,860] [202207061020403060030] [142648:140652751726400] - [db_models.init_database_tables] [line:87]: create table success: StorageConnectorModel
[INFO] [2022-07-06 10:22:15,860] [202207061020403060030] [142648:140652751726400] - [db_models.init_database_tables] [line:84]: start create table StorageTableMetaModel
[INFO] [2022-07-06 10:22:15,865] [202207061020403060030] [142648:140652751726400] - [db_models.init_database_tables] [line:87]: create table success: StorageTableMetaModel
[INFO] [2022-07-06 10:22:15,866] [202207061020403060030] [142648:140652751726400] - [_session.save_record] [line:345]: try to save session record for manager 202207061020403060030_hetero_pearson_0_0_guest_9998, computing EGGROLL 202207061020403060030_hetero_pearson_0_0_guest_9998
[INFO] [2022-07-06 10:22:15,870] [202207061020403060030] [142648:140652751726400] - [_session.save_record] [line:364]: save session record for manager 202207061020403060030_hetero_pearson_0_0_guest_9998, computing EGGROLL 202207061020403060030_hetero_pearson_0_0_guest_9998 successfully
[INFO] [2022-07-06 10:22:18,594] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:143]: run hetero_pearson_0 202207061020403060030_hetero_pearson_0 0 on guest 9998 task
[INFO] [2022-07-06 10:22:18,595] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:144]: component parameters on party:
{
    "dsl_version": "2",
    "initiator": {
        "party_id": 9998,
        "role": "guest"
    },
    "job_parameters": {
        "job_type": "train",
        "inheritance_info": {},
        "computing_engine": "EGGROLL",
        "federation_engine": "EGGROLL",
        "storage_engine": "EGGROLL",
        "engines_address": {
            "computing": {
                "cores_per_node": 20,
                "nodes": 1
            },
            "federation": {
                "host": "rollsite",
                "port": 9370
            },
            "storage": {
                "cores_per_node": 20,
                "nodes": 1
            }
        },
        "federated_mode": "MULTIPLE",
        "task_cores": 8,
        "task_parallelism": 1,
        "computing_partitions": 8,
        "federated_status_collect_type": "PUSH",
        "model_id": "arbiter-9999#guest-9998#host-9999#model",
        "model_version": "202207061020403060030",
        "auto_retries": 0,
        "auto_retry_delay": 1,
        "eggroll_run": {
            "eggroll.session.processors.per.node": 8
        },
        "spark_run": {},
        "rabbitmq_run": {},
        "pulsar_run": {},
        "adaptation_parameters": {
            "task_nodes": 1,
            "task_cores_per_node": 8,
            "task_memory_per_node": 0,
            "request_task_cores": 8,
            "if_initiator_baseline": false
        }
    },
    "role": {
        "guest": [
            9998
        ],
        "host": [
            9999
        ]
    },
    "local": {
        "role": "guest",
        "party_id": 9998
    },
    "module": "HeteroPearson",
    "CodePath": "HeteroPearson",
    "ComponentParam": {
        "column_names": [
            "x1",
            "x2",
            "x3",
            "x4",
            "x5",
            "x6",
            "x7",
            "x8",
            "x9",
            "x10"
        ],
        "column_indexes": [],
        "cross_parties": true,
        "need_run": true,
        "use_mix_rand": false,
        "calc_local_vif": true,
        "_name": "HeteroPearson#hetero_pearson_0",
        "_feeded_deprecated_params": [],
        "_user_feeded_params": [
            "column_names",
            "use_mix_rand",
            "need_run",
            "calc_local_vif",
            "cross_parties"
        ],
        "_is_raw_conf": false
    }
}
[INFO] [2022-07-06 10:22:18,595] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:145]: task input dsl {'data': {'data': ['reader_0.data']}}
[INFO] [2022-07-06 10:22:18,595] [202207061020403060030] [142648:140652751726400] - [tracker_client.get_output_data_info] [line:265]: Request read job 202207061020403060030 task 202207061020403060030_hetero_pearson_0 0 on guest 9998 data data info
[INFO] [2022-07-06 10:22:18,621] [202207061020403060030] [142648:140652751726400] - [task_executor.get_task_run_args] [line:339]: load computing table use 8
[INFO] [2022-07-06 10:22:24,184] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:156]: task input args {'data': {'reader_0': {'data': [<fate_arch.computing.eggroll._table.Table object at 0x7fec26bec828>]}}}
[INFO] [2022-07-06 10:22:24,186] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:194]: profile logging is disabled
[ERROR] [2022-07-06 11:33:10,709] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:243]: ('Failed to call command: CommandURI(_uri=v1/egg-pair/runTask) to endpoint: nodemanager-0:41349, caused by: ', <_Rendezvous of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Exception calling application: 

==== detail start, at 20220706.113309.735 ====
Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task
    value=self.functor_serdes.serialize(f(task)))
  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize
    return cloudpickle.dumps(_obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps
    cp.dump(obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump
    return Pickler.dump(self, obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce
    save(args)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce
    self._batch_setitems(dictitems)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems
    tmp = list(islice(it, self._BATCHSIZE))
RuntimeError: dictionary changed size during iteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_service.py", line 33, in call
    kwargs=getattr(command_request, '_kwargs'))
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 94, in dispatch
    raise e
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 91, in dispatch
    call_result = _method(_instance, *deserialized_args)
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 194, in wrapper
    raise RuntimeError(msg)
RuntimeError: 

==== detail start, at 20220706.113309.728 ====
Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task
    value=self.functor_serdes.serialize(f(task)))
  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize
    return cloudpickle.dumps(_obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps
    cp.dump(obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump
    return Pickler.dump(self, obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce
    save(args)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce
    self._batch_setitems(dictitems)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems
    tmp = list(islice(it, self._BATCHSIZE))
RuntimeError: dictionary changed size during iteration

==== detail end ====

==== detail end ====

"
    debug_error_string = "{"created":"@1657078389.737481467","description":"Error received from peer ipv4:10.244.2.5:41349","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Exception calling application: \n\n==== detail start, at 20220706.113309.735 ====\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task\n    value=self.functor_serdes.serialize(f(task)))\n  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize\n    return cloudpickle.dumps(_obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps\n    cp.dump(obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump\n    return Pickler.dump(self, obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump\n    self.save(obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce\n    save(args)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple\n    save(element)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce\n    self._batch_setitems(dictitems)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems\n    tmp = list(islice(it, self._BATCHSIZE))\nRuntimeError: dictionary changed size during iteration\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_service.py", line 33, in call\n    kwargs=getattr(command_request, '_kwargs'))\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 94, in dispatch\n    raise e\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 91, in dispatch\n    call_result = _method(_instance, *deserialized_args)\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 194, in wrapper\n    raise RuntimeError(msg)\nRuntimeError: \n\n==== detail start, at 20220706.113309.728 ====\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task\n    value=self.functor_serdes.serialize(f(task)))\n  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize\n    return cloudpickle.dumps(_obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps\n    cp.dump(obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump\n    return Pickler.dump(self, obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump\n    self.save(obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce\n    save(args)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple\n    save(element)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce\n    self._batch_setitems(dictitems)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems\n    tmp = list(islice(it, self._BATCHSIZE))\nRuntimeError: dictionary changed size during iteration\n\n==== detail end ====\n\n\n\n==== detail end ====\n\n","grpc_status":2}"
>)
Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/client.py", line 84, in sync_send
    response = _command_stub.call(request.to_proto())
  File "/opt/app-root/lib/python3.6/site-packages/grpc/_channel.py", line 604, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/app-root/lib/python3.6/site-packages/grpc/_channel.py", line 506, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Exception calling application: 

==== detail start, at 20220706.113309.735 ====
Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task
    value=self.functor_serdes.serialize(f(task)))
  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize
    return cloudpickle.dumps(_obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps
    cp.dump(obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump
    return Pickler.dump(self, obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce
    save(args)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce
    self._batch_setitems(dictitems)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems
    tmp = list(islice(it, self._BATCHSIZE))
RuntimeError: dictionary changed size during iteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_service.py", line 33, in call
    kwargs=getattr(command_request, '_kwargs'))
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 94, in dispatch
    raise e
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 91, in dispatch
    call_result = _method(_instance, *deserialized_args)
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 194, in wrapper
    raise RuntimeError(msg)
RuntimeError: 

==== detail start, at 20220706.113309.728 ====
Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task
    value=self.functor_serdes.serialize(f(task)))
  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize
    return cloudpickle.dumps(_obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps
    cp.dump(obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump
    return Pickler.dump(self, obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce
    save(args)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce
    self._batch_setitems(dictitems)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems
    tmp = list(islice(it, self._BATCHSIZE))
RuntimeError: dictionary changed size during iteration

==== detail end ====

==== detail end ====

"
    debug_error_string = "{"created":"@1657078389.737481467","description":"Error received from peer ipv4:10.244.2.5:41349","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Exception calling application: \n\n==== detail start, at 20220706.113309.735 ====\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task\n    value=self.functor_serdes.serialize(f(task)))\n  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize\n    return cloudpickle.dumps(_obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps\n    cp.dump(obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump\n    return Pickler.dump(self, obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump\n    self.save(obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce\n    save(args)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple\n    save(element)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce\n    self._batch_setitems(dictitems)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems\n    tmp = list(islice(it, self._BATCHSIZE))\nRuntimeError: dictionary changed size during iteration\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_service.py", line 33, in call\n    kwargs=getattr(command_request, '_kwargs'))\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 94, in dispatch\n    raise e\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 91, in dispatch\n    call_result = _method(_instance, *deserialized_args)\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 194, in wrapper\n    raise RuntimeError(msg)\nRuntimeError: \n\n==== detail start, at 20220706.113309.728 ====\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task\n    value=self.functor_serdes.serialize(f(task)))\n  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize\n    return cloudpickle.dumps(_obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps\n    cp.dump(obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump\n    return Pickler.dump(self, obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump\n    self.save(obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce\n    save(args)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple\n    save(element)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce\n    self._batch_setitems(dictitems)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems\n    tmp = list(islice(it, self._BATCHSIZE))\nRuntimeError: dictionary changed size during iteration\n\n==== detail end ====\n\n\n\n==== detail end ====\n\n","grpc_status":2}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/projects/fate/fateflow/python/fate_flow/worker/task_executor.py", line 195, in _run_
    cpn_output = run_object.run(cpn_input)
  File "/data/projects/fate/fate/python/federatedml/model_base.py", line 209, in run
    method(cpn_input)
  File "/data/projects/fate/fate/python/federatedml/model_base.py", line 244, in _run
    this_data_output = func(*real_param)
  File "/data/projects/fate/fate/python/federatedml/statistic/correlation/hetero_pearson.py", line 166, in fit
    FixedPointTensor.from_source("y", source[1]),
  File "/data/projects/fate/fate/python/federatedml/secureprotol/spdz/tensor/fixedpoint_table.py", line 174, in from_source
    share = spdz.communicator.get_share(tensor_name=tensor_name, party=source)[0]
  File "/data/projects/fate/fate/python/federatedml/secureprotol/spdz/communicator/federation.py", line 59, in get_share
    return self._share_variable.get_parties(party, suffix=(tensor_name,))
  File "/data/projects/fate/fate/python/fate_arch/federation/transfer_variable.py", line 241, in get_parties
    name=name, tag=tag, parties=parties, gc=self._get_gc
  File "/data/projects/fate/fate/python/fate_arch/federation/eggroll/_federation.py", line 56, in get
    raw_result = _get(name, tag, parties, self._rsc, gc)
  File "/data/projects/fate/fate/python/fate_arch/federation/eggroll/_federation.py", line 108, in _get
    v = future.result()
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/data/projects/fate/eggroll/python/eggroll/core/datastructure/threadpool.py", line 51, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/data/projects/fate/eggroll/python/eggroll/roll_site/roll_site.py", line 647, in _pull_one
    raise e
  File "/data/projects/fate/eggroll/python/eggroll/roll_site/roll_site.py", line 607, in _pull_one
    pull_status, all_finished, total_batches, cur_pairs = get_status(self)
  File "/data/projects/fate/eggroll/python/eggroll/roll_site/roll_site.py", line 566, in get_status
    all_status = store.with_stores(get_partition_status, options={"__op": "get_partition_status"})
  File "/data/projects/fate/eggroll/python/eggroll/core/aspects.py", line 30, in wrapper
    result = func(*args, **kwargs)
  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/roll_pair.py", line 1141, in with_stores
    ret_pair = future.result()[0]
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/data/projects/fate/eggroll/python/eggroll/core/datastructure/threadpool.py", line 51, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/data/projects/fate/eggroll/python/eggroll/core/client.py", line 97, in sync_send
    raise CommandCallError(command_uri, endpoint, e)
eggroll.core.client.CommandCallError: ('Failed to call command: CommandURI(_uri=v1/egg-pair/runTask) to endpoint: nodemanager-0:41349, caused by: ', <_Rendezvous of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Exception calling application: 

==== detail start, at 20220706.113309.735 ====
Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task
    value=self.functor_serdes.serialize(f(task)))
  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize
    return cloudpickle.dumps(_obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps
    cp.dump(obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump
    return Pickler.dump(self, obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce
    save(args)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce
    self._batch_setitems(dictitems)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems
    tmp = list(islice(it, self._BATCHSIZE))
RuntimeError: dictionary changed size during iteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_service.py", line 33, in call
    kwargs=getattr(command_request, '_kwargs'))
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 94, in dispatch
    raise e
  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 91, in dispatch
    call_result = _method(_instance, *deserialized_args)
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 194, in wrapper
    raise RuntimeError(msg)
RuntimeError: 

==== detail start, at 20220706.113309.728 ====
Traceback (most recent call last):
  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper
    return func(*args, **kw)
  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task
    value=self.functor_serdes.serialize(f(task)))
  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize
    return cloudpickle.dumps(_obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps
    cp.dump(obj)
  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump
    return Pickler.dump(self, obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce
    save(args)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce
    self._batch_setitems(dictitems)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems
    tmp = list(islice(it, self._BATCHSIZE))
RuntimeError: dictionary changed size during iteration

==== detail end ====

==== detail end ====

"
    debug_error_string = "{"created":"@1657078389.737481467","description":"Error received from peer ipv4:10.244.2.5:41349","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Exception calling application: \n\n==== detail start, at 20220706.113309.735 ====\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task\n    value=self.functor_serdes.serialize(f(task)))\n  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize\n    return cloudpickle.dumps(_obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps\n    cp.dump(obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump\n    return Pickler.dump(self, obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump\n    self.save(obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce\n    save(args)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple\n    save(element)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce\n    self._batch_setitems(dictitems)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems\n    tmp = list(islice(it, self._BATCHSIZE))\nRuntimeError: dictionary changed size during iteration\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_service.py", line 33, in call\n    kwargs=getattr(command_request, '_kwargs'))\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 94, in dispatch\n    raise e\n  File "/data/projects/fate/eggroll/python/eggroll/core/command/command_router.py", line 91, in dispatch\n    call_result = _method(_instance, *deserialized_args)\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 194, in wrapper\n    raise RuntimeError(msg)\nRuntimeError: \n\n==== detail start, at 20220706.113309.728 ====\nTraceback (most recent call last):\n  File "/data/projects/fate/eggroll/python/eggroll/core/utils.py", line 187, in wrapper\n    return func(*args, **kw)\n  File "/data/projects/fate/eggroll/python/eggroll/roll_pair/egg_pair.py", line 658, in run_task\n    value=self.functor_serdes.serialize(f(task)))\n  File "/data/projects/fate/eggroll/python/eggroll/core/serdes/eggroll_serdes.py", line 58, in serialize\n    return cloudpickle.dumps(_obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 931, in dumps\n    cp.dump(obj)\n  File "/opt/app-root/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 284, in dump\n    return Pickler.dump(self, obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 409, in dump\n    self.save(obj)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 606, in save_reduce\n    save(args)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 476, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 751, in save_tuple\n    save(element)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 521, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 631, in save_reduce\n    self._batch_setitems(dictitems)\n  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/pickle.py", line 841, in _batch_setitems\n    tmp = list(islice(it, self._BATCHSIZE))\nRuntimeError: dictionary changed size during iteration\n\n==== detail end ====\n\n\n\n==== detail end ====\n\n","grpc_status":2}"
>)
[INFO] [2022-07-06 11:33:10,716] [202207061020403060030] [142648:140652751726400] - [task_base_worker.report_task_info_to_driver] [line:124]: report TaskExecutor 202207061020403060030_hetero_pearson_0 0 guest 9998 to driver:
{'job_id': '202207061020403060030', 'component_name': 'hetero_pearson_0', 'task_id': '202207061020403060030_hetero_pearson_0', 'task_version': '0', 'role': 'guest', 'party_id': '9998', 'run_ip': '10.244.1.56', 'run_pid': 142648, 'party_status': 'failed', 'src_role': '', 'src_party_id': '', 'end_time': 1657078390716, 'elapsed': 4255460}
[INFO] [2022-07-06 11:33:10,716] [202207061020403060030] [142648:140652751726400] - [control_client.report_task] [line:41]: request update job 202207061020403060030 task 202207061020403060030_hetero_pearson_0 0 on guest 9998
[INFO] [2022-07-06 11:33:16,150] [202207061020403060030] [142648:140652751726400] - [task_executor._run_] [line:254]: finish hetero_pearson_0 202207061020403060030_hetero_pearson_0 0 on guest 9998 with failed
[INFO] [2022-07-06 11:33:16,155] [202207061020403060030] [142648:140652751726400] - [base_worker.run] [line:122]: worker TaskExecutor, process role: ProcessRole.WORKER, pid: 142648, elapsed: 4261521 ms

Environment:

github-actions[bot] commented 4 months ago

This issue has been marked as stale because it has been open for 365 days with no activity. If this issue is still relevant or if there is new information, please feel free to update or reopen it.

github-actions[bot] commented 4 months ago

This issue was closed because it has been inactive for 1 days since being marked as stale. If this issue is still relevant or if there is new information, please feel free to update or reopen it.