search

blockchain-systems / ScaleSFL

A scalable sharding solution for Blockchain based Federated Learning. SCaFL or ScaleSFL?
66 stars 7 forks source link

Problems in use #3

Open deamonyi opened 1 year ago

deamonyi commented 1 year ago

Ask about the version of fabrics used in the current project and how to use this federal learning feature

evanwrm commented 1 year ago

We used Fabric v2.3.3 when running experiments. Hopefully the instructions in the Readme should get the system up and running (let me know if anything is missing).

I believe there have also been updates to other 3rd party libraries in the example participant so make sure to use the lockfile versions

deamonyi commented 1 year ago

Thank you for your last answer! But I have encountered some problems again.The first issue is when running manager. py E2NXJM(ZV4$ZK T6(A3ME9P I think this may be a clerical error, so I changed ':=' to '=='，then I encountered the second problem KU0_L3{P3ERAW2PPBKOR}G4 So I want to know if the modifications I made are correct, and what is the version of the flower used in this project.

evanwrm commented 1 year ago

:= is know as the walrus operator in Python. Feel free to change this if you run on older version of Python.

We used version "0.17.0" of flower (as seen in the lockfile). It seems the api has changed slightly since the v1.0 release.

deamonyi commented 1 year ago

Thank you very much for your reply. Now I am encountering new issues when using startFabric.sh. At the run ./startFabric.sh , my peer nodes are not functioning properly. Here are the docker logs for these nodes： IDR6X~NC2AV}R)}YI5I68S2 4$T7@ES$D@`@ MQ$825Z4@2 I attempted to copy the escc.so file to the specified absolute path and granted it all permissions, but the error still occurred. Thank you again for your answer to my previous question.

qadria commented 1 year ago

I m facing the same problem and it seems some configuration/permission issue. Trying giving docker permission for network files but still the same.

deamonyi commented 1 year ago

I m facing the same problem and it seems some configuration/permission issue. Trying giving docker permission for network files but still the same.

It shouldn't be a permission issue. I tried to change the hosts file and dock permissions but still couldn't function properly.

qadria commented 1 year ago

It seems error in building the escc.co file. Are you following the steps for building the plugin? There are two steps and second method is easier.

Also, you need to modify absolute ip to localhost in docker files in test_network/docker folder

qadria commented 1 year ago

This error is coming because the escc.so the file is not appropriately configured. To fix the error, you can use the second method as suggested in the documentation. Copy the Dockerfile.peer from ../ScaleSFL/plugins/Dockerfile.peer to the ../fabric/images/peer/. Make sure to rename the Dockerfile to Dockerfile.copy and rename Dockerfile.peer to Dockerfile. Afterwards, try to build the plugin as advised in the documentation.

I hope the error will be resolved.

deamonyi commented 1 year ago

This error is coming because the escc.so the file is not appropriately configured. To fix the error, you can use the second method as suggested in the documentation. Copy the Dockerfile.peer from ../ScaleSFL/plugins/Dockerfile.peer to the ../fabric/images/peer/. Make sure to rename the Dockerfile to Dockerfile.copy and rename Dockerfile.peer to Dockerfile. Afterwards, try to build the plugin as advised in the documentation.

I hope the error will be resolved.

Thank you very much. I tried the second method in the document to refactor the plugin, and the error disappeared.

qadria commented 1 year ago

Now I am getting a new error in deploying the chain code.

Error: proposal failed with status: 500 - endorsing with plugin failed: Could not evaluate model: Post "http://localhost:3000/endorse/evaluate": dial tcp 127.0.0.1:3000: connect: connection refused Chaincode definition approved on peer0.org1 on channel 'mainline' failed Deploying chaincode failed

Snag_b672f8

deamonyi commented 1 year ago

I also encountered the same mistake. $7L8D9K2WWY4M_K~ 5G @7B

D{O$5`CF )E8U7)T$VXJG}I

6M%R`%B)`3_))BW4UVX1FCR

}$)$(SHU4@8G6~0N8_KSYPD

image I guess this error may be related to the update of anchor nodes.

qadria commented 1 year ago

Yes you are right, it seems, chain code deployed correctly till org3/shard2 but failed in org4/shard3

shard 2 

`+ configtxlator proto_decode --input config_block.pb --type common.Block
+ jq '.data.data[0].payload.data.config'
+ jq '.channel_group.groups.Application.groups.Org3MSP.values += {"AnchorPeers":{"mod_policy": "Admins","value":{"anchor_peers": [{"host": "peer0.org3.example.com","port": 11051}]},"version": "0"}}' Org3MSPconfig.json
^[[0;34mGenerating anchor peer update transaction for Org3 on channel shard2^[[0m
+ configtxlator proto_encode --input Org3MSPconfig.json --type common.Config
+ configtxlator proto_encode --input Org3MSPmodified_config.json --type common.Config
+ configtxlator compute_update --channel_id shard2 --original original_config.pb --updated modified_config.pb
+ configtxlator proto_decode --input config_update.pb --type common.ConfigUpdate
+ jq .
++ cat config_update.json
+ echo '{"payload":{"header":{"channel_header":{"channel_id":"shard2", "type":2}},"data":{"config_update":{' '"channel_id":' '"shard2",' '"isolated_data":' '{},' '"read_set":' '{' '"groups":' '{' '"Application":' '{' '"groups":' '{' '"Org3MSP":' '{' '"groups":' '{},' '"mod_policy":' '"",' '"policies":' '{' '"Admins":' '{' '"mod_policy":' '"",' '"policy":' null, '"version":' '"0"' '},' '"Endorsement":' '{' '"mod_policy":' '"",' '"policy":' null, '"version":' '"0"' '},' '"Readers":' '{' '"mod_policy":' '"",' '"policy":' null, '"version":' '"0"' '},' '"Writers":' '{' '"mod_policy":' '"",' '"policy":' null, '"version":' '"0"' '}' '},' '"values":' '{' '"MSP":' '{' '"mod_policy":' '"",' '"value":' null, '"version":' '"0"' '}' '},' '"version":' '"0"' '}' '},' '"mod_policy":' '"",' '"policies":' '{},' '"values":' '{},' '"version":' '"0"' '}' '},' '"mod_policy":' '"",' '"policies":' '{},' '"values":' '{},' '"version":' '"0"' '},' '"write_set":' '{' '"groups":' '{' '"Application":' '{' '"groups":' '{' '"Org3MSP":' '{' '"groups":' '{},' '"mod_policy":' '"Admins",' '"policies":' '{' '"Admins":' '{' '"mod_policy":' '"",' '"policy":' null, '"version":' '"0"' '},' '"Endorsement":' '{' '"mod_policy":' '"",' '"policy":' null, '"version":' '"0"' '},' '"Readers":' '{' '"mod_policy":' '"",' '"policy":' null, '"version":' '"0"' '},' '"Writers":' '{' '"mod_policy":' '"",' '"policy":' null, '"version":' '"0"' '}' '},' '"values":' '{' '"AnchorPeers":' '{' '"mod_policy":' '"Admins",' '"value":' '{' '"anchor_peers":' '[' '{' '"host":' '"peer0.org3.example.com",' '"port":' 11051 '}' ']' '},' '"version":' '"0"' '},' '"MSP":' '{' '"mod_policy":' '"",' '"value":' null, '"version":' '"0"' '}' '},' '"version":' '"1"' '}' '},' '"mod_policy":' '"",' '"policies":' '{},' '"values":' '{},' '"version":' '"0"' '}' '},' '"mod_policy":' '"",' '"policies":' '{},' '"values":' '{},' '"version":' '"0"' '}' '}}}}'
+ configtxlator proto_encode --input config_update_in_envelope.json --type common.Envelope
^[[34m2023-05-02 11:03:37.337 UTC [channelCmd] InitCmdFactory -> INFO 001^[[0m Endorser and orderer connections initialized
^[[34m2023-05-02 11:03:37.348 UTC [channelCmd] update -> INFO 002^[[0m Successfully submitted channel update
^[[0;32mAnchor peer set for org 'Org3MSP' on channel 'shard2'^[[0m
^[[0;32mChannel 'shard2' joined^[[0m

`

shard 3


`+ configtxlator proto_decode --input config_block.pb --type common.Block
+ jq '.data.data[0].payload.data.config'
^[[0;34mGenerating anchor peer update transaction for Org4 on channel shard3^[[0m
^[[0;31mOrg4 unknown^[[0m
+ jq '.channel_group.groups.Application.groups.Org4MSP.values += {"AnchorPeers":{"mod_policy": "Admins","value":{"anchor_peers": [{"host": "","port": }]},"version": "0"}}' Org4MSPconfig.json
jq: error: syntax error, unexpected '}' (Unix shell quoting issues?) at <top-level>, line 1:
.channel_group.groups.Application.groups.Org4MSP.values += {"AnchorPeers":{"mod_policy": "Admins","value":{"anchor_peers": [{"host": "","port": }]},"version": "0"}}                                                                      
jq: error: syntax error, unexpected '}' (Unix shell quoting issues?) at <top-level>, line 1:
.channel_group.groups.Application.groups.Org4MSP.values += {"AnchorPeers":{"mod_policy": "Admins","value":{"anchor_peers": [{"host": "","port": }]},"version": "0"}}                                                                      
jq: 2 compile errors
+ configtxlator proto_encode --input Org4MSPconfig.json --type common.Config
+ configtxlator proto_encode --input Org4MSPmodified_config.json --type common.Config
configtxlator: error: Error decoding: error decoding input: error unmarshaling intermediate JSON: EOF
+ configtxlator compute_update --channel_id shard3 --original original_config.pb -`
qadria commented 1 year ago
  • jq '.channel_group.groups.Application.groups.Org3MSP.values += {"AnchorPeers":{"mod_policy": "Admins","value":{"anchor_peers": [{"host": "peer0.org3.example.com","port": 11051}]},"version": "0"}}' Org3MSPconfig.json ^[[0;

Seem Org4 ~ 7 not existing ... therefore error "Org4 unknown" Snag_1448f38

Snag_14c1fcd

GeorgeNiotis commented 7 months ago

Did you manage to run it? Ive fixed most of the issues regarding the creation of Anchor peers for the the rest of the orgs and also compiled the plugin succesfully. Got an issue regarding the caliper benchmark. After running the test im getting this error caliper|2023.11.15-11:18:56.455 error [caliper] [connectors/peer-gateway/PeerGateway] Failed to perform submit transaction [CreateModel] using arguments [model_1e994ced126e24a273bef66a099da7b395b4286db2aa338b622f83b3d01b55ae,1e994ced126e24a273bef66a099da7b395b4286db2aa338b622f83b3d01b55ae,worker0,http://192.168.1.170:3003,1,84.06386749888998], with error: Error: Failure: Inconsistent type in JSPB repeated field array. Got number expected object

gold-4N commented 5 months ago

Did you manage to run it? Ive fixed most of the issues regarding the creation of Anchor peers for the the rest of the orgs and also compiled the plugin succesfully. Got an issue regarding the caliper benchmark. After running the test im getting this error caliper|2023.11.15-11:18:56.455 error [caliper] [connectors/peer-gateway/PeerGateway] Failed to perform submit transaction [CreateModel] using arguments [model_1e994ced126e24a273bef66a099da7b395b4286db2aa338b622f83b3d01b55ae,1e994ced126e24a273bef66a099da7b395b4286db2aa338b622f83b3d01b55ae,worker0,http://192.168.1.170:3003,1,84.06386749888998], with error: Error: Failure: Inconsistent type in JSPB repeated field array. Got number expected object

need some help to run the fabric @GeorgeNiotis

gold-4N commented 5 months ago

@qadria are you successfully run the project?

qadria commented 5 months ago

@gold-4N make sure you follow the exact steps and keep all libs as outline in the readme. If any package is not same then fabric dont work as expected. I tested, if all specs are same then it will work fine.

qadria commented 5 months ago

I am working on new version of ScaleSFL with an application use case. Let me know if anyone interested to contribute in the research paper.

gold-4N commented 5 months ago

I am interested

I am working on new version of ScaleSFL with an application use case. Let me know if anyone interested to contribute in the research paper.

gold-4N commented 5 months ago

@gold-4N make sure you follow the exact steps and keep all libs as outline in the readme. If any package is not same then fabric dont work as expected. I tested, if all specs are same then it will work fine.

I follow the steps but some how its not find org4-8 and also error in chaincode deploy image

qadria commented 5 months ago

go version is same?

gold-4N commented 5 months ago

go version is same?

yes

majiajun7 commented 1 month ago

@gold-4N 确保遵循确切的步骤，并保留所有库，如自述文件中所述。如果任何包不相同，则 Fabric 不会按预期工作。我测试过，如果所有规格都相同，那么它将正常工作。

Did you run into this problem in your last test

2024-05-30T07:30:55.855Z - error: [Transaction]: Error: No valid responses from any peers. Errors: at newEndorsementError (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:53:12) at getResponsePayload (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:21:23) at Transaction.submit (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:234:28) at processTicksAndRejections (internal/process/task_queues.js:95:5) at async V2FabricGateway._submitOrEvaluateTransaction (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:376:26) at async V2FabricGateway._sendSingleRequest (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:170:16) at async V2FabricGateway.sendRequests (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-core/lib/common/core/connector-base.js:78:28) at async CreateModelsWorkload.submitTransaction (/root/ScaleSFL/caliper-tests/workload/createModel.js:51:9) 2024.05.30-00:30:55.856 error [caliper] [connectors/v2/FabricGateway] Failed to perform submit transaction [CreateModel] using arguments [model_62c93a9c4102b63ec33d6a05a3501f1bbd536f49826447b0f69b0bad26bf5f23,62c93a9c4102b63ec33d6a05a3501f1bbd536f49826447b0f69b0bad26bf5f23,worker0,http://192.168.142.129:3001,1,76.20625408515761], with error: Error: No valid responses from any peers. Errors: at newEndorsementError (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:53:12) at getResponsePayload (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:21:23) at Transaction.submit (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:234:28) at processTicksAndRejections (internal/process/task_queues.js:95:5) at async V2FabricGateway._submitOrEvaluateTransaction (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:376:26) at async V2FabricGateway._sendSingleRequest (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:170:16) at async V2FabricGateway.sendRequests (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-core/lib/common/core/connector-base.js:78:28) at async CreateModelsWorkload.submitTransaction (/root/ScaleSFL/caliper-tests/workload/createModel.js:51:9)

gold-4N commented 1 month ago

@gold-4N 确保遵循确切的步骤，并保留所有库，如自述文件中所述。如果任何包不相同，则 Fabric 不会按预期工作。我测试过，如果所有规格都相同，那么它将正常工作。

Did you run into this problem in your last test

2024-05-30T07:30:55.855Z - error: [Transaction]: Error: No valid responses from any peers. Errors: at newEndorsementError (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:53:12) at getResponsePayload (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:21:23) at Transaction.submit (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:234:28) at processTicksAndRejections (internal/process/task_queues.js:95:5) at async V2FabricGateway._submitOrEvaluateTransaction (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:376:26) at async V2FabricGateway._sendSingleRequest (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:170:16) at async V2FabricGateway.sendRequests (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-core/lib/common/core/connector-base.js:78:28) at async CreateModelsWorkload.submitTransaction (/root/ScaleSFL/caliper-tests/workload/createModel.js:51:9) 2024.05.30-00:30:55.856 error [caliper] [connectors/v2/FabricGateway] Failed to perform submit transaction [CreateModel] using arguments [model_62c93a9c4102b63ec33d6a05a3501f1bbd536f49826447b0f69b0bad26bf5f23,62c93a9c4102b63ec33d6a05a3501f1bbd536f49826447b0f69b0bad26bf5f23,worker0,http://192.168.142.129:3001,1,76.20625408515761], with error: Error: No valid responses from any peers. Errors: at newEndorsementError (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:53:12) at getResponsePayload (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:21:23) at Transaction.submit (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:234:28) at processTicksAndRejections (internal/process/task_queues.js:95:5) at async V2FabricGateway._submitOrEvaluateTransaction (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:376:26) at async V2FabricGateway._sendSingleRequest (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:170:16) at async V2FabricGateway.sendRequests (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-core/lib/common/core/connector-base.js:78:28) at async CreateModelsWorkload.submitTransaction (/root/ScaleSFL/caliper-tests/workload/createModel.js:51:9)

yes was run the projuct

majiajun7 commented 1 month ago

@gold-4N

After running the command:

npx caliper launch manager \
  --caliper-workspace . \
  --caliper-benchconfig benchmarks/model-creation.yaml \
  --caliper-networkconfig networks/fabric-config.yaml \
  --caliper-progress-reporting-interval 2000 \
  --caliper-flow-only-test \
  --caliper-worker-pollinterval 250 \
  --caliper-fabric-gateway-enabled

I encountered the following error:

2024-05-31T08:10:36.621Z - error: [Transaction]: Error: No valid responses from any peers. Errors:
    at newEndorsementError (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:53:12)
    at getResponsePayload (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:21:23)
    at Transaction.submit (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:234:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async V2FabricGateway._submitOrEvaluateTransaction (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:376:26)
    at async V2FabricGateway._sendSingleRequest (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:170:16)
    at async V2FabricGateway.sendRequests (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-core/lib/common/core/connector-base.js:78:28)
    at async CreateModelsWorkload.submitTransaction (/root/ScaleSFL/caliper-tests/workload/createModel.js:50:9)
2024.05.31-16:10:36.622 error [caliper] [connectors/v2/FabricGateway]  Failed to perform submit transaction [CreateModel] using arguments [model_e9d71f6d9fcb36604c83fe23af82ef661521dcd41cbc287adfb7919c5c67c095,e9d71f6d9fcb36604c83fe23af82ef661521dcd41cbc287adfb7919c5c67c095,worker0,http://localhost:3001,1,63.14484394569415],  with error: Error: No valid responses from any peers. Errors:
    at newEndorsementError (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:53:12)
    at getResponsePayload (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:21:23)
    at Transaction.submit (/root/ScaleSFL/caliper-tests/node_modules/fabric-network/lib/transaction.js:234:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async V2FabricGateway._submitOrEvaluateTransaction (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:376:26)
    at async V2FabricGateway._sendSingleRequest (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:170:16)
    at async V2FabricGateway.sendRequests (/root/ScaleSFL/caliper-tests/node_modules/@hyperledger/caliper-core/lib/common/core/connector-base.js:78:28)
    at async CreateModelsWorkload.submitTransaction (/root/ScaleSFL/caliper-tests/workload/createModel.js:50:9)

My channel should have eight peers. Here is the output from checking the channel list:

(ScaleSFL) root@user-VMware-Virtual-Platform:~/ScaleSFL/caliper-tests# cd ~/ScaleSFL/test-network
export PATH=$(realpath ${PWD}/../bin):$PATH
export FABRIC_CFG_PATH=$(realpath ${PWD}/../config/)
. scripts/envVar.sh
setGlobals 1

peer channel list
Using organization 1
2024-05-31 16:08:14.049 +08 0001 INFO [channelCmd] InitCmdFactory -> Endorser and orderer connections initialized
Channels peers have joined: 
shard0
mainline

Can anyone help me identify what might be causing this issue and how to resolve it?

majiajun7 commented 1 month ago

After running the ./startFabric.sh command, I received the following errors which might be causing the issue:

2024-05-31 08:14:32.849 UTC [channelCmd] fetch -> INFO 004 Retrieving last config block: 3
Decoding config block to JSON and isolating config to Org4MSPconfig.json
Generating anchor peer update transaction for Org4 on channel mainline
jq: error: syntax error, unexpected '}' (Unix shell quoting issues?) at <top-level>, line 1:
.channel_group.groups.Application.groups.Org4MSP.values += {"AnchorPeers":{"mod_policy": "Admins","value":{"anchor_peers": [{"host": "","port": }]},"version": "0"}}
configtxlator: error: Error decoding: error decoding input: error unmarshaling intermediate JSON: EOF
configtxlator: error: Error computing update: error computing config update: no channel group included for updated config
Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'mainline': error authorizing update: ConfigUpdate for channel '' but envelope for channel 'mainline'
Anchor peer update failed for Org4

Similar errors occurred for other organizations (Org5, Org6, Org7, Org8).

These errors seem to be related to jq syntax errors and decoding issues with configtxlator when processing the updates. Any help to resolve these issues would be appreciated.