search

Xtra-Computing / FedTree

A tree-based federated learning system (MLSys 2023)
https://fedtree.readthedocs.io/en/latest/index.html
Apache License 2.0
140 stars 38 forks source link

Error_when building with -DDISTRIBUTED=ON #64

Closed stamatisvas closed 1 year ago

stamatisvas commented 1 year ago

Hi. Even though everything works fine when I do the build with -DDISTRIBUTED=OFF when I am running: cmake .. -DDISTRIBUTED=ON make -j I am getting the following error

(base) fps@fps:~/Documents/FL/FedTree/build$ make -j Scanning dependencies of target FedTree_DIST Scanning dependencies of target ft_grpc_proto Scanning dependencies of target FedTree [ 1%] Building CXX object src/FedTree/CMakeFiles/ft_grpc_proto.dir/grpc/fedtree.grpc.pb.cc.o [ 2%] Building CXX object src/FedTree/CMakeFiles/ft_grpc_proto.dir/grpc/fedtree.pb.cc.o In file included from /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.cc:4: /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.h:17:2: error: #error This file was generated by an older version of protoc which is 17 | #error This file was generated by an older version of protoc which is | ^~~~~ In file included from /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.grpc.pb.cc:5: /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.h:17:2: error: #error This file was generated by an older version of protoc which is 17 | #error This file was generated by an older version of protoc which is | ^~~~~ /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please 18 | #error incompatible with your Protocol Buffer headers. Please | ^~~~~ /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please 18 | #error incompatible with your Protocol Buffer headers. Please | ^~~~~ /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.h:19:2: error: #error regenerate this file with a newer version of protoc. 19 | #error regenerate this file with a newer version of protoc. | ^~~~~ /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.h:19:2: error: #error regenerate this file with a newer version of protoc. 19 | #error regenerate this file with a newer version of protoc. | ^~~~~ In file included from /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.grpc.pb.cc:5: /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.h:26:10: fatal error: google/protobuf/generated_message_table_driven.h: No such file or directory 26 | #include <google/protobuf/generated_message_table_driven.h> | ^~~~~~~~~~~~~~ In file included from /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.cc:4: /home/fps/Documents/FL/FedTree/src/FedTree/grpc/fedtree.pb.h:26:10: fatal error: google/protobuf/generated_message_table_driven.h: No such file or directory 26 | #include <google/protobuf/generated_message_table_driven.h> | ^~~~~~~~~~~~~~ compilation terminated. compilation terminated. make[2]: [src/FedTree/CMakeFiles/ft_grpc_proto.dir/build.make:82: src/FedTree/CMakeFiles/ft_grpc_proto.dir/grpc/fedtree.grpc.pb.cc.o] Error 1 make[2]: Waiting for unfinished jobs.... make[2]: [src/FedTree/CMakeFiles/ft_grpc_proto.dir/build.make:95: src/FedTree/CMakeFiles/ft_grpc_proto.dir/grpc/fedtree.pb.cc.o] Error 1 make[1]: [CMakeFiles/Makefile2:173: src/FedTree/CMakeFiles/ft_grpc_proto.dir/all] Error 2 make[1]: Waiting for unfinished jobs.... [ 4%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/DP/differential_privacy.cpp.o [ 5%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Encryption/diffie_hellman.cpp.o [ 6%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Encryption/paillier.cpp.o [ 8%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Encryption/paillier_gmp.cpp.o [ 10%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/FL/partition.cpp.o [ 10%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/FL/FLtrainer.cpp.o [ 12%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/FL/party.cpp.o [ 13%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/FL/server.cpp.o [ 15%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Tree/gbdt.cpp.o [ 16%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Tree/function_builder.cpp.o [ 17%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/DP/differential_privacy.cpp.o [ 19%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Encryption/diffie_hellman.cpp.o [ 20%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Encryption/paillier.cpp.o [ 21%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Tree/hist_cut.cpp.o [ 23%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Tree/hist_tree_builder.cpp.o [ 24%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Encryption/paillier_gmp.cpp.o [ 26%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Tree/tree.cpp.o [ 27%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/FL/FLtrainer.cpp.o [ 28%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/FL/distributed_party.cpp.o [ 30%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/Tree/tree_builder.cpp.o [ 31%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/booster.cpp.o [ 32%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/dataset.cpp.o [ 34%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/FL/distributed_server.cpp.o [ 35%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/FL/party.cpp.o [ 36%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/FL/partition.cpp.o [ 38%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/metric/metric.cpp.o [ 39%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/metric/multiclass_metric.cpp.o [ 41%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/metric/pointwise_metric.cpp.o [ 43%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/FL/server.cpp.o [ 43%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Tree/function_builder.cpp.o [ 45%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Tree/gbdt.cpp.o [ 46%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/metric/rank_metric.cpp.o [ 47%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Tree/hist_cut.cpp.o [ 49%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/objective/multiclass_obj.cpp.o [ 50%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/objective/objective_function.cpp.o [ 52%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/objective/ranking_obj.cpp.o [ 53%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/parser.cpp.o [ 54%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Tree/hist_tree_builder.cpp.o [ 56%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/predictor.cpp.o [ 57%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Tree/tree.cpp.o [ 58%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/Tree/tree_builder.cpp.o [ 60%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/booster.cpp.o [ 61%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/scikit_fedtree.cpp.o In file included from /home/fps/.local/include/absl/base/config.h:86, from /home/fps/.local/include/absl/base/const_init.h:25, from /home/fps/.local/include/absl/synchronization/mutex.h:67, from /home/fps/.local/include/grpcpp/impl/sync.h:30, from /home/fps/.local/include/grpcpp/impl/codegen/sync.h:25, from /home/fps/.local/include/grpcpp/completion_queue.h:43, from /home/fps/.local/include/grpcpp/channel.h:25, from /home/fps/.local/include/grpcpp/grpcpp.h:52, from /home/fps/Documents/FL/FedTree/include/FedTree/FL/distributed_party.h:8, from /home/fps/Documents/FL/FedTree/src/FedTree/FL/distributed_party.cpp:5: /home/fps/.local/include/absl/base/policy_checks.h:79:2: error: #error "C++ versions less than C++14 are not supported." 79 | #error "C++ versions less than C++14 are not supported." | ^~~~~ [ 63%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/syncmem.cpp.o [ 64%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/dataset.cpp.o [ 65%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/util/common.cpp.o [ 67%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/metric/metric.cpp.o [ 68%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/metric/multiclass_metric.cpp.o [ 69%] Building CXX object src/FedTree/CMakeFiles/FedTree.dir/util/log.cpp.o [ 71%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/metric/pointwise_metric.cpp.o In file included from /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.grpc.pb.h:7, from /home/fps/Documents/FL/FedTree/include/FedTree/FL/distributed_party.h:14, from /home/fps/Documents/FL/FedTree/src/FedTree/FL/distributed_party.cpp:5: /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.pb.h:17:2: error: #error This file was generated by an older version of protoc which is 17 | #error This file was generated by an older version of protoc which is | ^~~~~ /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please 18 | #error incompatible with your Protocol Buffer headers. Please | ^~~~~ /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.pb.h:19:2: error: #error regenerate this file with a newer version of protoc. 19 | #error regenerate this file with a newer version of protoc. | ^~~~~ [ 72%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/metric/rank_metric.cpp.o [ 73%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/objective/multiclass_obj.cpp.o In file included from /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.grpc.pb.h:7, from /home/fps/Documents/FL/FedTree/include/FedTree/FL/distributed_party.h:14, from /home/fps/Documents/FL/FedTree/src/FedTree/FL/distributed_party.cpp:5: /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.pb.h:26:10: fatal error: google/protobuf/generated_message_table_driven.h: No such file or directory 26 | #include <google/protobuf/generated_message_table_driven.h> | ^~~~~~~~~~~~~~ compilation terminated. [ 75%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/objective/objective_function.cpp.o make[2]: [src/FedTree/CMakeFiles/FedTree_DIST.dir/build.make:147: src/FedTree/CMakeFiles/FedTree_DIST.dir/FL/distributed_party.cpp.o] Error 1 make[2]: Waiting for unfinished jobs.... [ 76%] Building CXX object src/FedTree/CMakeFiles/FedTree_DIST.dir/objective/ranking_obj.cpp.o In file included from /home/fps/.local/include/absl/base/config.h:86, from /home/fps/.local/include/absl/base/const_init.h:25, from /home/fps/.local/include/absl/synchronization/mutex.h:67, from /home/fps/.local/include/grpcpp/impl/sync.h:30, from /home/fps/.local/include/grpcpp/impl/codegen/sync.h:25, from /home/fps/.local/include/grpcpp/completion_queue.h:43, from /home/fps/.local/include/grpcpp/channel.h:25, from /home/fps/.local/include/grpcpp/grpcpp.h:52, from /home/fps/Documents/FL/FedTree/include/FedTree/FL/distributed_server.h:8, from /home/fps/Documents/FL/FedTree/src/FedTree/FL/distributed_server.cpp:5: /home/fps/.local/include/absl/base/policy_checks.h:79:2: error: #error "C++ versions less than C++14 are not supported." 79 | #error "C++ versions less than C++14 are not supported." | ^~~~~ In file included from /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.grpc.pb.h:7, from /home/fps/Documents/FL/FedTree/include/FedTree/FL/distributed_server.h:25, from /home/fps/Documents/FL/FedTree/src/FedTree/FL/distributed_server.cpp:5: /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.pb.h:17:2: error: #error This file was generated by an older version of protoc which is 17 | #error This file was generated by an older version of protoc which is | ^~~~~ /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please 18 | #error incompatible with your Protocol Buffer headers. Please | ^~~~~ /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.pb.h:19:2: error: #error regenerate this file with a newer version of protoc. 19 | #error regenerate this file with a newer version of protoc. | ^~~~~ In file included from /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.grpc.pb.h:7, from /home/fps/Documents/FL/FedTree/include/FedTree/FL/distributed_server.h:25, from /home/fps/Documents/FL/FedTree/src/FedTree/FL/distributed_server.cpp:5: /home/fps/Documents/FL/FedTree/include/FedTree/FL/../../../src/FedTree/grpc/fedtree.pb.h:26:10: fatal error: google/protobuf/generated_message_table_driven.h: No such file or directory 26 | #include <google/protobuf/generated_message_table_driven.h> | ^~~~~~~~~~~~~~ compilation terminated. make[2]: [src/FedTree/CMakeFiles/FedTree_DIST.dir/build.make:160: src/FedTree/CMakeFiles/FedTree_DIST.dir/FL/distributed_server.cpp.o] Error 1 [ 78%] Linking CXX shared library ../../lib/libFedTree.so make[1]: [CMakeFiles/Makefile2:146: src/FedTree/CMakeFiles/FedTree_DIST.dir/all] Error 2 [ 78%] Built target FedTree make: [Makefile:103: all] Error 2

QinbinLi commented 1 year ago

Hi @stamatisvas ,

Did you install grpc 1.50.0? If you are using a different version than grpc 1.50.0, you may need to go to src/FedTree/grpc directory and run the following commands before compiling FedTree. 

protoc -I ./ --grpc_out=. --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` ./fedtree.proto
protoc -I ./ --cpp_out=. ./fedtree.proto

If the above command does not work, one solution is to install grpc 1.50.0. In this link (https://grpc.io/docs/languages/cpp/quickstart/), you need to change v1.53.0 to v1.50.0 when cloning grpc.

stamatisvas commented 1 year ago

Thank you for the help. The problem was that I installed grpc 1.53.0. I downgraded it to 1.50.0 and the build finished successfully!

I tried to test FedTree using my computer as the server and 2 Ubuntu virtual machines (virtual box) as the clients. I am getting the following error:

user@user-VirtualBox:~/Documents/FedTree$ ./build/bin/FedTree-distributed-party ./party2.conf 1 2023-04-07 16:44:55,878 INFO dataset.cpp:219 : loading LIBSVM dataset from file ## ./dataset/adult/a9a_horizontal_p1 ## 2023-04-07 16:44:55,899 INFO dataset.cpp:366 : #instances = 16281, #features = 122 2023-04-07 16:44:55,899 INFO dataset.cpp:388 : Load dataset using time: 0.0210823 s 2023-04-07 16:44:55,899 INFO dataset.cpp:219 : loading LIBSVM dataset from file ## ./dataset/adult/a9a_horizontal_test ## 2023-04-07 16:44:55,924 INFO dataset.cpp:366 : #instances = 16281, #features = 122 2023-04-07 16:44:55,924 INFO dataset.cpp:388 : Load dataset using time: 0.0253543 s 2023-04-07 16:45:15,931 ERROR distributed_party.cpp:1367 : BeginBarrier rpc failed. 2023-04-07 16:45:15,931 INFO fedtree_distributed_party.cpp:108 : training start 2023-04-07 16:45:15,933 ERROR distributed_party.cpp:539 : SendRange rpc failed. 2023-04-07 16:45:15,934 ERROR distributedparty.cpp:577 : GetRange rpc failed. 2023-04-07 16:45:15,934 FATAL syncarray.h:77 : Check failed: [size > 0] 2023-04-07 16:45:15,934 WARNING syncarray.h:77 : Aborting application. Reason: Fatal log at [/home/user/Documents/FedTree/include/FedTree/syncarray.h:77] Aborted (core dumped)

QinbinLi commented 1 year ago

Hi @stamatisvas ,

Can you show me the configuration files you run? Thanks!

stamatisvas commented 1 year ago

The conf files I am using are the ones (server.conf, party1.conf ) from Distributed Horizontal FedTree with Secure Aggregation in https://fedtree.readthedocs.io/en/latest/Examples.html I only removed the privacy_tech=sa from both server and parties. The error appears on both terminals running party.conf files.

Also please note that when I am using a virtual box Ubuntu as a server and for both parties I am using my computer, everything works fine.

QinbinLi commented 1 year ago

In that case, I think there is a network issue in your configuration. You need to ensure that your virtual box can connect to your computer with the specified ip_address.

stamatisvas commented 1 year ago

Yes, indeed this was the problem. Now everything works. Thanks for the support!