The test accuracy is not changed

[ciciwell]

Hi Snwagh When I use different commands and arguments, there is no change in the Rolling accuracy (100%). Could you give some tips for it or did I run it in wrong way?

./Falcon.out 2 files/IP_localhost files/keyC files/keyAC files/keyBC AlexNet CIFAR10 Semi-honest >/dev/null & ./Falcon.out 1 files/IP_localhost files/keyB files/keyBC files/keyAB AlexNet CIFAR10 Semi-honest >/dev/null & ./Falcon.out 0 files/IP_localhost files/keyA files/keyAB files/keyAC AlexNet CIFAR10 Semi-honest argc == 9 Loading data done..... Rolling accuracy: 128 out of 128 (100 %) MPC Output over uint32_t:

Thank you very much!

[snwagh]

Can you please provide more details on what you're running? 100% accuracy would either mean the data is trivial (all labels the same) or that there is a bug in the accuracy computation part. Is this done using a pre-trained network?

[ciciwell]

After I built the Falcon, I made the following changes to the source code: 1.In the main.cpp, I uncomment the following line to run the inference.

network += " test";
test(PRELOADING, network, net);

2.In the secondary.cpp, I uncomment the "net->getAccuracy(maxIndex, counter);" to get the test accuracy.

3. And, I ran the commands "make terminal" and "make command" to get the same accuracy without making any changes to the makefile.

///////////////////////////////////////////////////////////////////////////////////////
 $ make terminal
./Falcon.out 2 files/IP_localhost files/keyC files/keyAC files/keyBC >/dev/null & 
./Falcon.out 1 files/IP_localhost files/keyB files/keyBC files/keyAB >/dev/null &
./Falcon.out 0 files/IP_localhost files/keyA files/keyAB files/keyAC 
argc == 6
Loading data done.....
Rolling accuracy: 128 out of 128 (100 %)
MPC Output over uint32_t:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
----------------------------------------------
Wall Clock time for SecureML train test: 0.424083 sec
CPU time for SecureML train test: 0.4 sec
----------------------------------------------
----------------------------------------------
Total communication: 13.5822MB (sent) and 13.5822MB (recv)
Total calls: 698 (sends) and 698 (recvs)
----------------------------------------------
----------------------------------------------
Communication, SecureML train test, P0: 4.23044MB (sent) 4.23044MB (recv)
Rounds, SecureML train test, P0: 228(sends) 228(recvs)
----------------------------------------------
----------------------------------------------
Run details: 3PC (P0), 1 iterations, batch size 128
Running Semi-honest SecureML train test on MNIST dataset
----------------------------------------------

----------------------------------------------
(1) FC Layer  784 x 128
 128 (Batch Size)
----------------------------------------------
(2) ReLU Layer  128 x 128
----------------------------------------------
(3) FC Layer  128 x 128
 128 (Batch Size)
----------------------------------------------
(4) ReLU Layer  128 x 128
----------------------------------------------
(5) FC Layer  128 x 10
 128 (Batch Size)
----------------------------------------------
(6) ReLU Layer  10 x 128
----------------------------------------------
Execution completed

$ make command
./Falcon.out 2 files/IP_localhost files/keyC files/keyAC files/keyBC MiniONN MNIST Semi-honest >/dev/null & 
./Falcon.out 1 files/IP_localhost files/keyB files/keyBC files/keyAB MiniONN MNIST Semi-honest >/dev/null &
./Falcon.out 0 files/IP_localhost files/keyA files/keyAB files/keyAC MiniONN MNIST Semi-honest 
argc == 9
Loading data done.....
Rolling accuracy: 128 out of 128 (100 %)
MPC Output over uint32_t:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
----------------------------------------------
Wall Clock time for MiniONN train test: 8.94568 sec
CPU time for MiniONN train test: 9.28 sec
----------------------------------------------
----------------------------------------------
Total communication: 431.593MB (sent) and 431.593MB (recv)
Total calls: 1191 (sends) and 1191 (recvs)
----------------------------------------------
----------------------------------------------
Communication, MiniONN train test, P0: 139.828MB (sent) 139.828MB (recv)
Rounds, MiniONN train test, P0: 391(sends) 391(recvs)
----------------------------------------------
----------------------------------------------
Run details: 3PC (P0), 1 iterations, batch size 128
Running Semi-honest MiniONN train test on MNIST dataset
----------------------------------------------

----------------------------------------------
(1) CNN Layer  28 x 28 x 1
 5 x 5   (Filter Size)
 1 , 0 (Stride, padding)
 128 (Batch Size)
 24 x 24 x 16 (Output)
----------------------------------------------
(2) Maxpool Layer  24 x 24 x 16
 2   (Pooling Size)
 2 (Stride)
 128 (Batch Size)
----------------------------------------------
(3) ReLU Layer  2304 x 128
----------------------------------------------
(4) CNN Layer  12 x 12 x 16
 5 x 5   (Filter Size)
 1 , 0 (Stride, padding)
 128 (Batch Size)
 8 x 8 x 16 (Output)
----------------------------------------------
(5) Maxpool Layer  8 x 8 x 16
 2   (Pooling Size)
 2 (Stride)
 128 (Batch Size)
----------------------------------------------
(6) ReLU Layer  256 x 128
----------------------------------------------
(7) FC Layer  256 x 100
 128 (Batch Size)
----------------------------------------------
(8) ReLU Layer  100 x 128
----------------------------------------------
(9) FC Layer  100 x 10
 128 (Batch Size)
----------------------------------------------
(10) ReLU Layer  10 x 128
----------------------------------------------
Execution completed

4.Finally, I changed the values of parameters NETWORK and DATASET in the Makefile to AlexNet and CIFAR10 respectively, and then ran the command "make command", and the test accuracy (100%) remained unchanged.

BTW, do I need to recreate the datasets for learning and testing? Because I did not find them in the code you uploaded.

[snwagh]

I think it is because the data is not getting loaded. For preloading data, you want to uncomment the following two lines as well 55, 56.

About your final command, yes, you do need to create the datasets manually. 100% accuracy might result from everything being 0 otherwise I don't expect perfect accuracy.

[ciciwell]

How can I get these datasets. are they obtained by executing the python code in the scripts folder? In addition, I found that in the secondary.cpp file, the following code is used to process datasets for training and testing. Can I know how these datasets are generated? thanks very much!

if (partyNum < 3)
    {
        filename_train_data_next = "files/train_data_A";
        filename_train_data_prev = "files/train_data_B";
        filename_test_data_next = "files/test_data_A";
        filename_test_data_prev = "files/test_data_B";
        filename_train_labels_next = "files/train_labels_A";
        filename_train_labels_prev = "files/train_labels_B";
        filename_test_labels_next = "files/test_labels_A";
        filename_test_labels_prev = "files/test_labels_B";
    }
float temp_next = 0, temp_prev = 0;
    ifstream f_next(filename_train_data_next);
    ifstream f_prev(filename_train_data_prev);
    for (int i = 0; i < TRAINING_DATA_SIZE * INPUT_SIZE; ++i)
    {
        f_next >> temp_next; f_prev >> temp_prev;
        trainData.push_back(std::make_pair(floatToMyType(temp_next), floatToMyType(temp_prev)));
    }
    f_next.close(); f_prev.close();

[snwagh]

The raw datasets are available online, for the MNIST one, to parse the data in the right format, you can use this as reference code (you might need to tweak it a bit).

The "files/train_data_A" is legacy stuff and was 8 manually extracted samples of the MNIST dataset. The code will also have to be tweaked for a 2-out-of-3 secret sharing, currently, all files load the same data. The import structure will have to be similar to this to load data in a replicated secret sharing form (in a consistent manner).

[ciciwell]

Thank you very much for your guidance, it helped me a lot. I have one more question that needs your help. How to output FALCON Inference Accuracy and Training Accuracy using FALCON source code. I did not find a function to calculate them in the source code.(The getAccuracy function does not seem to be available)

[snwagh]

Falcon training accuracy presented in the paper (table 6 here) is computed using the plaintext models and the code is given in scripts.

For the inference accuracy, you can (1) complete the function getAccuracy (which on a cursory looks seems nearly ready) (2) reconstruct the final output vector and process the data (which includes computing the maximum value and compare that against a ground truth) in python to find the accuracy (this is already provided in this script).

[ciciwell]

Thank you very much for taking the time to help me. I built a 3-party FAlCON model based on LAN. Could you help me confirm whether my approach is correct? Specific steps are as follows. (1)　In the pre-trained data you uploaded, only the data of party0 has specific values, while the input data of the other two parties are all 0. Therefore, in order to input different data for the three parties, I divided the original MNIST training set of 60，000 examples group into 3 evenly, each containing a training set of 20,000 examples.

(2)　Next, I use the SecureML script to generate three sets of pre-trained data and use them as input for three users，severally.

　According to the above operations, the three parties used severally different data. However, the inference accuracy obtained is very bad. Could you give some tips for it or did I run it in the wrong way?

Thanks again.

[snwagh]

About (1), the dividing has to be using Shamir's secret sharing not just dividing into 3 parts. In particular, each parties data will contain 60,000 images, such that when you add these 3 databases, you end up with MNIST. Take a look at the Section 3.1 here and try to thoroughly understand the content (maybe use Wiki as well)

About (2), you generate a single set of pre-trained data. Ideally you want to split this into 3 parts as described in my response to (1) above. For a quick check, you can generate one pre-trained data and the other two parts can be zeros and this is easy to do. The code to generate zeros is also included in the codebase.