search

ucbrise / piranha

Piranha: A GPU Platform for Secure Computation
MIT License
92 stars 26 forks source link

Low accuracy of LeNet #11

Open llCurious opened 1 year ago

llCurious commented 1 year ago

Hey. I try to run the experiments for LeNet model. However i noticed that the convergence of LeNet is weird. The test accuracy of LeNet remains low (far from 98%). I only changed the localhost_config.json to run locally with the following configurations:

  1. 3PC computation based on Falcon
  2. network: files/models/lenet.json
  3. batch size: 128
  4. epoch: 5
  5. lr: 2^-4
  6. precision: 16

The output is as follows: image

llCurious commented 1 year ago

Hey, @jlwatson. Sry to bother you. But do you have any ideas?

jlwatson commented 1 year ago

Hey! I can't immediately run some tests, but as a starting point, could you try running it with models/lenet-norelu.json?

That's the actual LeNet architecture we evaluated with, it just doesn't have the last relu layer, which doesn't interact well with our loss function.

llCurious commented 1 year ago

Thanks to you quick reply!! I just run the model of lenet-norelu. The accuracy results seems to work fine. BTW, do you mean for other networks like secureml, minionn and so on, I should remove the last relu layer? image

jlwatson commented 1 year ago

Hey just to follow up, you should use the versions without the last ReLU layer -- it plays nicer with our loss function. And LeNet is pretty easy to train, so I'm not very surprised to see high accuracy after only 5 rounds. Looks like things are working properly.

On Fri, Apr 7, 2023 at 2:36 AM llCurious @.***> wrote:

I just check the figures on the paper. The presented figure is only about 97% for training 10 epochs. The above figures that I obtain is for 5 epochs. Is there something wrong with this part? [image: image] https://user-images.githubusercontent.com/21785695/230585340-9c2b1514-730d-4432-a4a6-ca75932d5a23.png

— Reply to this email directly, view it on GitHub https://github.com/ucbrise/piranha/issues/11#issuecomment-1500121530, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMIC4622H26EBOFTXHCA7DW77N3TANCNFSM6AAAAAAWVKBAQY . You are receiving this because you were mentioned.Message ID: @.***>

allenlin316 commented 2 weeks ago

I also tried the same training using local_running.sh with MNIST dataset with files/models/lenet-norelu.json model. But somehow I only got the result as below image. I wonder what did I do wrong? Thanks in advance! image

allenlin316 commented 2 weeks ago

I also tried the same training using local_running.sh with MNIST dataset with files/models/lenet-norelu.json model. But somehow I only got the result as below image. I wonder what did I do wrong? Thanks in advance! image

I just solved it by remaking the Makefile with 23 bits because of paper mentioned LeNet secure training of neural network command remake is as: make -j8 PIRANHA_FLAGS="-DFLOAT_PRECISION=23 -DTWOPC"