search

DachengLi1 / MPCFormer

(ICLR 2023 Spotlight) MPCFormer: fast, performant, and private transformer inference with MPC
89 stars 11 forks source link

Question about results #4

Closed TheSunWillRise closed 1 year ago

TheSunWillRise commented 1 year ago

Hello, thanks for a great job. I have a question about the results in Table 2 and 4. Are the acc results in these tables obtained by conducting MPC inference on encrypted data or obtained on original test data? Do you test how mpc process affects acc performance?

DachengLi1 commented 1 year ago

@TheSunWillRise It is based on the data in plain-text. I haven't tested the acc in mpc since Crypten reports there should be no difference.

However, I highly suspect this may change when the model size is large. Because essentially, mpc is doing some sort of quantization, and quantization in LLMs is known to introduce some issues. I think this worths another paper honestly.

TheSunWillRise commented 1 year ago

That's right. I also think the encrypting process could affect the actual accuracy due to the quantization. I'm trying to check this out based on your codes.

DachengLi1 commented 1 year ago

Great! Let me know if you find anything and happy to talk about this more. I always want to know this but didn’t get enough bandwidth.

DachengLi1 commented 1 year ago

Closing this @TheSunWillRise Feel free to re-open if you have follow up.