Generate Test Cases for Evaluations

[betuldurak]

There were several proposals discussed regarding to Attribution Reporting API with Aggregate Reports. All are based on MPC with different tradeoffs. Moving forward, I propose that we decide what aggregate functions are the most meaningful and evaluate the proposals efficiency with desired settings. How do we want to proceed and make progress? Should we set up a meeting to discuss and generate some test cases?

[martinthomson]

For us to start developing test cases, we need agreement on what the goals are and what sort of information leakage is acceptable. That is, what is the output of the system? (Not so much the specific values, but the types of leakage.)

[AramZS]

During tests participants should be able to talk to whether costs are particularly blocking of any participants.

[betuldurak]

For us to start developing test cases, we need agreement on what the goals are and what sort of information leakage is acceptable. That is, what is the output of the system? (Not so much the specific values, but the types of leakage.) So, one idea on how we want to proceed: agreement on the goal and acceptable leakage. Do you mean the goal as in the end-application? Acceptable leakage: there are theoretical-level leakages of proposals with different threat models. What more concrete step we can take to agree on that? @martinthomson Do you mean to analayze the leakage in implementation level?

[martinthomson]

This is where I think that having tests is getting a little too far ahead of the other developments. When we discussed IPA with a few cryptographers a few weeks back it became clear to me (as it was already clear to others) that the precise nature of the information leakage from the design has a somewhat intimate relationship with the performance of the system. We have to carefully balance that leakage in order to get acceptable performance in all senses: computation, communication, and utility for advertising use cases.

We still have a range of designs that are fairly divergent in certain ways, with different ways of approaching the problems. So it might be difficult to agree that one particular type of leakage (for instance, the number of events that are input to the system from a single user might leak to MPC participants) needs to fit a particular envelope when other designs have a completely different approach that doesn't include that leakage at all.

[betuldurak]

What you shared is quite useful @martinthomson. I think I was not clear with the intention with this issue, perhaps not well chosen (order of) words. I don't think my first goal is to start testing efficiencies, instead to make progress with proposals and their evaluations such as deciding on settings (my first question) I think, the type of leakage you mention with IPA would be good to have discussions on to understand these settings for evaluations.

My other question was how we want to proceed and make progress. Am I understanding you well that since we do not seem to be easily agree on a particular type of leakage, we should continue working on proposals within our groups/circles? If that is what each proposal wants to do, I want to understand that.

Last question was if we should set up a meeting to discuss and generate some test cases (again bad choice of unclear words). I am slowly understanding the way the CG is working so please help me understand this. Is there anyone who is interested in discussing more to understand what would be the most important to prioritize to continue improving on or the road blocker of their proposals? I don't think it would be pleasing to talk to myself in a vacuum if there is no interest on having an open conversation about what each proposal's shortcomings and strengths. We see that Mozilla guys are quite involved and invested with Prio, or Google folks experimenting with Prio/heavy-hitters (or Poplars -kind of confusing as @ekr was saying Mozilla gave the problem for heavy-hitters but works on another proposal called Poplar for the same problem. Doesn't seem to be the case.) I think it is interesting to hear your progress with IPA. I would be interested in hearing more from experts on Prio and Poplar/heavy-hitters if there are any shortcomings they want to work on or improve or address such as an unexpected leakage/adding DP. I will admit that I am not as clear as I should be with the real-world implications of DP any of these proposals bringing (including Bucketization which I am a co-author of). I am thinking the group would benefit discussing the real-world guarantees of DP as much as MPC and TEE.

I am not ignoring whole TEE vs MPC discussions. I am only trying to make progress with MPC part by discussing. Again, unless there is an interest to discuss these, I cannot proceed. So, please help me understand if there is any interest to extend any of these in next calls. I am happy to work with any of you to make progress and hear more about others progress.