Closed michal-kalisz closed 9 months ago
Hi Michal,
Appreciate you posting this feedback. I'm going to close this as I believe we have addressed some of these through the private aggregation stable ramp up and the proposal to add labels, which you are already engaged on. Please feel free to split out any remaining feedback into separate issues.
Thanks Asha
The aim of the tests is to verify the possibility of using the Private Aggregation API to collect statistics of Fledge auctions and events related to it, specifically the registration of impressions and clicks on the banner. In most cases, our banners consist of a main banner (ad) and "product" ad components.
What mechanisms will we be using?
We register only 1% of histograms related to bid (point 1 and point 3 reserved.loss).
During the tested period, the stable version of Chrome was 113. Private Aggregation API was enabled on 1% Origin Trial, Extended Private Aggregation Reporting in FLEDGE was available for 115 (dev/canary) chrome version. 114 is the beta chrome version.
The following results are based on reports received over 7 consecutive days, limited to 5 days based on the schedule date:
Time
Time between registering a histogram and the schedule date value.
Let's compare the moment of receiving the report with the histogram registration
Comparing PAA reports with other sources
Impressions
Reported through contributeToHistogram
During the test period, in the stable version (113), the Private Aggregation API was enabled for 1% of OT traffic, while the Protected Audience accounted for 6%. So expected #first report CTH / imp ratio is ~ 1/6.
Reported through contributeToHistogramOnEvent(‘reserved.win’, …)
The extension for the Private Aggregation API was primarily available for version 115, which is why the quantity of reports and impressions is relatively small and susceptible to noise.
For the reserved.loss reports, the values were passed in separate buckets using signalValue with bid-reject-reason. The received values were consistent with the forDebuggingOnly reports from the bidding function.
Comparing debug reports to normal reports.
Normal reports are understand as “non-debug” reports retrieved by /.well-known/private-aggregation/report-protected-audience endpoint
All reports
Reports sent for 1% of bids through contributeToHistogram
Reports sent for all impressions through contributeToHistogram
Summary
During our testing of the Private Aggregation API in Fledge auctions, we encountered a limitation related to reaching the limit of 1000 pending reports in the browser. That’s why we decided to report only 1% of bids.
One notable observation was the difference between debug reports and normal reports, with a gap as high as 5%.
Currently, there is a waiting period of up to 12 hours to receive 95% of the reports. This delay in report delivery can impact the timeliness of data analysis and machine learning processes.
Furthermore, we would like to utilise the Private Aggregation API for various purposes such as machine learning, reporting, and system monitoring. Each of these use cases has distinct characteristics, with some requiring data to be delivered as quickly as possible (e.g., monitoring), while others prioritise data accuracy and precision. However, it is important to note that reports originating from a specific hour (exact schedule date truncated to the hour) can only be processed once by the Attribution Service (AS).
Expanding the scale of our tests is necessary to achieve credible results and effectively evaluate the Extended Private Aggregation Reporting in FLEDGE. By conducting tests on a larger scale, we can gather more comprehensive data, gain more accurate insights.