prebid / prebid-server

Open-source solution for running real-time advertising auctions in the cloud.
https://prebid.org/product-suite/prebid-server/
Apache License 2.0
425 stars 730 forks source link

Analytics enhancement: return nobids, errors, and removals #2367

Open bretg opened 2 years ago

bretg commented 2 years ago

Client-side analytics adapters don't have access to what actually happened with PBS bids unless they're successful.

The ORTB protocol was designed to carry just actual successful bids, not information about the status of all bid attempts. However, with the Prebid world being part client-side and part server-side, holisitic analytics is made more difficult. It's quite complicated to log auction info/results separately: some client-side, some server-side and then sew them all together at scale on the backend.

e.g. say a request contains calls to bidderA, bidderB, and bidderC. If only bidderA responds, then the resulting seatbid[].bid[] contains only bidderA. The client doesn't know what happened to bidderB or bidderC: no bid? error? GDPR removal? Price floor removal?

So we would like to propose a new optional request flag that instructs PBS to include the status of all bid attempts in the response.

Proposal

1) Support a new request flag ext.prebid.returnallbidstatus: true - this causes PBS to create special entries in the response for every imp+bidder in the request, even nobids and errors.

This is independent from the debug/test option.

2) When the flag is set and a bidder doesn't have a bid for any reason, create a 'non-bid' response just before "Stage 8 - Auction Response"

ext.seatnonbid: [{
     seat: "bidderA",               // required - biddercode to be processed by client
     ext.prebid: {
         origseat: "bidderC"         // optional - aligning with issue #2424
         meta.adaptercode: "bidderM"  // optional - aligning with issue #2174
     },
     nonbid: [{
         impid: STRING.              // required, $.imp[].id
         statuscode: ENUM,           // required
         ext: {  },                  // for future extension
     }]
}]

We considered just collating this data in seatbid.bid, but we can't be sure that that all clients will know what to do with the extra info: e.g. SDK, SSAI servers, etc. Sure, they shouldn't be passing ext.prebid.returnallbidstatus if they can't handle the response, but it seems safer to place the non-bids in a different location.

We also considered whether to put this extension in ext.prebid.seatnonbid but decided that reporting on the non-bids is not prebid-specific.

3) The bids list included should be the list of imps+bidders present just before Stage 3 - Processed Auction Request.

4) Some reasons a bidder might not have a bid response include:

Each of these scenarios should be considered test cases. It's not necessary for PBS-core to be able to immediately be able to distinguish these. If there are scenarios where there's not enough info currently available to detail why a bid request/response was removed, it can supply a reason of "unknown" for now.

bretg commented 2 years ago

Here'a a proposed design. If the ext.prebid.returnallbidstatus flag is true, then as part of the ORTB response generation

Loop through the imps in the original request after the raw auction stage (do we have a snapshot?)
    Loop through each bidder in imp (imp[n].ext.prebid.bidder.*
        // Find which **requests** were dropped before they even went out
        Loop though the requests as sent to bid adapters (do we have a snapshot?)
            If the request wasn't sent to the adapter, create an ext.seatnonbid entry with code 'reqrm' and, if possible, details.message with why it was dropped (e.g. "gdpr")
        // Find which **responses** were dropped
        Else PBS should have some kind of bidresponse object, so while creating the seatbid entries, also create seatnonbid entries for "nobid", "error" and "resprm" scenarios.
            If no reason can be found for "error" or "resprm" types, ext.status.details.message should be "unknown"

Here's the ext.prebid.seatnonbid entry

            ext.prebid.seatnonbid: [{
              seat: "bidderB"
              nonbid: [{
                 impid is set to $.imp[].id
                 ext.status.code is set to "nobid", "error", "reqrm", "resprm"
                 ext.status.details[].code is set to a code // optional
                 ext.status.details[].message is set to a string version of the code // optional
              }]
            }]
bretg commented 1 year ago

Ok, here's another iteration on a syntax. Changes:

            ext.seatnonbid: [{
              seat: "bidderB",         // the actual bidder     required
              mediary: "bidderC",  // the bid adapter        optional
              nonbid: [{
                 impid is set to $.imp[].id        // required
                 status: {
                      code: ENUM,                      // required
                      detail: "above floor"          // optional
                  }
              }]
            }]

Where the initial code is an enum:

  1. nobid
  2. error
  3. request not sent to bidder
  4. bidder response not accepted

Some example details that Prebid Server could respond with:

code 2: error

code 3: request not sent to bidder

code 4: bidder response not accepted

SyntaxNode commented 1 year ago

Thank you for iterating on this @bretg.

I presented this proposal to the IAB Supply Chain Working Group (the publishers of OpenRTB) on behalf of Prebid and received a lot of encouragement to elevate this beyond a Prebid specification. The recommended path forward is to pursue a sanctioned extension with the potential to be included directly in OpenRTB based on how much use it gains. I suspect there will be a lot of momentum with Prebid publishers who are seeking greater transparency into the bidding process.

I'd like to share my opinions on how to best position this for potential future inclusion into OpenRTB as well some direct feedback I've received from the community.

removed the ext inside nonbid.

All OpenRTB objects define an ext to support future growth. I think it would be good to include seatnonbid.ext and nonbid.ext in the design.

status.code is now an enum

I like this approach. This is more in line with OpenRTB and I think makes the reason easier to parse. I explored the idea of reusing existing OpenRTB enums like nbr and the loss reason code. There may be some merit to including the nbr since we could directly forward the bidder's nbr value if supplied in their response and many of the loss reason codes are related to bid rejection. While there is clearly overlap with these exsiting enums, it's not a perfect mapping and the usage is notably different.

making details a singleton object and paring down to single text string

Good move. I foresee the majority of the time having only a single reason with multiple reasons being an edge case. In most situations, Prebid Server (and many other bidding services) will stop at the first error and not be able to provide a list of reasons.

How about we go one step further and push code and detail fields up to the nonbid object? I don't see a reason to keep the status as a sub-object when there is now just a single instance.

moving up to ext.seatnonbid (removing the prebid level)

Yay!

trying to fix the seat-vs-adapter issue. Leave 'seat' as it's supposed to mean in ORTB. Inventing a new term 'mediary' to refer to the local bid adapter name. e.g. if calling the appnexus endpoint could result in a bid from bidderX, we want both pieces of data.

This would break symmetry with the existing seatbid object. Perhaps it's best to leave this out for now and use the same ext solution for both seatbid and seatnonbid?

As for the enum, perhaps we could push more scenarios into the main enum and use the detail for reasons we cannot predefine at this moment - such the specific PBS module which rejected the request.

Value Description
1 No Bid
2 Timeout
3 Invalid Bid Response
4 Bidder Unreachable
100 Blocked - General
101 Blocked - GDPR
102 Blocked - Unsupported Channel (app/site/dooh)
103 Blocked - Unsupported Media Type (banner/video/native/audio)
104 Blocked - Optimized
200 Creative Filtered - General
201 Creative Filtered - Size Not Allowed
202 Creative Filtered - Not Secure
203 Creative Filtered - Incorrect Format
204 Creative Filtered - Malware
205 Creative Filtered - Advertiser Exclusions
300 Rejected - Below Floor
301 Rejected - Duplicate
500+ Vendor-specific codes.

.. and then for Prebid Server specifically:

Value Description
500 Prebid - Rejected By Module
501 Prebid - Make Bids Error
502 Prebid - Make Requests Error
bretg commented 1 year ago

Thanks @SyntaxNode - I'm not thrilled with the concept of enums, but will swallow that if that help get it through committee and we're going to be able to add values quickly.

So taking into account these comments, here's the next iteration:

ext.seatnonbid: [{
     seat: "bidderA",               // required - biddercode to be processed by client
     ext: {
         origseat: "bidderC"         // optional - aligning with issue #2424
         prebid.meta.adaptercode: "bidderM"  // optional - aligning with issue #2174
     },
     nonbid: [{
         impid: STRING.              // required, $.imp[].id
         code: ENUM,                 // required
         nbr: ENUM,                  // optional, IAB nbr applies only if code is 1
         detail: ENUM,               // optional
         ext: {  },                  // for future extension
     }]
}]

There was an internal conversation about whether the nobid scenario could be considered the default. The client can assume nobid if it knows bidderA should have a bid on impId=1, if it doesn't see anything explicit. I think this is ok, but from a PBS perspective, this doesn't work in a stored request scenario where the bidders aren't known client-side.

Would suggest adding a "Rejected - General" code as 300 in your list.

Lastly, trying to wrap my head around "seat", which has gotten complicated lately. Here are several scenarios:

  1. The seat bidderM is actually from bid adapterM. The seat matches the biddercode in the imp, which matches the bid adapter. Life is good. seat: bidderM
  2. bidderA is a hardcoded alias for bid adapterM. The adapter needs to know that it was called as "bidderA" and use that as the seat. seat: bidderA, adaptercode: bidderM
  3. bidderA is a soft alias for bid adapterM. Likewise, the adapter needs to know that it was called as "bidderA" and use that as the seat. seat: bidderA, adaptercode: bidderM
  4. Bid adapterM represents multiple backend demand sources, and so wants to inject multiple nobids. e.g. I'm adapterM, reporting a nobid from bidderB, bidderC, and bidderD. seat: bidderC, adaptercode: bidderM
  5. bidderA is an alias for adapterM, which represents multiple backend demand sources biddersB-D. seat: bidderA, adaptercode: bidderM, origseat: bidderC
bretg commented 1 year ago

Based on another round of comments, here's another cut:

ext.seatnonbid: [{
     seat: "bidderA",               // required - biddercode to be processed by client
     ext.prebid: {
         origseat: "bidderC"         // optional - aligning with issue #2424
         meta.adaptercode: "bidderM"  // optional - aligning with issue #2174
     },
     nonbid: [{
         impid: STRING.              // required, $.imp[].id
         code: ENUM,                 // required
         detail: ENUM,               // optional
         ext: {  },                  // for future extension
     }]
}]

changes:

Code Enum

There should be discussion about whether we need two enums. One advantage is that analytics systems will find it useful to roll-up nonbids into high level categories because the followup actions are different for each:

Value Description Interpretation Action
1 No Bid everything's ok, just no demand bidders with low demand may be re-evaluated or optimized
2 Error there's a technical problem the publisher, PBS host company, or bidder may need to investigate
3 Request Rejected this imp never went to this bidder may be expected due to server-side config
4 Response Rejected this bidder responded with a bid that was invalid bidder may need to resolve

But it could be argued that we should skip the high level rollup entirely and just offer one detailed status field. Servers that don't support fine-grained errors can just define the 'general' values. If we group the codes into 'series', analytics could still support rollup. See below.

Details Enum

Value Description
100 No Bid - general
103 No Bid - Known Web Crawler
104 No Bid - Suspected Non-Human Traffic
105 No Bid - Cloud, Data Center, or Proxy IP
106 No Bid - Unsupported Device
107 No Bid - Blocked Publisher or Site
108 No Bid - Unmatched User
109 No Bid - Daily User Cap Met
110 No Bid - Daily Domain Cap Met
111 No Bid - Ads.txt Authorization Unavailable
112 No Bid - Ads.txt Authorization Violation
113 No Bid - Ads.cert Authentication Unavailable
114 No Bid - Ads.cert Authentication Violation
115 No Bid - Insufficient Auction Time
116 No Bid - Incomplete SupplyChain
117 No Bid - Blocked SupplyChain Node
200 Error - general
201 Error - timeout
202 Error - bad input parameters
300 Request Blocked - General
301 Request Blocked - GDPR
302 Request Blocked - Unsupported Channel (app/site/dooh)
303 Request Blocked - Unsupported Media Type (banner/video/native/audio)
304 Request Blocked - Optimized
400 Response Rejected - General
401 Creative issue - General
402 Creative Filtered - Size Not Allowed
403 Creative Filtered - Not Secure
404 Creative Filtered - Incorrect Format
405 Creative Filtered - Malware
406 Creative Filtered - Advertiser Exclusions
410 Rejected - Below Floor
411 Rejected - Duplicate
500+ Vendor-specific codes.
SyntaxNode commented 1 year ago

moved origseat under ext.prebid

Looks great.

dropped nbr in favor of merging it with detail. Many nbr values don't apply to individual bidders and imps.

I was on the fence about this. I think I prefer the simplicity of a single enum. Folding the nbr values still allow us to provide a direct mapping which is "nbr + 100" in the current enum declaration.

But it could be argued that we should skip the high level rollup entirely and just offer one detailed status field. Servers that don't support fine-grained errors can just define the 'general' values. If we group the codes into 'series', analytics could still support rollup.

Both approaches have merit.

The code/detail pair allows folks to ignore the detail if they'd like and keep metrics/analytics high level. Although, it duplicates information, introduces the possibility for code and detail to be mismatch forcing a decision on how this should be handled, and increases the payload size a bit.

The single code value required code for high level rollup. Guidance will be provided in the spec and ranges should make it fairly easy. I would like us to use the range of <100 to keep more values <500 useful for codes.

I find myself leaning towards the simplicity of skipping the rollup and letting other systems perform grouping as they desire.

SyntaxNode commented 1 year ago

I would like to consider the name "code" within the context of existing OpenRTB enum field names. Here's the list from the original 2.6 spec:

cattax, batter, pos, expdir, api, startdelay, protocols, podseq, placement, linearity, skip, slotinpod, battr, playbackmethod, playbackend, delivery, companiontype, feed, nvol, prodq, context, qajmediarating, devicetype, connectiotype, type, ipservice, atype, source, nbr, attr, apis

I do not see a clear naming pattern. There appears to be a slight preference for including the term "type". Most are specific names, but there is currently use of "type" which seems just as generic as "code".

Should we give "code" a more specific name? We obviously can't call it a "no bid reason" and "non bid reason" isn't unique enough. I don't have any good ideas here. My thoughts are snbr for seat level no bid reason, but that's likely confusing with nbr.

I'm good keeping it with code, but wanted to put this out there if someone else has a name alternative.

bretg commented 1 year ago

Next iteration. Getting skinnier...

ext.seatnonbid: [{
     seat: "bidderA",               // required - biddercode to be processed by client
     ext.prebid: {
         origseat: "bidderC"         // optional - aligning with issue #2424
         meta.adaptercode: "bidderM"  // optional - aligning with issue #2174
     },
     nonbid: [{
         impid: STRING.              // required, $.imp[].id
         statuscode: ENUM,           // required
         ext: {  },                  // for future extension
     }]
}]

I would like us to use the range of <100 to keep more values <500 useful for codes.

Not sure I understand this @SyntaxNode -- can you elaborate, or even better, post the next iteration of a draft statuscode table?

SyntaxNode commented 1 year ago

Next iteration for the statuscode. I tried to use OpenRTB terminology: exchange instead of host, seat instead of bidder?

Status Code: Guidance

Status Code enumeration values are purposefully divided into the following ranges to assist with higher level classification: Range         Description                  Interpretation Action
0-99 No Bid Auction ran successfully without demand for the impression. Seats with low demand may be re-evaluated or optimized.
100-199 Error Technical problem occurred during the auction. Exchange and/or seat may need to investigate.
200-299 Request Rejected Impression was explicitly not sent to the seat. Bid Request should be re-evaluated for unsupported impressions. May be expected due to exchange configuration.
300-399 Response Rejected Seat responded with a bid that was rejected by the exchange. Seat may need to resolve.

If a bid was received with a nbr populated, set the status code to the nbr value. Exchanges are encouraged to provide as much detail as possible, but it is acceptable to use the general codes (0, 100, 200, 300) when details aren't known.

Status Code

Value Description
0 No Bid - General
1 No Bid - Internal Technical Error
2 No Bid - Invalid Request
3 No Bid - Known Web Crawler
4 No Bid - Suspected Non-Human Traffic
5 No Bid - Cloud, Data Center, or Proxy IP
6 No Bid - Unsupported Device
7 No Bid - Blocked Publisher or Site
8 No Bid - Unmatched User
9 No Bid - Daily User Cap Met
10 No Bid - Daily Domain Cap Met
11 No Bid - Ads.txt Authorization Unavailable
12 No Bid - Ads.txt Authorization Violation
13 No Bid - Ads.cert Authentication Unavailable
14 No Bid - Ads.cert Authentication Violation
15 No Bid - Insufficient Auction Time
16 No Bid - Incomplete SupplyChain
17 No Bid - Blocked SupplyChain Node
100 Error - General
101 Error - Timeout
102 Error - Invalid Bid Response
103 Error - Bidder Unreachable
200 Request Blocked - General
201 Request Blocked - Unsupported Channel (app/site/dooh)
202 Request Blocked - Unsupported Media Type (banner/video/native/audio)
203 Request Blocked - Optimized
204 Request Blocked - Privacy
300 Response Rejected - General
301 Response Rejected - Below Floor
302 Response Rejected - Duplicate
350 Response Rejected - Invalid Creative
351 Response Rejected - Invalid Creative (Size Not Allowed)
352 Response Rejected - Invalid Creative (Not Secure)
353 Response Rejected - Invalid Creative (Incorrect Format)
354 Response Rejected - Invalid Creative (Malware)
355 Response Rejected - Invalid Creative (Advertiser Exclusions)
500+ Vendor-specific codes.

I changed "Request Blocked - GDPR" to the more general "Request Blocked - Privacy" as we're at the start of a privacy regulation explosion which will be difficult to keep up with.

bretg commented 1 year ago

Thanks @SyntaxNode - thumbs up from me.

bretg commented 1 year ago

Ok - I'm going to go ahead and prioritize this work for the PBS-Java team, with the goal of being done in January. I recognize that this syntax is provisional and could change as the IAB discussions continue. We'll change it if needed. Will hold off on the Prebid.js side for another few weeks.

bretg commented 1 year ago

As far as PBS implementation, I would suggest phasing this in, because carrying some of the extra detail is not currently a part of the module infrastructure.

Phase 1 - Basic

In this phase:

If the ext.prebid.returnallbidstatus flag is true, then as part of the ORTB response generation

Loop through the imps in the original request after the raw auction stage
    Loop through each bidder in imp (imp[n].ext.prebid.bidder.*
        // Find which **requests** were dropped before they even went out
        Loop though the requests as sent to bid adapters
            If the request wasn't sent to the adapter, create an ext.seatnonbid entry with status code 200 for this imp/seat // no details at this point
        // Find which **responses** were dropped
        Else PBS should have some kind of bidresponse object, so while creating the seatbid entries, also create ext.seatnonbid entries with these statuscodes:
            if no-bid, statuscode=0
            else if timeout, statuscode=101
            else if other error, statuscode=100
            else if response rejected due to price floors, statuscode=301
            else if response rejected for some other reason, statuscode=300

Note: if it's possible to make this functionality a module, that would be fine. Somehow I think that would make it a lot more difficult and not bring a lot of value.

Phase 2 - Rich metadata

Much later in 2023 (or beyond0, this phase expands the feature:

bretg commented 1 year ago

Got some interesting internal feedback that ought to be considered -- if there are 10 imps in an auction and a bidder is rejected, there's a lot of duplication. i.e. it would be rather redundant to say "bidderA rejected by GDPR for imp1, rejected by GDPR for imp2, rejected by GDPR for imp3, ..."

Some options: 1) allow a wildcard in the impid to apply that status to all imps in the auction. Not sure if there's a convention for ORTB for wildcards -- *, null? Or perhaps just define that impid is optional, and if not present, that indicates 'all imps'. 2) add a status at the same level as nonbid. Could be awkward to have to check status in multiple locations.

I prefer #1 with an explicit impid: "*" signal, though am ok with absence of impid implying wildcard.

bretg commented 1 year ago

Internal feedback is that we shouldn't encourage wildcards until we know for sure this is a problem.

This issue will be taken up in the appropriate IAB committee.

bretg commented 1 year ago

Discussed in PBS committee that we're aligned on these points:

bretg commented 1 year ago

@ShriprasadM has proposed adding bid fields to the nonbid object. This covers the case where there was an actual bid that was rejected (e.g. floors) and the client-side analytics wants to have insight into the key bid parameters.

{
ext.seatnonbid: [{
     seat: "bidderA",
     ext: {
       origbidcpm: FLOAT, // optional
       origbidcur: STRING, // optional
       prebid: {
         origseat: "bidderC"
         meta.adaptercode: "bidderM"
       }
     },
     nonbid: [{
         impid: STRING.              // required, $.imp[].id
         statuscode: ENUM,           // required
         // optional bid params
         price: FLOAT,
         cur: STRING,
         adomain: STRING,
         cattax: INTEGER,
         cat: STRING_ARRAY,
         dealid: STRING,
         w: INTEGER,
         h: INTEGER,
         dur: INTEGER,
         mtype: INTEGER,
         ext: {  },                  // for future extension
     }]
}]
}
bretg commented 1 year ago

@SyntaxNode - did you get a chance to consider the request for additional fields above?

bretg commented 1 year ago

FYI - PBS-Java 1.108 supports the most basic version of this feature. It returns only code 0 at this point.

bretg commented 1 year ago

Discussed with @SyntaxNode and @ShriprasadM . We're suggesting placing any actual bid params in nonbid.ext.prebid.bid. This will make the IAB conversation easier. The idea is that any of the fields that could cause a post-bid rejection can be placed here so that client-side analytics adapters can get is.

{
ext.seatnonbid: [{
     seat: "bidderA",
     ext: {
       origbidcpm: FLOAT, // optional
       origbidcur: STRING, // optional
       prebid: {
         origseat: "bidderC"
         meta.adaptercode: "bidderM"
       }
     },
     nonbid: [{
         impid: STRING.              // required, $.imp[].id
         statuscode: ENUM,           // required
         ext: {
           prebid: {
             bid: {
               price: FLOAT,  // converted bid after currency and adjustments
               cur: STRING,   // converted bid currency
               origbidcpm: FLOAT,
               origbidcur: STRING,
               adomain: STRING_ARRAY,
               cattax: INTEGER,
               cat: STRING_ARRAY,
               dealid: STRING,
               w: INTEGER,
               h: INTEGER,
               dur: INTEGER,
               mtype: INTEGER
           }
         }
     }]
}]
}
bretg commented 1 year ago

Copied from the pull request https://github.com/prebid/prebid-server/pull/2505#discussion_r1158652639


I pretty strongly disagree with adding the bid object to the nonbid. This issue lays out the fields that are needed in the nonbid. If there are fields of interest to analytics that are missing, please feel to suggest.

Specifically, I will not accept passing adm through the nonbid. That field can be many KB large and would be a waste of space. To a lesser concern, the various url fields (burl, nurl, etc) would also a waste of network bandwidth.

Because of those fields, I think it's unwise to pretend that we can "just pass the whole seatbid.bid object". But I will add it to the agenda for the next meeting. If SyntaxNode and the committee think it's acceptable to just strip the long-and-useless fields, I will abide.

pm-harshad-mane commented 1 year ago

@ShriprasadM can you please share your thoughts on this issue?

ShriprasadM commented 1 year ago

@pm-harshad-mane : I have already added my comments on that PR. here is the link - https://github.com/prebid/prebid-server/pull/2505#discussion_r1158834062

pm-harshad-mane commented 1 year ago

Thanks for the update @ShriprasadM, can you please resolve the conflicts and take #2505 to the conclusion?

ShriprasadM commented 1 year ago

@pm-harshad-mane : Sure, Will resolve it.

bretg commented 1 year ago

Updated ext.seatnonbid.nonbid.ext.prebid.bid to include origbidcpm and origbidcur. Note orig and not original to align them with the extension PBS supports in the seatbid. https://docs.prebid.org/prebid-server/endpoints/openrtb2/pbs-endpoint-auction.html#original-bid-cpm

@ShriprasadM - in the writeup you have in the PR, origbidcpm and origbidcur are described as being outside the bid object. The existing origbidcpm and origbidcur extensions are actually in bid.ext. Are you ok with just placing them in ext.seatnonbid.nonbid.ext.prebid.bid ?

bretg commented 1 year ago

The official list of codes is now at https://github.com/InteractiveAdvertisingBureau/openrtb/blob/master/extensions/community_extensions/seat-non-bid.md