search

prebid / Prebid.js

Setup and manage header bidding advertising partners without writing code or confusing line items. Prebid.js is open source and free.
https://docs.prebid.org
Apache License 2.0
1.28k stars 2.05k forks source link

Proposal: Standardizing First Party Data #3687

Closed bretg closed 4 years ago

bretg commented 5 years ago

Type of issue

Feature

Description

Provide a standard way for the page to supply first party data and control which bidders have access to it.

A number of adapters support taking key/value pairs as arguments, but they're all different. e.g.

Proposal

1) Establish a standard convention for where pages and apps can place first party data 2) Give the publisher control over which bidders are allowed to see the first party data 3) Adapters can then be updated to read from the standard location, mapping values to their bidder-specific locations. 4) We don't support AdUnit-specific user data, as that scenario doesn't appear to have use-cases. 5) Use OpenRTB conventions where possible

Proposed conventions

A) Global (cross-adunit) First Party Data open to all bidders

pbjs.setConfig({
   fpd: {
       context: {
           keywords: "power tools, cars",
           search: "drill",
           content: { userrating: 4 },
           data: {
               pageType: "article",
               category: "tools"
           }
        },
        user: {
           keywords: "a,b",
           gender: "M",
           yob: "1984",
           geo: { country: "ca" },
           data: {
              registered: true,
              interests: ["cars"]
           }
        }
    }
});

B) Global (cross-adunit) First Party Data open only to a subset of bidders:

pbjs.setBidderConfig({
   bidders: ['bidderA', 'bidderB'],
   config: {
       fpd: {
           context: {
               keywords: ["power tools"],
               search: "drill",
               content: { userrating: 4 },
               data: {
                  pageType: "article",
                  category: "tools"
               }
            },
            user: {
               keywords: ["a","b"],
               gender: "M",
               yob: "1984",
               geo: { country: "ca" },
               data: {
                  registered: true,
                  interests: ["cars"]
               }
          }
      }
   }
});

C) AdUnit specific values (always available to all bidders)

    var adUnits = [{
        code: '/19968336/header-bid-tag-1',
        mediaTypes: {
            banner: {
                sizes: sizes
            }
        },
       fpd: {
         context: {
            adUnitSpecificContextAttribute: "123"
         }
        },
        bids: [{
           ...
        }]
    }];

Examples of adUnit-specific context values are: position and viewability. Note that user-specific first party data is not supported per-adunit.

The design of these attributes is based on OpenRTB -- attributes like keyword and search are specifically mentioned in OpenRTB. Anything not part of the OpenRTB-standard should go in the 'data' section.

To support publisher's ability to constrain which bidders can access the context or user data, this feature will utilize the bidder-specific config described in https://github.com/prebid/Prebid.js/pull/4334 . i.e..

Bid adapters that call getConfig() will receive either their bidder-specific first party data or the global first party data as appropriate.

As a corollary, any data placed directly in the AdUnit will be global.

Client-side adapters

Bid adapters should call getConfig("fpd") and translate the values to the syntax required by their endpoints. If the adapter's protocol is OpenRTB-like, see below for a mapping for where these values are placed for Prebid Server.

Prebid Server Bid adapter

The Prebid Server Bid adapter needs to have access to the bidder-specific first party data and which bidders are eligible.

The PBJS values would be copied to these OpenRTB locations:

1) set of bidders allowed to receive global ext.data --> ext.prebid.data.bidders[] 2) PBJS global context.data --> OpenRTB site.ext.data or app.ext.data 3) PBJS global context.ATTRIBUTE --> OpenRTB site.ATTRIBUTE (i.e. keywords, search) 4) PBJS global user.ATTRIBUTE --> OpenRTB user.ATTRIBUTE (i.e. keywords, gender, yob, geo) 5) PBJS global user.data --> OpenRTB user.ext.data 6) adunit-specific context.data --> OpenRTB imp[].ext.context.data 7) adunit-specific context.ATTRIBUTE --> OpenRTB imp[].ext.context.ATTRIBUTE (i.e. keywords, search) 8) bidder-specific fpd --> OpenRTB ext.prebid.bidderconfig

To align with what PBJS will be doing, permissions apply only to site.ext.data, app.ext.data, and user.ext.data.

Example OpenRTB location for scenarios 1-7 above:

{
    ext: {
       prebid: {
           data: { bidders: [ 'bidderA', 'bidderB' ] }
       }
    },
    site: {
         keywords: "",
         search: "",
         ext: {
             data: { GLOBAL CONTEXT DATA } // only seen by bidders named in ext.prebid.data.bidders[]
         }
    },
    user: {
        keywords: "", 
        gender: "", 
        yob: 1999, 
        geo: {},
        ext: {
            data: { GLOBAL USER DATA }  // only seen by bidders named in ext.prebid.data.bidders[]
        }
    },
    imp: [
        ext: {
            context: {
                keywords: "",
                search: "",
                data: { ADUNIT USER DATA }
            }
         }
    ]

Example OpenRTB location for scenario 8 above:

{
    ext: {
       prebid: {
           bidderconfig: [ {    // similar syntax as pbjs.setBidderConfig()
               bidders: [ 'bidderA', 'bidderB' ],
               config: {
                  fpd: { site: { ... }, user: { ...} }
               }
            },{
               bidders: [ 'bidderC' ],
               config: {
                  fpd: { site: { ... }, user: { ...} }
               }
            }]
       }
    },

As it's passing the OpenRTB request to each server-side bid adapter, Prebid Server core will: 1) look for ext.prebid.data.bidders -- if present, and current bidder is not on the list, remove site.keywords, site.search, site.ext, user.keywords, user.gender, user.yob, user.geo, and user.ext. 2) look for ext.prebid.bidderconfig and loop through the values - if one applies to the current bidder, will overwrite the site and user data. Support a "*" value that applies to all bidders. Remove ext.prebid.bidderconfig before passing to the adapter.

Note that, ext.prebid.bidderconfig with a "*" scope is the same as having globally scoped first party data.

SDK

The SDK would place values in the same OpenRTB fields as noted above for PrebidServerBidAdapter.

AMP

Prebid Server should support first party data from AMP as well.

json='{ "targeting": {
   bidders: ["*"],
   adunit: {
       context: {
           viewability: 0.6
       }
   },
   config: {
     fpd: {
       context: {
           keywords: "power tools, cars",
           search: "drill",
           content: { userrating: 4 },
           data: {
               pageType: "article",
               category: "tools"
           }
        },
        user: {
           keywords: "a,b",
           gender: "M",
           yob: "1984",
           geo: { country: "ca" },
           data: {
              registered: true,
              interests: ["cars"]
           }
        }
    }
  }
 }
}';

Only bidder-specific FPD is supported, which degrades to global when the bidder is "*".

Prebid Server core should copy the AMP targeting to adunit.context, site.ext.data, site.ATTRIBUTE, user.ext.data, user.ATTRIBUTE for relevant bidders before sending to the adapter.

AntoineJac commented 5 years ago

@bretg , that looks good to me, to respect OpenRTB should we also add:

To jump on your open questions, for the adunit-specific, would it be just enough to pass KV:

So we would have:

  1. global context.data - site.ext.data / app.ext.data
  2. global context.keywords - site.keywords / app.keywords
  3. global user.data - user.ext.data
  4. global user.keywords - user.keywords
  5. adunit-specific data - imp[].ext.data
mkendall07 commented 5 years ago

Looks like a good start Bret.

do we really need adunit-specific user data? That may be over complicating things.

I don't think we need it.

Bid adapters would look for global values with getConfig() along with adunit-specific values and handle them in an exchange-appropriate way.

What if instead the core just copied the global params into the adUnits if the adUnit did not override? That way the adapter only has to look at one location? This could also be accomplished with some utility function so we don't have the same code duplicated across all the adapters.

and finally I guess we'd need a decent transition period where we accept both until all the publisher can update.

astephensxandr commented 5 years ago

I theoretically support the alignment of first-party data APIs to a Prebid-wide standard, but I agree with @mkendall07 that significant thought should be given to making the implementation as simple as possible.

bretg commented 5 years ago

What if instead the core just copied the global params into the adUnits if the adUnit did not override?

I don't feel strongly, but my thought had been that global context is much more common than adunit-level overrides. Seemed like unnecessary string processing and extra memory use to schlep objects from the global config into 10 child adunits.

we'd need a decent transition period where we accept both

By "we", you mean each adapter, right? I'd think that adapters would accept the new location and their legacy location for a long time, yes. Pubs would be able to update their pages once the bidders they pass first party to can read from the new location. I'd think we don't want the pubs to have to pass values in both locations. So I'd think we'd want to send a note to all the adapter owners that support first party data so they can make it happen sooner rather than later.

bretg commented 5 years ago

Updated description to incorporate:

bretg commented 5 years ago

Sounds like we need more discussion in the internal interface, but are we settled on the external interfaces? i.e. the openRTB locations and pbjs.setConfig({context}) and pbjs.setConfig({user})

@AntoineJac would like to get started with SDK updates.

bszekely1 commented 5 years ago

I'd think we don't want the pubs to have to pass values in both locations. So I'd think we'd want to send a note to all the adapter owners that support first party data so they can make it happen sooner rather than later.

As much as we tell publishers not to, we should handle the situation where a publisher passes data in both locations. Example would be if they pass the old way, and simply add the new method suggested above.

If a publisher does pass in both locations, i would imagine that we would take preference of the new location over the old one and not do validation of matching or diff'ing. If new location present, use it and ignore old data, otherwise use old location.

bretg commented 5 years ago

As much as we tell publishers not to, we should handle the situation where a publisher passes data in both locations.

Sure, it's fine if they do, but if I were a pub, I'd prefer to make only one set of changes once all the adapters relevant to me are ready to support the standard location.

So rather than passing FPD into 4 different places, they could simplify and setConfig once.

Again, I don't feel strongly about the PBJS adapter interface -- will leave that to @mkendall07.

AskRupert-DM commented 5 years ago

agree on the suggestion that pubs would rather pass FPD once as opposed to 4 different places however do also think you probably need to cater for situations where data maybe passed into two locations (very much like DFP - you can define page-level and slot-level targeting)...

some real-example use-cases / scenarios I've seen this being needed/used are:

(1) Sending positional information to the bid-request - eg: pos=mpu1 vs pos=mpu2 or pos=atf/btf

(2) Providing viewability scores - some SSPs can't target by viewability and some publishers and SSPs are working around this by passing some sort of identifier in the bid-request for the ad-slot to pass some sort of viewability score - eg: view=hi/med/low or view=80 (slot has a average viewability of 80%).

(3) Provide exclusions - I've seen scenario's where publishers have universal/one SSP ID in place across all their slots - so in order to flag whether a particular type of creative format (expandable ad/display ad with video) can serve or not - they pass KVPs at the adUnit/slot level to identify this. cat-exclude=video - as they may not want to exclude this as the placement level.

This will go a long way in helping enrich bid-request - right now its a lot of work for publishers to send FPD to all SSPs as they all have different requirements as to how to send this which just complicates and delays integrations - along with the fact they then have to apply that potentially to every ad-slot. Hope that's useful feedback !

mkendall07 commented 5 years ago

for the ORTB schema - what if the data is already defined in the spec? for example: global user.gender Would that get copied to user.ext.data.gender and then copied back? Or the client side adapter should do that work by checking if it's predefined?

bszekely1 commented 5 years ago

for the ORTB schema - what if the data is already defined in the spec? for example: global user.gender Would that get copied to user.ext.data.gender and then copied back? Or the client side adapter should do that work by checking if it's predefined?

The two locations would have different purposes. The user.gender could potentially be passed to the DSP in addition to providing a transparent personally identifiable attribute, whereas first party data is typically not passed to the DSP, is used for Deal targeting and is treated as just data (not as yob, gender, etc.). The use case for this solution is to come up with a generic method to pass free form data, regardless of the contents of the data.

mkendall07 commented 5 years ago

thanks @bszekely1 for clarifying. I'm good for this on the ORTB side. I'll think about the PBJS interface a bit more next week.

bretg commented 5 years ago

Good input all. I've updated the proposal to list out other standard OpenRTB locations and to clarify:

Any attribute not part of the OpenRTB-standard should go in a 'data' (ext) section. Any attribute outside of a 'data' (ext) section is meant for direct DSP consumption, while attributes in the 'data' section may not necessarily go to DSPs, perhaps utilized for determining deal eligibility.

@AskRupert-DM - I updated to add a specific mention of your use cases. I believe they were accommodated before, but examples can always help.

Now we just need to decide the internal interface. My $0.02... I believe that FPD values can be large in some cases (hundreds of bytes), and there can be many adunits on a page (~10), so I feel that copying global FPD to each adunit isn't necessary. Adapters will have to copy values to their outgoing requests in either case. We'd spend fewer client cycles if we let adapters copy data if they care. And we wouldn't lose the potentially useful information about whether an attribute was defined globally or locally.

bretg commented 5 years ago

Discussed in this week's Prebid.js meeting. It came to light that publishers will want to control which adapters have access to first party data. Two approaches were covered:

1) Keep this design -- publishers would not put sensitive data into this feature. Any data placed in the context or user objects would be cross-bidder.

2) Support a list of bidders which which supplied FPD is allowed. This would change the design, as the setConfig system is currently global. A couple of options:

a) add a generalized 'ACL' feature to setConfig. e.g. `pbjs.setConfig(CONFIG_OBJECT, [bidder array])` Bidders that call `getConfig()` will receive global objects and those specific to their bidderCode.

b) create a new API to register first party data. e.g. `pbjs.setFpd(CONFIG_OBJECT, [bidder array])` . Bidders would call a new `getFpd()` function.

e.g.

pbjs.setConfig({
      context: {
          data: {
              pageType: "article",
              category: "tools"
          }
      },
      user: {
          gender: "M",
      }
}, ["rubicon","appnexus"])`

Support in Prebid Server would need to be modified as well. Proposal:

The PBJS values would be copied to these OpenRTB locations:

Would propose that permissions apply only to the ext fields: site.ext.data, user.ext.data, and imp.ext.site.data -- anything put in a standard OpenRTB field is available to all bidders.

Prebid Server core would look for ext.prebid.data.bidders -- if present, it would pass those values only to the named bidder adapters.

patmmccann commented 5 years ago

This proposal looks really strong to us and addresses our needs much better than option one.

bszekely1 commented 5 years ago

This would have to extend to Prebid SDK as well.

Implementation would be best / easiest to perform the ACL method vs registering them. Keep everything in one spot.

For clarity, the ACL is for access to information in the entire context object. This just serves as a reminder in the event we decide to add something to the context object. It's easy to forget these details.

Otherwise, i like this design.

bretg commented 5 years ago

In a future sprint, we plan to start implementing the internal interface where adapters need to look in two places -- global and adunit-specific.

jaiminpanchal27 commented 5 years ago

@bretg @mkendall07

Config module provides global object shared with all of prebid code + modules. Do we want to pollute this module with permissions ? We will need to add bunch of checks when someone other than allowed bidders tries to access first party data.

I would suggest to create a new new api setFirstPartyData and store separately.

At high level it will look somewhat like this

let options = {
  context: {},
  user:{},
  allowedBidders: []
}
pbjs.setFirstPartyData(options)

We would create new module to store this data. Data stored here will not be exposed to anything outside of src folder.

AdapterManager calls callBids method defined in each adapter. We would add new property firstPartyData on bidderRequest only for allowed bidders. Doing it this way will make sure that nobody other than allowed bidder is able to access data and bidders also do not need to call any method to get data.

bretg commented 5 years ago

Obviously, we got delayed on this implementation, but as @jaiminpanchal27 is questioning the design, let's discuss in this week's PBJS meeting.

We considered the security angle, and thought that was easily solvable with code inspections. Adapters simply shouldn't be looking in the context object and we can prevent that.

mkendall07 commented 5 years ago

one possible solution would be to make context a "special" case that cannot be accessed by the adapters. So the interface to the publisher seems consistent, while we control the adapter interface and only pass the data allowed to each bidder.

Thoughts?

bretg commented 4 years ago

This item has been on the PBJS agenda every 2 weeks since august and we've never gotten to it. We're going to move forward with building this feature as specified in the description.

1) adding new APIs (e.g. setFirstPartyData()) isn't desirable, IMO. We got away from that in the early days of PBJS to avoid proliferation of external functions. 2) having per-bidder set-config seems like a useful feature beyond first party data 3) code inspection can prevent adapters from sniffing the global context. If that turns out not to work, we can weigh the system down with a mechanism of preventing adapters from accessing context

mkendall07 commented 4 years ago

I agree with points 1 and 2 but not on point 3. We don't want to continue to add burden on core team reviewers to enforce a set of rules. We need to do things programmatically. I prefer a solution that enforces on the code level.

bretg commented 4 years ago

What does enforcement look like for this rule? Javascript doesn't know who's calling it.

We could take this opportunity to create a separate adapter scanning utility integrated into the build. Every time there's a build, this utility would automatically check certain rules about every file with the naming pattern *BidAdapter.js: 1) there are no calls to getConfig("context") 2) there are no references to HTTP that aren't followed by an "S" 3) Verify code under review has at least 80% unit test coverage 4) check for reference to $$PREBID_GLOBAL$$

we could add other programmatic checks in the future

bretg commented 4 years ago

@msm0504 and I came up with a different way to handle the security here. The drawback is that bidder adapters will need to look in two places for first party data.

1) when setConfig() is called with a list of allowedBidders, that data is not merged with the global config data. 2) such protected data is not available via getConfig() 3) protected data is made available only on bidrequest.protected.ATTR, where ATTR can be context, user, or any other non-global field.

Publishers can use the first party data feature in two ways:

1) global -- just use regular setConfig(). Adapters will pull getConfig("context") to obtain the values 2) protected -- use the new 'protected' option on setConfig(). Adapters will need to check bidrequest.protected.context to obtain the values

Yes, this is a little complex, but so are the requirements. And there really aren't that many bidders that support first party data right now, so this won't effect that many adapters.

mkendall07 commented 4 years ago

@bretg I like this protected approach with sending privileged data on the bid request object 👍

msm0504 commented 4 years ago

I've opened a pull request with my implementation of this proposal: https://github.com/prebid/Prebid.js/pull/4269

There is one potential issue that I know of. After the protected data is stored, I needed to create a way for the adapter manager to get the appropriate piece for an adapter. There is a new function called getProtectedData that will return the accessible protected data for a given bidder code. NO adapter should call this. It should only be used in the adapter manager (or anywhere else in the core code if needed later).

I think checking for this should be added to the adapter review checklist. What do you think?

bretg commented 4 years ago

Updated the design to utilize the new setBidderConfig design described in https://github.com/prebid/Prebid.js/pull/4334

I think that eliminates the need for a getProtectedData() function. (?)

bretg commented 4 years ago

@snapwich pointed out in the pull request that this proposal doesn't cover the Prebid Server scenario where there's global FPD available to all bidders but bidder-specific FPD as well.

Here's an attempt to fill that gap:

1) define a new ext.prebid.bidderconfig array. 2) support overriding global FPD for groups of bidders

{
    ext: {
       prebid: {
           bidderconfig: [ {   // same syntax as pbjs.setBidderConfig()
               bidders: [ 'bidderA', 'bidderB' ],
               config: {
                  fpd: { site: { ... }, user: { ...} }
               }
            },{
               bidders: [ 'bidderC' ],
               config: {
                  fpd: { site: { ... }, user: { ...} }
               }
            }]
       }
    },
    site: { ... },
    user: { ... }
bretg commented 4 years ago

Added AMP FPD convention

msm0504 commented 4 years ago

@bretg Is it ok if the new ext.prebid.bidderconfig section doesn't have the exact same format as what was passed into setBidderConfig?

Currently, the contents of setBidderConfig are separated out and stored in a map by bidder:

{
    bidderA: {
        fpd: { ... }
    },
    bidderB: {
        fpd: { ... }
    }
}

Combining to have ext.prebid.bidderconfig.[].bidders be an array would require additional logic to determine which bidders have the same first party data.

bretg commented 4 years ago

@msm0504 - Would like to have the syntax support a bidder array, but PBS-bidAdapter can implement this in a straightforward way with duplicate data.

e.g.

pbjs.setBidderConfig({
bidders: ['bidderA', 'bidderB'],
   config: {
      CONFIG_BLOCK
   }
});

Could translate to this OpenRTB:

ext: {
  prebid: {
    bidderconfig: [{
       bidders: ['bidderA'],
       config: {
           CONFIG_BLOCK
       }
     },{
       bidders: ['bidderB'],
       config: {
           CONFIG_BLOCK  // same block as A
       }
     }
});
patmmccann commented 4 years ago

Notably index takes first party data for all slots that persist for all future requests. We have a use case where we would like to pass different information for each slot. Does this issue solve that use case?

It seems like @mkendall07 suggested removing this from the proposal on march 27? Am i reading correctly that it is / was removed? In other words, what are the current thoughts on ad unit supplements or overrides to page level first party data? Our use case is similar to browsi or optimera and described in the index doc, in which we are attempting to qualify requests for pmps based on combinations of predicted viewability and the categorization of the surrounding content

For reference from the index bidder params doc:

Setting First Party Data (FPD) FPD allows you to specify key-value pairs which will be passed as part of the query string to IX for use in Private Marketplace Deals which rely on query string targeting for activation. For example, if a user is viewing a news-related page, you can pass on that information by sending category=news. Then in the IX Private Marketplace setup screens you can create Deals which activate only on pages which contain category=news. Please reach out to your IX representative if you have any questions or need help setting this up.

To include FPD in a bid request, it must be set before pbjs.requestBids is called. To set it, call pbjs.setConfig and provide it with a map of FPD keys to values as such:

pbjs.setConfig({ ix: { firstPartyData: { '': '', '': '', // ... } } }); The values can be updated at any time by calling pbjs.setConfig again. The changes will be reflected in any proceeding bid requests.

AskRupert-DM commented 4 years ago

Notably index takes first party data for all slots that persist for all future requests. We have a use case where we would like to pass different information for each slot. Does this issue solve that use case?

I had previously suggested some use-cases where I can see the need for this also - https://github.com/prebid/Prebid.js/issues/3687#issuecomment-479550814

patmmccann commented 4 years ago

Notably index takes first party data for all slots that persist for all future requests. We have a use case where we would like to pass different information for each slot. Does this issue solve that use case?

I had previously suggested some use-cases where I can see the need for this also - #3687 (comment)

Great point, yes we are currently developing towards one of those use cases exactly.