Passing userIds to Prebid Server using AMP

ian-lr commented 3 years ago

It is not currently possible to pass userIds (e.g. user.ext.eids) to Prebid Server via AMP. While cookie-based syncs are enabled with the amp-iframe, this limits the information available to bidders processing requests from AMP pages. Given the growing popularity of both userIds and AMP, this seems like a good opportunity to extend Prebid Server capability.

Acceptance Criteria

Publishers should be able to pass known userIds to supported bidders via a well-defined interface in amp-ad.
Bidders should be able to process userIds sent from AMP pages in a standard eid format.

Note, as AMP-RTC does not support eids , userIDs will likely need to be added to the query string or through some other Prebid-specific interface. A potential strawman follows.

<amp-iframe width="1" title="User Sync with eids"
  height="1"
  sandbox="allow-scripts"
  frameborder="0"
  src="https://<PBSERVER_DOMAIN>/load-cookie.html?endpoint=appnexus&max_sync_count=5&eids=%5B%7B%22source%22%3A%22liveramp.com%22%2C%22uids%22%3A%5B%7B%22id%22%3A%22AovIJXGIWHHMhHyOeQiDk0_rtTQ--fVkmWU7xftkAh9rxgUeLHBcsoUE6gdZwFFYmvAJXw%22%2C%22atype%22%3A1%7D%5D%7D%5D">
  <amp-img layout="fill" src="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==" placeholder></amp-img>
</amp-iframe>

bretg commented 3 years ago

I'd rather not store the eids in the PBS cookie. 1) that cookie can already get big. 2) more privacy headaches

How about we just define a new macro USER_IDS in the RTC vendors:

  prebidrubicon: {
    url:
      'https://prebid-server.rubiconproject.com/openrtb2/amp?tag_id=REQUEST_ID&w=ATTR(width)&h=ATTR(height)&ow=ATTR(data-override-width)&oh=ATTR(data-override-height)&ms=ATTR(data-multi-size)&slot=ATTR(data-slot)&targeting=TGT&curl=CANONICAL_URL&timeout=TIMEOUT&adc=ADCID&purl=HREF&gdpr_consent=CONSENT_STRING&account=ACCOUNT_ID&userids=USER_IDS',
    macros: ['REQUEST_ID', 'CONSENT_STRING', 'ACCOUNT_ID', 'USER_IDS'],
    disableKeyAppend: true,
  },

Then the publisher would be responsible for passing the eids array in:

  <amp-ad width="300" height="50"
    type="doubleclick"
    data-slot="/11111/amp_test"
    data-multi-size-validation="false"
    rtc-config='{"vendors": {"prebidrubicon": {"REQUEST_ID": "14062-amp-AMP_Test-300x250"}, "ACCOUNT_ID": "1001", USER_IDS="%5B%7B%22source%22%3A%22id5-sync.com%22%2C%22uids%22%3A%5B%7B%22id%22%3A%22ID5-12345%22%7D%5D%7D%2C%7B%22source%22%3A%22netid.de%22%2C%22uids%22%3A%5B%7B%22id%22%3A%2211111111%22%7D%5D%7D%5D"}}'
    json='{ "targeting": {"site": {"tags": "autoestima","url": "/amp/familia/materias/33559-princesa-africana-da-disney-lembra-por-que-toda-crianca-precisa-se-sentir-representada"}}}' >
  </amp-ad>

We could come up with a short-hand for this JSON, but would rather not PBS have to understand it, and there are exceptions to the general rule of source+id. e.g. TDID has an ext, as will SharedId.

ian-lr commented 3 years ago

@bretg I like that approach better! I wasn't sure how much flexibility we have on the vendor macros, but this is more approachable and simpler from my perspective.

SyntaxNode commented 3 years ago

Discussed and accepted by Prebid Server Committee.

We'll just need to work out the JSON format. To keep things simple, does Base64 encoded JSON work for you? PBS would decode the Base64 was representation and parse it as eids. If there are no parse errors, it will be set as-is in the request. If there is a parse error, should this be an error or a warning?

ian-lr commented 3 years ago

@SyntaxNode From my perspective, that would work well. I would suggest throwing an error unless there is precedent (other macro validation?) to do things differently. In most cases, I'd expect PBS would expect a valid user ID string.

SyntaxNode commented 3 years ago

We need to settle on the eid format to begin implementing. Is a base64 encoded json blob alright? Do we want to go with escaped json as you originally included? Did you @bretg want to suggest a short hand encoding?

ian-lr commented 3 years ago

Base64 JSON blob works for me. To illustrate, something like:

let encodedEids = btoa(JSON.stringify(pbjs.getUserIdsAsEids())) will be sent through, then PBS will do the equivalent of JSON.parse(atob(encodedEids))?

pycnvr commented 3 years ago

Would this approach work with cached pages? It seems a customized page has to generated per user, but cache on google and bing serve the same copy.

ian-lr commented 3 years ago

@pycnvr Good question. Could you use something like https://amp.dev/documentation/components/amp-access/ to get user-specific data without invalidating the cache?

pycnvr commented 3 years ago

@ian-lr Possibly. Not sure how to link dynamic values to rtc-config, though. The list of available macros is controlled by doubleclick rtc.

ian-lr commented 3 years ago

@pycnvr OK, let me see if I can experiment a bit with this and see if I can get the cache working, too.

SyntaxNode commented 3 years ago

@pycnvr OK, let me see if I can experiment a bit with this and see if I can get the cache working, too.

Sounds good. We'll hold off implementing for now.

bretg commented 3 years ago

We don't need to use macros. We can evolve the First Party Data feature to support eids similar to the way proposed for the Publisher-Provided User ID feature in https://github.com/prebid/Prebid.js/issues/5690

[deleted obsolete straw example. see below for the most recent proposal]

pycnvr commented 3 years ago

@bretg As far as I can tell, AMP is not making this easy. When a user does a google search, and click on one of the AMP links, it's actually a page cached by google. So the publisher doesn't even have a chance to fill in the first party data. The properties in amp-ad are static, except for those that can be supplied via macros.

For example, the following are the same page but served from different domains.

Google Cache from cdn.ampproject.org https://www-si-com.cdn.ampproject.org/v/s/www.si.com/.amp/soccer/2020/09/10/lionel-messi-cristiano-ronaldo-lead-fifa-21-player-rankings?usqp=mq331AQFKAGwASA=&amp_js_v=0.1
Publisher si.com https://www.si.com/.amp/soccer/2020/09/10/lionel-messi-cristiano-ronaldo-lead-fifa-21-player-rankings?usqp=mq331AQFKAGwASA=&amp_js_v=0.1

bretg commented 3 years ago

@pycnvr - can arbitrary javascript run on an AMP page? Does it have DOM access to tags?

If so, could that javascript scan the tags and add/update data like either the 'rtc-config' or the 'json'?

We haven't finalized how we're going to pass eids to Prebid Server -- anyone have thoughts about whether that's easier either way?

bretg commented 3 years ago

Nevermind. I found the reference that <amp-script> tags can't currently create <amp-ad> tags. https://amp.dev/documentation/components/amp-script/

ian-lr commented 3 years ago

@bretg I found this AMP Issue tagged with INTENT TO IMPLEMENT that, depending on the implementation, could be useful: https://github.com/ampproject/amphtml/issues/28095

bretg commented 3 years ago

That's interesting @ian-lr , though I'm not sure how helpful it will be if it's tied to Permutive. No one seems to have pushed back on a vendor-specific tag.

adamleslie commented 3 years ago

@bretg non-Permutive specific implementation:

https://github.com/ampproject/amphtml/issues/30193

LMK if we can help test a PBS execution on our domains once the above is built

SyntaxNode commented 3 years ago

@bretg @adamleslie Is this spec complete / ready to implement?

bretg commented 3 years ago

AMP 30193 is still in the proposal stage. Could take a while for them to implement. My understanding is that it would allow dynamic 'json' targeting to be applied to amp-ad RTC. (corrections welcome)

In the meantime, we can discuss how to pass that data through to Prebid Server. If it's on the json field, I'd propose we come up with a way to pass an eids structure through along with data permissioning as described in https://github.com/prebid/Prebid.js/issues/5814

Here's a straw:

  <amp-ad width="300" height="50"
    type="doubleclick"
    data-slot="/11111/amp_test"
    data-multi-size-validation="false"
    rtc-config='{"vendors": {"prebidrubicon": {"REQUEST_ID": "14062-amp-AMP_Test-300x250"}, "ACCOUNT_ID": "1001"}}'
    json='{"targeting":{"eids":[{"source": "example.com", "uids": [{"id": "11111111"}]}], "eidPermissions": [{"source": "example.com", "bidders": ["bidderA"]}]}}' >
  </amp-ad>

Prebid Server would look in the targeting field for "eids", validate it, and inject into user.ext.eids. Assuming an EIDs permissioning scheme is approved, it would also handle that.

ian-lr commented 3 years ago

@bretg Returning back to this, do you think that we should hold off until the eid permissioning is locked in before proceeding with the suggested straw here?

bretg commented 3 years ago

AMP has a feature now that allows RTC to call a script. But I'll have to admit I'm not connecting the dots for how to integrate the two pieces. Here's the example on the AMP page

    <amp-ad width="320" height="50"
            type="doubleclick"
            data-slot="/4119129/mobile_ad_banner"
            rtc-config='{"urls": ["amp-script:targetingFns.getTargeting"]}'
    </amp-ad>
    <amp-script nodom data-ampdevmode id="targetingFns" script="targetingFnsScript"></amp-script>
    <script id="targetingFnsScript" type="text/plain" target="amp-script">
      exportFunction("getTargeting", () => {
        return {
          targeting: {food: ["chicken", "beans"]},
          categoryExclusions: ["sports", "food", "fun"]
        };
      });
    </script>

From this example it's not clear that we can use the standard vendors in rtc-config. (?) But this example seems like a high level "idea" because the getTargeting function doesn't actually return a URL.

I'm not particularly fond of a making pubs hardcode the PBS URLs in their pages because we change the vendor config sometimes. e.g. we recently added AMP consent fields.

Maybe one of the Prebid AMP experts can weigh in on the possibilities here?

philipwatson commented 3 years ago

I'm currently experimenting with this feature (though I'm not an AMP expert). You can use the standard vendors in the rtc-config: simply by adding the "vendors" field. eg:

rtc-config='{"vendors": {"prebidappnexus": {"PLACEMENT_ID": "13144370"}}, "urls": ["amp-script:targetingFns.getTargeting"], "timeoutMillis":1000}'

The script RTC callouts don't return a url. They return JSON containing the key/values. So for doubleclick's AMP-AD, we want to return the same structure as your example (he same as prebid-server's RTC response).

Not sure if you can configure a vendor in rtc-config to be a script RTC (in AMP's callout-vendors.js).

However, you can use an external script to reduce publisher maintenance: <amp-script nodom id="targetingFnsScript" src="https://example.com/script.js" data-ampdevmode></amp-script>

As far as I know, you can't use the MACROS with scripts. Instead, the script will get dynamic values from amp-analytics.

{
    "transport": {"amp-script": true},
    "requests": {
      "pageview": "amp-script:targetingFns.receiveFn"
    },
    "vars": {
      "clientId": "CLIENT_ID(ampId)",
    },
    "triggers": {
      "trackPageview": {
        "on": "visible",
        "request": "pageview"
      }
    },
    "extraUrlParams": {
      "cid": "${clientId}",
      "canonicalUrl": "${canonicalUrl}"
    },
...

And in your script:

exportFunction("receiveFn", (msgObj) => {...});

The msgObj will have the contents of extraUrlParams.

Not sure about consent unfortunately.. but you can get other bits of info. Documented here.

Not sure how to get the EIDS. The script runs in webworker (and inside another iframe) so it has no access to local storage. So might only be limited to TP cookie.

Also RTC callouts happen at the same time. Not processed/merged sequentially. So prebid-server RTC callout can't get results from another callout. Or you guys thinking about changing PBS callout to script callout?

Hope this info helps a bit.

bretg commented 3 years ago

Thanks @philipwatson , but I'm not following.

you guys thinking about changing PBS callout to script callout

We don't care what the AMP syntax is. Publishers that want to pass dynamic data (like user IDs) can use a different syntax if needed. The original use case can stay with the currently documented 'vendors' approach. We'll document whatever other scenarios are necessary.

Not sure how to get the EIDS

In order to be useful for Prebid Server, the requirement is straightforward: some kind of AMP setup that gathers the desired dynamic data and passes it through to the Prebid Server /amp endpoint.

script RTC callouts

What is this -- is it ["amp-script:targetingFns.getTargeting"]?

You also say "The script RTC callouts don't return a url.", but why then is ["amp-script: in the 'url' section of rtc-config?

bretg commented 3 years ago

Discussed in committee today. It does seem possible to thread the need here, but that may not be the most valuable thing to do here.

Assuming that a url protocol of amp-script: tricks the system into calling a local script rather than making an HTTP call, here's a general approach that might be made to work to get dynamic values from an AMP page into PBS:

    <amp-ad width="320" height="50"
            type="doubleclick"
            data-slot="/4119129/mobile_ad_banner"
            rtc-config='{"urls": ["amp-script:targetingFns.getTargeting"]}'
    </amp-ad>
    <amp-script nodom data-ampdevmode id="targetingFns" script="targetingFnsScript"></amp-script>
    <script id="targetingFnsScript" type="text/plain" target="amp-script">
      exportFunction("getTargeting", () => {
            // grab dynamic values off page, add them to a PBS URL
            // use XHR to call the PBS URL
            // return the response so the amp-ad block can add targeting values to the ad server call
      });
    </script>

The only things we'd need to do are:

define a new parameter for IDs. I'd go for something simple like &eids=URL-ENCODED-EIDS-BLOCK. Then PBS would merge the value of this param into user.ext.eids.
document the approach

FWIW, I couldn't get this approach to work, but didn't spend much time with it. If someone has an example they could post here, that would be great.

Alternate approach

But in the meeting, an alternate approach was proposed: use a new host company cookie to store the eids block.

1) define a new PBS ID cookie. e.g. 'eids' 2) update the /setuid endpoint to accept an new 'eid' parameter

/setuid?eid=URL-ENCODED-EIDS-ENTRY

3) when /setuid sees this parameter, it updates the 'eids' cookie with the additional value 4) when /auction or /amp are called, the eids cookie is parsed and merged into user.data.eids 5) Customize load-cookie.html to call an ID endpoint and parse the response into the new /setuid?eid (lots of details here to work out)

Any thoughts about either of these approaches?

philipwatson commented 3 years ago

Hi @bretg Yes, the "script" RTC callout is ["amp-script:targetingFns.getTargeting"]. Regarding the url: I mean the function that this RTC points to (getTargeting) doesn't return a URL. Sorry, I think I misunderstood what you said.

Not sure if readers are aware, but getTargeting can return a Promise. Obviously needed because of the PBS request. Also needed if using amp-analytics to receive dynamic values - because the receiveFn (using my example) and getTargeting can be called in any order. Though this could be dependent on the trigger/event being used. Note: I found out recently that you can get consent string via amp-analytics.

Might have time to create an example later today or tomorrow.

SyntaxNode commented 2 years ago

@philipwatson I think an example would be great, when your time allows. Thank you.

philipwatson commented 2 years ago

Hi @SyntaxNode Here is my example.

But there is a problem: AMP does not support query params on the script URLs. This is needed to pass in placement-level parameters placementId, slot, width, height, timeout, etc. So this example is only good if you have one placement.

I added a feature request for this on the amphtml project: https://togithub.com/ampproject/amphtml/issues/35097

<!doctype html>
<html amp lang="en">
    <meta charset="utf-8">
    <script async src="https://cdn.ampproject.org/v0.js"></script>
    <script async custom-element="amp-analytics" src="https://cdn.ampproject.org/v0/amp-analytics-0.1.js"></script>
    <script async custom-element="amp-ad" src="https://cdn.ampproject.org/v0/amp-ad-0.1.js"></script>
    <script async custom-element="amp-script" src="https://cdn.ampproject.org/v0/amp-script-0.1.js"></script>
    <title>Hello AMP</title>
    <link rel="canonical" href="https://amp.dev/documentation/guides-and-tutorials/start/create/basic_markup/">
    <meta name="viewport" content="width=device-width,minimum-scale=1,initial-scale=1">
    <style amp-boilerplate>body{-webkit-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-moz-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-ms-animation:-amp-start 8s steps(1,end) 0s 1 normal both;animation:-amp-start 8s steps(1,end) 0s 1 normal both}@-webkit-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-moz-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-ms-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-o-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}</style><noscript><style amp-boilerplate>body{-webkit-animation:none;-moz-animation:none;-ms-animation:none;animation:none}</style></noscript>
</head>
<body>
<h1>Test</h1>

<amp-analytics>
    <script type="application/json">
        {
            "transport": {
                "amp-script": true
            },
            "requests": {
                "pageview": "amp-script:targetingFns.receive"
            },
            "vars": {
                "clientId": "CLIENT_ID(foo,,foo,false)"
            },
            "triggers": {
                "trackPageview": {
                    "on": "visible",
                    "request": "pageview"
                }
            },
            "extraUrlParams": {
                "accountId": "12345",
                "consentString": "CONSENT_STRING",
                "curl": "${canonicalUrl}",
                "purl": "${ampdocUrl}",
                "gdprApplies": "CONSENT_METADATA(gdprApplies)",

                "placementId": "13144370",
                "slot": "/19968336/universal_creative",
                "timeout": "1000",
                "width": "300",
                "height": "250"
            }
        }
    </script>
</amp-analytics>

<amp-script nodom data-ampdevmode id="targetingFns" script="targetingFnsScript"></amp-script>
<script id="targetingFnsScript" type="text/plain" target="amp-script">
var receivedParams;

function getTargetingFromPBS(params) {
  var url = "https://prebid.adnxs.com/pbs/v1/openrtb2/amp?";
  url += "tag_id=" + params.placementId;
  url += "&timeout=" + params.timeout;
  url += "&w=" + params.width;
  url += "&h=" + params.height;
  url += "&slot=" + encodeURIComponent(params.slot);
  url += "&curl=" + encodeURIComponent(params.curl);
  url += "&purl=" + encodeURIComponent(params.purl);
  url += "&gdpr_consent=" + params.consentString;

  return fetch(url, {method: 'GET'})
    .then(function (response) {
      return response.json();
    })
}

exportFunction('receive', function(msg) {
  receivedParams = msg;
});

exportFunction('getTargeting', function() {
  return new Promise(function (res, rej) {
    var start = new Date().getTime();
    function tryResolve() {
      if (receivedParams) {
        getTargetingFromPBS(receivedParams)
          .then(function(json) {
            res(json);
          });
      }
      else if (Date.now() - start < 1000) {
        setTimeout(function () {
          tryResolve();
        }, 250);
      }
    }
    tryResolve();
  });
});
</script>
<amp-ad width="300" height="250"
        type="doubleclick"
        data-slot="/19968336/universal_creative"
        rtc-config='{"urls": ["amp-script:targetingFns.getTargeting"], "timeoutMillis": 1000}'>
</amp-ad>
</body>
</html>

Note: can generate the amp-analytics config on the server-side. Adds other possibilities like adding TP cookie values. From the docs:

<amp-analytics
  config="https://example.com/analytics.account.config.json"
></amp-analytics>

SyntaxNode commented 2 years ago

Thank you @philipwatson. To be clear, this example depends on AMP implementing this feature request?

philipwatson commented 2 years ago

Hi @SyntaxNode
The example works as-is. But we run into trouble if we want to support multiple ads on the page. There are "hacky" workarounds.

It only depends on it if we want a nice, harmonious solution. This is where we need the AMP implementation.

An alternative workaround, as mentioned on that feature request, is having a separate analytics config for each . This does not feel good.

Another possible workaround that came to mind recently is associating the function name with the subset of configs. Not perfect but feels better.

For example:

<amp-ad width="300" height="250"
        type="doubleclick"
        data-slot="/19968336/universal_creative"
        rtc-config='{"urls": ["amp-script:targetingFns.getTargeting1"], "timeoutMillis": 1000}'>
</amp-ad>
<amp-ad width="320" height="50"
        type="doubleclick"
        data-slot="/19968336/universal_creative"
        rtc-config='{"urls": ["amp-script:targetingFns.getTargeting2"], "timeoutMillis": 500}'>
</amp-ad>

And in analytics configs:

"extraUrlParams": {
  ...
  "placementId1": "13144370",
  "slot1": "/19968336/universal_creative",
  "timeout1": "1000",
  "width1": "300",
  "height1": "250",

  "placementId2": "98383113",
  "slot2": "/19968336/universal_creative",
  "timeout2": "500",
  "width2": "320",
  "height2": "50"
}

And in the script:

exportFunction('getTargeting1', function() {
    // know we want to use slot1, placement1, etc
});
exportFunction('getTargeting2', function() {
    // know we want to use slot2, placement2, etc
});

I tried this and it works.

We also have to keep in mind that the script runs inside a web worker. So we won't have access to FP storage (window.localStorage, document.cookie). We would always have to rely on making successful calls to the external providers on every ad request. However, if I read it right, Google may make the storage api less restrictive - by allowing 3p scripts access to the storage API provided that the page is delivered from the publisher's domain - not from cache or AMP Viewer (google.com). I2I here: https://togithub.com/ampproject/amphtml/issues/30872

bretg commented 2 years ago

Appreciate the proposal @philipwatson, but honestly this solution is beyond my ability to understand. What do "analytics configs" have anything to do with RTC ads? I don't follow what would be in the getTargeting1 function. Maybe you could post a well-commented full page solution tying all of these pieces together?

Am guessing that this approach would preclude use of the 'simple' vendor-callout approach of RTC and force publishers to manually create the entire Prebid-Server URL.

At a high level then, we would define a place on that URL for user IDs to flow. e.g.

...&eids=URL-ENCODED-EIDS-BLOCK&...

Could the getTargeting1 function handle creating an encoded eids block?

philipwatson commented 2 years ago

Hi @bretg

What do "analytics configs" have anything to do with RTC ads?

Because it's the only way to get placement parameters into the getTargeting1 function. You can't have query parameters on the script RTC URL. So the workaround is to add the placement parameters to the parameters we can get from amp analytics (the extraUrlParams field of the analytics config).

I don't follow what would be in the getTargeting1 function. Maybe you could post a well-commented full page solution tying all of these pieces together?

Sorry, the snippets are based on the code on a previous comment (https://github.com/prebid/prebid-server/issues/1404#issuecomment-877994378). You'll modify the targetingFnsScript to be something like this:

var receivedParams;

function getTargetingFromPBS(params, id) {
  var url = "https://prebid.adnxs.com/pbs/v1/openrtb2/amp?";

  // placement specific params
  url += "tag_id=" + params["placementId" +id];
  url += "&timeout=" + params["timeout" + id];
  url += "&w=" + params["width" + id];
  url += "&h=" + params["height" + id];
  url += "&slot=" + encodeURIComponent(params["slot" + id]);

  // general params
  url += "&curl=" + encodeURIComponent(params.curl);
  url += "&purl=" + encodeURIComponent(params.purl);
  url += "&gdpr_consent=" + params.consentString;

  // Now the reason why we are here: add the user ids somehow
  // url += "&eids=" + encodeURIComponent(params.eids);
  // -- or --
  // fetch eids from server(s)? could probably start this sooner in the 'receive' exported function?
  // can also receive parameters from AMP that may be useful for some user id providers. e.g. SCREEN_WIDTH 
  // for fingerprinting. but can come with a problem: getting blacklisted by browsers
  // unfortunately can't cache results here for future page loads

  return fetch(url, {method: 'GET'})
    .then(function (response) {
      return response.json();
    })
}

function run(id) {
  return new Promise(function (res, rej) {
    var start = new Date().getTime();
    function tryResolve() {
      if (receivedParams) {
        return getTargetingFromPBS(receivedParams, id)
        .then(function(json) {
            res(json);
        });
      }
      else if (Date.now() - start < 1000) {
        setTimeout(function () {
          tryResolve();
        }, 250);
      }
    }
    tryResolve();
  });
}

exportFunction('receive', function(msg) {
  receivedParams = msg;
});

exportFunction('getTargeting1', function() {
    return run(1);
});

exportFunction('getTargeting2', function() {
    return run(2);
});

Am guessing that this approach would preclude use of the 'simple' vendor-callout approach of RTC and force publishers to manually create the entire Prebid-Server URL.

Not sure I follow here. The script above creates the prebid server url.. by using the parameters received. And I think the script can be hosted externally.

I think the script rtc url can be a custom url callout or vendor-specified callout (an entry in AMP's vendor.js). Remember, we don't have access to the chosen adnetworks macros (like doubeclick's ATTR(data-override-height) or any of the ones provided by AMP (like CONSENT_STRING) because, again, query parameters are not supported for RTC script urls. So the publisher do not need to worry about this stuff when using the custom url callout.

However the publisher, integrator or something will need to provide the analytics config for the placement parameters. This too can be hosted externally.. this comes with the ability to generate the config dynamically. The server generating this config can use tp cookies and perhaps call external user id providers via server-to-server approach.

I must admit I haven't been keeping up with AMPs development since my last comment. Things may of changed.

OllyTriple13 commented 2 years ago

Hi @bretg, I thought I would wade in here if that is ok - I proposed the alternative suggestion mentioned in your comment of 25th June.

The current proposal seems very complex to me given there is already an ID management solution in place for third party IDs, so I still think that we should start here and extend the current solution before introducing a whole new paradigm. An additional potential benefit of using the same mechanism is that we might be able to enhance the effectiveness of the third party ID sync using 1st party IDs - e.g. if we don't have any third party IDs in a cookie, but we do have a first party ID that has previously had third party IDs attached, we can pre-populate the third party IDs.

I see two potential ways of this working, not mutually exclusive (i.e. we could support both):

Switch the usersync IFrame call out to a script hosted on the publisher, which acts as a proxy for the current usersync endpoint and adds userids into the payload along the way
Extend the current usersync config to support publisher specific endpoints that are called along with the bidder specific endpoints

The result of either method would be an additional user id cookie on the prebid server domain which would be used by the server to populate the appropriate part of the openrtb schema.

The obvious argument against this approach is that the usersync process is asynchronous and that it is likely not to have occurred before the first bid request is made in a cold start situation. So we would end up with a fair proportion of bids without the publisher provided ids, which is the current issue we have with third party IDs. My counter argument to this is the AMP philosophy is that the source is cacheable and processes should not be blocked on one another - so we are in a situation where we have to make another call to the publisher to get the IDs and this really shouldn't be blocking, which means we end up in an async situation anyhow. If we can solve this for publisher provided IDs, we should also apply it to third party IDs if we can.

So, in summary, to move forward, I suggest we extend the existing mechanism - give it a go, see what feedback we get, and improve from there, for all types of IDs.

bretg commented 2 years ago

@OllyTriple13 - this is a good idea to explore.

Publisher-specified IDs are different than 3rd party IDs in where they appear in the openrtb:

PBS faithfully slots each bidder's synced ID into the user.buyeruid field.
But publisher-specified IDs would go into user.ext.eids[]

In order for PBS to know to treat them differently, the values would need to either be stored in a different PBS cookie or in a special block within the existing uids cookie. I like the latter because adding cookies is a hassle -- host companies need to document which cookies they set and why.

Where to store these new things

So here's a proposal for where to store eids in the existing 'uids' cookie:

{
    "uids": {},
    "tempUIDs": {... this is where the /setuid endpoint puts values ...}
    "eids": [
                {
                    "source": "example.com",
                    "uid": "6bac6f03-9e81-40ac-a676-1f7853faa4d9",
                    "atype": 1,
                    "stype": "ppuid",
                    "expires": "2021-10-22T13:37:33.151793Z"
                }
    ]

Setting the eid values

To get values into this new field, we need to either expand the /setuid endpoint or create a new /seteid endpoint.

(A) Expand /setuid

We could expand this with a new mode on the existing endpoint:

/setuid?type=eid&source=example.com&atype=1&stype=ppuid&gdpr=&gdpr_consent=&f=i&uid=111111&ttl=TTL_IN_DAYS

(B) Create a new /seteid endpoint

If it's too hacky to shoehorn eids into /setuid, a new endpoint could be created

/seteid?source=example.com&atype=1&stype=ppuid&gdpr=&gdpr_consent=&f=i&id=111111&ttl=TTL_IN_DAYS

Writing user.ext.eids

The main auction process would be enhanced to read the new eids block of the cookie and expand it into user.ext.eids. e.g.

user.ext.eids: [
                {
                    "source": "example.com",
                    "uids": [   // if the same source is present multiple times, there would be multiple array entries here
                        {
                            "id": "6bac6f03-9e81-40ac-a676-1f7853faa4d9",  // required
                            "atype": 1,    // only if the atype parameter exists
                            "ext": {
                                "stype": "ppuid" // only if the stype parameter exists
                            }
                        }
                    ]
                }
    ]
}

Note that the same source could conceivably have multiple entries in the uids cookie.

Invoking /setuid or /seteid

Whichever endpoint, the next question is how does it get invoked? A couple of options

(A) Let the publisher come up with their own amp-iframe. It could solve the problem however they want -- hit an existing pub endpoint and parse it however they want -- all it has to do is drop a pixel to /setuid or /seteid.

(B) Expand load-cookie.html with a new argument to hit a defined endpoint

https://HOST/load-cookie.html?endpoint=appnexus&max_sync_count=5&source=amp&eidurl=https%3A%2F%2Fexample.com%2Furl-encoded%3Fpath

This endpoint would respond in some new format. e.g.

{
    "eids": [
        {
            "source": "example.com",
            "atype": true,
            "stype": "ppuid",
            "id": "11111",
            "ttl": NUM_DAYS
        }
    ]
}

Then load-cookie.html would parse the results and create the pixels for /seteid

bretg commented 2 years ago

We discussed this internally and agreed that it makes sense to consider this together with https://github.com/prebid/prebid-server/issues/1985 - the effort to shrink the PBS cookie with protobuf.

OllyTriple13 commented 2 years ago

@bretg apologies for the delay in response on this. I support this approach and it would be great to get it included in the new protobuf spec.

bretg commented 2 years ago

Update: the other 2 cookie-sync improvements (#2173 and #1986) have been deemed higher priority and big enough chunks to chew on. The Magnite team has these prioritized higher than either the protobuf (#1985) or this enhancement. Given the magnitude of this EIDs change with other cookie-related changes in progress, if there's another team considering doing this, I'd recommend close coordination via the PBS committee.

And given the issues we're having with the "full cookie" scenario, I propose expanding the cookie-sync.pri feature outlined in #2173 to include eids. I think we can just blend biddercodes and EID sources into the priority field. e.g.

cookie-sync.pri: ['bidderA', 'example.com']

The cookie-sync code will need to figure out what can get kicked out of the cookie in the increasingly common 'full' scenario.

bretg commented 1 year ago

PBS-Java has released the cookie-sync refactoring. Which is good, but putting eids into it is not currently supported. The uids cookie already only holds around 30 IDs and there are 100+ bid adapters that want a share of it.

bretg commented 10 months ago

@OllyTriple13 and @ian-lr - is this still relevant given the other various identify efforts the industry is working on?

jdelhommeau commented 8 months ago

Hi all, Catching up on this discussion, sorry if I missing part of it. My understanding is this thread is discussing about ability to pass eid in prebid server in AMP env, which is currently not supported. Various options were evaluated, but last one was to have eid set as cookies, using some dedicated / shared cookie endpoint (/setuid or /seteid). The AMP prebid server RTC code would then read that cookie to retrieve eid and pass them in the ad call. Correct? I am not sure to understand how the eid is collected in the first place here. Where is it picked from before being passed in the /seteid or /setuid url?

bretg commented 8 months ago

Hi @jdelhommeau - you must have been at the Summit yesterday where this was mentioned, eh? :-)

As noted, no one has stepped up to help pin this feature down. The proposal above is obviously in partly-done state:

Define a place to store the EIDs. Proposed is shoving the EIDs in the already way-too-small uids cookie.
Define a PBS endpoint to populate that storage location. Probably a new /seteid endpoint?
Define a mechanism to invoke that endpoint. The existing load-cookie or a new thing?
Define a convention for how AMP publishers share the EID data with that mechanism. (not the faintest idea here)

The AMP prebid server RTC code would then read that cookie to retrieve eid and pass them in the ad call. Correct?

Once the eids are stored in the uids (or another) cookie, yes, the job of Prebid-Server core is easy. Read it and add to the ORTB sent to bidders.

I am not sure to understand how the eid is collected in the first place here. Where is it picked from before being passed in the /seteid or /setuid url?

The people currently active in Prebid don't know the intricacies of AMP well enough to have a useful opinion here. The assumption made in the example above is that the publisher could set up a first party server (https%3A%2F%2Fexample.com%2Furl-encoded%3Fpath) that could return the EIDs in JSON. But perhaps there's some way to store EIDs in the javascript-poor environment that can just be read by load-cookie? Is there an equivalent of a "dom" or other local storage where EID data can be retrieved from a conventional location?

Publishers who rely on AMP need to help drive this feature with what's possible and desirable.

jdelhommeau commented 8 months ago

Thank you for your response! And no, I wasn't part of the prebid summit unfortunately, but glad to hear this was discussed :)

I am not sure the publisher first party server would work, because:

the publisher is often not the one coming up with the eid. Most of the time, this is provided to the pub by a 3rd party that does this by executing a script on the page. Not doable here, or at least, not through standard script
even if the publisher had somehow already established access as a first party cookie for the eid (I am guessing this is where you are going with with the first party server solution), AMP page are often loaded from a third party server (google's) so publisher cookie wouldn't be accessible

I need to look further into how an ID provider may establish his ID on an AMP page, and how it may be able to make that ID available to rest of the page (including prebid server).

Also, as an alternative to using the uids cookie, I believe we could rely on RTC fast fetch macro, to retrieve the eid from somewhere on the page, and pass it directly as parameter to the prebid server endpoint. This would likely simplify steps 1 to 3. Still remains the questions of where do we get the EID from in the first place.

I will start to look into it and keep this thread posted on my findings.

bretg commented 8 months ago

Great feedback. Thanks for looking into this @jdelhommeau .

If the EIDs could somehow be appended to the Prebid server /amp RTC call, that would be the best possible solution, IMO. e.g. "/amp?tag_id=BLAH&...&eids=ENCODEDDATASTRUCTURE

Putting new things into the already-crowded and soon-to-die 3rd party PBS cookie seems sub-optimal.

jdelhommeau commented 8 months ago

yes, agree with PBS cookie being sub optimal here.

What I am looking at is this: https://github.com/ampproject/amphtml/blob/main/extensions/amp-a4a/rtc-documentation.md#fast-fetch-implementation-defined-macros Which would allow to resolve a PBS macro by executing some js function that would retrieve the eid(s). The question that remain is, where does that function get the eid from :/ This is what I will be looking into. I will keep you posted.

adamleslie commented 8 months ago

Considering most Publishers are expected AMP deprecation from Google in the next 6 months I don't think anyone will build a first party server to deliver this and agree a publisher defined ID may in fact hold little value to external buyers - I don't think we would look to deploy this at this stage unless it was turnkey.

I perceive to get a macro for RTC you'll need buy in from the AMP team (specifically, to put something in amp cache); the method proposed sounds similar to the amp-consent work we done; where the CMP process/service sets some consent string which is then cached and made available as a macro. Having something similar to call user ID services which then cache and are fetchable in RTC via some macro could work but I'm not sure how the AMP team would feel about that considering the amount of work it may entail vs. their wider plans for deprecation of AMP.

You would also need some control to only call the user ID service IFF someone had consented on TCF (GDPR); this could be handled through amp-consent most likely.

I hope my input helps and let us know if some simpler solution is finalised and we may be interested in testing it.

jdelhommeau commented 7 months ago

Hi all, We have established a path toward making the EID available on the AMP page, which doesn't require changes to amp core code. The one thing we would need in order to pass it to prebid server, is for prebid server to have some placeholder in the url, along with a publisher provided MACRO support. To be more specific, I am referring to making changes in this file:

https://github.com/ampproject/amphtml/blob/main/src/service/real-time-config/callout-vendors.js

Then vendors that rely on prebid server would need to update their code as such:

prebidappnexuspsp: { url: 'https://ib.adnxs.com/prebid/amp?tag_id=PLACEMENT_ID&w=ATTR(width)&h=ATTR(height)&ow=ATTR(data-override-width)&oh=ATTR(data-override-height)&ms=ATTR(data-multi-size)&slot=ATTR(data-slot)&targeting=TGT&curl=CANONICAL_URL&timeout=TIMEOUT&adcid=ADCID&purl=HREF&consent_string=CONSENT_STRING&account=ACCOUNT_ID&gdpr_applies=CONSENT_METADATA(gdprApplies)&addtl_consent=CONSENT_METADATA(additionalConsent)&consent_type=CONSENT_METADATA(consentStringType)&pvid=PAGEVIEWID&eid=EIDS', macros: ['PLACEMENT_ID', 'CONSENT_STRING', 'ACCOUNT_ID', 'EIDS'], disableKeyAppend: true, },

So adding the "eid=" placeholder in the url, and adding the EIDS macro in the list of macros that can be substituted by publisher.

Change on prebid server would be to update the "/prebid/amp" url in order to support EID. Regarding format, I would suggest to go with the openRTB spec, turned in base64 url-safe. So for example, for EID below: [{ "source": "utiq.com", "uids": [{"id": "ZJ8clANh49BP0WZaI4qfW4zJrvy3d12V33ISRjjESpYYDa2oRv5pbQG2l1fI18RvJsAZ+RL9tTwsw6+Oz6Sgag==-ndye", "atype": 3}] }] would be turned into

W3sKICAgICJzb3VyY2UiOiAidXRpcS5jb20iLAogICAgInVpZHMiOiBbeyJpZCI6ICJaSjhjbEFOaDQ5QlAwV1phSTRxZlc0ekpydnkzZDEyVjMzSVNSampFU3BZWURhMm9SdjVwYlFHMmwxZkkxOFJ2SnNBWitSTDl0VHdzdzYrT3o2U2dhZz09LW5keWUiLCAiYXR5cGUiOiAzfV0KfV0

Let me know what you think of the above. @adamleslie , I am also quite interested in your point about Google deprecating AMP. I haven't seen anything about this, and several publishers still rely heavily on AMP (>70% of mobile web traffic). The only thing I saw was that as of couple of years ago, Google is no longer prioritizing AMP content over non AMP content in news feed, but still ranking based on page performance. Thanks to the above, some publishers have moved away from AMP, and optimize their mobile web pages performances not to be impacted by Google SEO mechanism, but some publishers haven't been able to optimize page perf to AMP benchmark, and can't yet move away from it. If you have sources regarding Google deprecating AMP soon, could you please share?

adamleslie commented 7 months ago

@adamleslie , I am also quite interested in your point about Google deprecating AMP. I haven't seen anything about this, and several publishers still rely heavily on AMP (>70% of mobile web traffic). The only thing I saw was that as of couple of years ago, Google is no longer prioritizing AMP content over non AMP content in news feed, but still ranking based on page performance. Thanks to the above, some publishers have moved away from AMP, and optimize their mobile web pages performances not to be impacted by Google SEO mechanism, but some publishers haven't been able to optimize page perf to AMP benchmark, and can't yet move away from it. If you have sources regarding Google deprecating AMP soon, could you please share?

You'll need to email me or message me on LinkedIn or something - not officially confirmed but there are a number of indicators.

jdelhommeau commented 7 months ago

@bretg , any feedback from prebid server side on the proposed solution? I understand AMP isn't top priority, but considering the level of effort on prebid server should be fairly low, I was wondering if you would be supportive of it (to summarize, simply need to update the prebid/amp url to add placeholder for eid macro. Then link that placeholder in prebid server to eid logic that should already exist for prebid server non AMP pages I imagine. full details here: https://github.com/prebid/prebid-server/issues/1404#issuecomment-1809998917

bretg commented 7 months ago

This seems ok to me. I have some concern about the length of the overall GET string, but that can be addressed by placing it at the end in the template. Formalizing the requirements a bit:

The PBS /amp endpoint should support an eid parameter.
When a non-empty value to the eid param is received, it should be base-64 decoded and tested to be valid eid JSON. If there's a problem decoding or validating the value, or if there are no eids, it should be ignored, with a warning in the ext.prebid.warning output when in debug/test mode.
Assuming the value is valid JSON, copy it to the ORTB user.eid field.
Note that the stored request may define ext.prebid.data.eidpermissions to limit which bidders get which eids.

Sound right?

jdelhommeau commented 6 months ago

Thank you @bretg . Regarding 1 and 2, fully agree. Regarding #3, I thought it should be copied under user rather than user.data field. I looked at openRTB2.6, which stipulates that eids object should be set directly under user object, not user.data. Can you confirm that point? 4 - makes sense as well yes

If we clarify point 3, and as we agree on the other points, can you confirm if this is something that could be supported? What LOE do you expect, and associated timeline? Sorry, not that familiar (anymore) with prebid process, so maybe not questions you can answer right away.

prebid / prebid-server