[Master Feature] Support IAB Consent Framework using amp-consent

[jasti]

At the moment, consent within amp-consent is binary. Either the user can accept consent for all vendors on the page or reject consent for all vendors on the page. This feature provides the ability for a user to accept or reject consent on a per vendor basis. Further, this could also be enhanced to support consent on a per vendor and per purpose basis. for e.g. a user should be able to give consent at a granular level to Vendor A for the purpose of analytics but reject consent to Vendor A for the purpose of ads. Master issue is #15651

AMP will provide the generic framework for any 3rd party CMP to directly integrate with amp-consent to manage consent workflows.

If you are a CMP, please get in touch with us on this issue.

[jeffnordlund-ghost]

Hey, I work for a CMP and would love to know how you are planning to implement the IAB support and am happy to provide any feedback if needed.

[jasti]

@jeffnordlund-ghost thanks! We've made some progress and @zhouyx is working on documenting the design for community feedback. @zhouyx please post design here when ready.

[fchristant]

I think this feature is much needed in order to meet GDPR requirements which explicitly states that a user must be able to give individual consent on a per processing purpose basis.

I don't really understand how the idea is vendor driven, imho it is purpose driven. A user gives consent to a data processing purpose, not to a specific vendor doing the processing? I suppose it is possible to have two vendors on the same page executing the same processing purpose, whereby a user can give consent to each vendor, but that seems a bit of an edge case to me.

[felixarntz]

Hey there! I have been working on a cookie consent plugin for WordPress that supports AMP via amp-consent, and I'd like for it to support more granular cookie control as well (it already does for non-AMP sites). I agree with @fchristant that I'd expect those controls to work per cookie category / purpose, not vendor. Per-vendor management might be useful as way, in that case I think both should be covered.

As an example, here is what cookie types my cookie consent allows to be specifically allowed/disallowed:

• Functional (automatically-checked, can't be disabled when accepting cookies)
• Preferences
• Analytics
• Marketing

[jasti]

@fchristant thanks for the feedback. We were worried initially about being able to have the capacity to store vendor and purpose information in local storage but further analysis shows we can. So we'll simply rely on the CMP passing AMP with the "daisy bit" as defined by the IAB: https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework/blob/master/v1.1%20Implementation%20Guidelines.md#consent-management-provider-guidelines- which supports both vendor and purpose.

[jeffnordlund-ghost]

Yeah, just leaving that generic would be best and just set up a communication mechanism for that information. Currently the IAB supports consent by purpose, or by vendor, but the vendor is all or nothing (no granular purposes on a vendor). But, rumor has it they are looking into the granular purposes by vendor support (not really sure why they didn't go that way to start).

For us we are supporting consent via the iframe component because our UI is so rich and we would need to basically re-write it to natively support amp-consent. So please keep that in mind and enable this through the message flow for the iframe.

Since the native IAB __cmp interface is able to support cross-frame messaging that should be relatively straight-forward to accomplish.

[jasti]

@jeffnordlund-ghost We weren't planning on supporting the IAB version from an external iframe because the external iframe will then need to be page render blocking which is overall a pretty bad experience of the user from a latency standpoint.

We'd recommend considering adding native support unless you can think of alternatives that ensure good performance. @zhouyx anything else to add?

[jeffnordlund-ghost]

I don't understand why that would be any more blocking then normal consent. I assume you are going to store the consent string in local storage as you do the consent marker now, correct? Also, the __cmp interface is designed to be non-blocking to page render. It will prevent ads from firing until a response is returned, but the page can render as normal. It just queues up the calls made to it until they can be answered. It shouldn't be that difficult to just set up a message interaction to route those calls to a calling iframe.

The problem we have with the standard amp-consent is it's just too simplistic. We deal with huge global brands where they support dozens of languages and most countries. Our platform simplifies that for them, centralizing everything from messaging to keeping their vendor lists up-to-date. It just wouldn't be possible to do that in the typical amp-consent tag, there is too much dynamic stuff happening.

[jasti]

I agree that it's not different from the current amp-consent solution but the external iframe solution was meant as a stop-gap solution for publishers who have simplistic consent gathering UIs. For CMPs, we recommend native support. Note that you will have full control over the UI in this new version. I'd suggest we continue discussing this topic once @zhouyx publishes a design for how CMPs can integrate?

[jeffnordlund-ghost]

Sure. I was not aware you had plans to make the consent UI support more robust. I would very much like to see what is on the roadmap for that so we can plan accordingly.

[zhouyx]

Yes we are planning to support the standard IAB purpose list and vendor list. But currently has no plan to support the granular purposes by vendor because it requires much space to store the consent and from the feedback we gather from some CMP they always collect purposes for all set of vendors. Please let us know if there's requirement for more granular purposes per vendor.

@jeffnordlund-ghost As @jasti pointed out, we don't have a good way to ensure good performance via iframe. Which is very important especially given the UI dialog is blocking user from consuming the page. Could you give us an example of your UI? And we can see what's the best way to support your use case. Thank you!

[jeffnordlund-ghost]

You can go to a number of places to see our consent ui. https://www.cosmopolitan.com/uk/, www.vmware.com, www.crownpeak.com, cbs.com, most nestle sites. Some of them require you come in from an EU country to see the UI though.

Our UI isn't dramatically different from other CMP's. But it is fully customizable (colors, fonts, size, which buttons are shown, what those buttons look like, etc.) and it automatically picks up on things like language settings and current geo so our customers can deploy a common script on every one of their properties and then manage everything from our interface. We also have a process to automatically scan their pages and detect vendors so they don't have to keep updating their vendor lists. So the more of that dynamic nature that can be supported the easier it is for our customers to deploy and manage. But most of that requires scripts to function.

Additionally, we need to be able to do reporting. Currently we report by dropping image pings with the information we need for reporting. But reporting is crucial for compliance with regulations like the GDPR. Our current scripts will log a number of actions, including consent, with enough information to satisfy the EU regulators if necessary.

[zhouyx]

Thank you @jeffnordlund-ghost This is really helpful.

We also have a process to automatically scan their pages and detect vendors so they don't have to keep updating their vendor lists. So the more of that dynamic nature that can be supported the easier it is for our customers to deploy and manage. But most of that requires scripts to function.

This is an interesting requirement that we haven't heard from other CMP before. I wonder how do you scan the page to detect vendors, especially given that some vendors are not directly included in the page but requested by another vendor. What kind of information do you expect from AMP to help you achieve that? Because even with the iframe approach you won't be able to run script on the parent page.

Our current scripts will log a number of actions, including consent, with enough information to satisfy the EU regulators if necessary.

AMP do supports some sort of reporting. For example it send a POST request to the configured endpoint with user's decision. Is that sufficient?

[jeffnordlund-ghost]

In regards to the page scanning there isn't anything the AMP framework needs to do to support that. But what that means is we get a frequently changing list of vendors and that is what we need to support. So some way to pull in the list of vendors/tags (both IAB and non-IAB) would be really helpful. If our customers need to re-add that to their AMP pages as a static list that can become a maintenance nightmare. The IAB is planning to support something that might make this easier, a publisher.json file that allows the list of vendors to be stored in a json file. Support for something like that might be enough to enable this.

It would be nice if you supported some additional events. We frequently get requests from customers for things like opt-in rates. So we need to store both the number of impressions as well as the number of users who accepted or declined. They ask us for details on which vendors are being opt-ed out of, or how often the user elects to opt-out of all vendors (for some scenarios). Our expectation is they will ask for similar reporting for the IAB platform: which purposes are enabled/disabled, how often does a user opt out of everything, how often do they even look at the choices, etc. If you could make the reporting more generic instead of just listing specific actions that would give people the flexibility to report on what is important to their customers.

Also, it would be really nice if you were flexible enough to support GET requests as well as POST's. We have our reporting set up the same way as many advertisers do where we just drop image pings, which are all GET requests obviously.

Other nice to have: Support for a settings file. This would allow our customers to deploy a central amp script and let the settings dictate things such as which geo's need consent, which vendors/tech are on the page, what does the consent dialog look like. Things like that.

[zhouyx]

@jeffnordlund-ghost Thank you for the information. Again this is really helpful!

To support different analytics events, and different request method, we are planning to support a CMP based template. Then you can use or component to collect analytics with much flexibility.

Other nice to have: Support for a settings file. This would allow our customers to deploy a central amp script and let the settings dictate things such as which geo's need consent, which vendors/tech are on the page, what does the consent dialog look like. Things like that.

Yes we have onConsentHref that's supposed to support features like this.

[jeffnordlund-ghost]

I assume onConsentHref is something new since I don't see any docs for it yet.

If you can start to build in support for GET requests that would be great. We don't use servers for 99% of our stuff, just using our CDN to distribute the scripts and settings. This makes our scripts faster and our uptime is basically 100% as opposed to using a server. So supporting that would be great for us.

[janwinkler]

We are a CMP too, how can we contribute?

[consentmanager]

:-)

[jawadst]

@jasti To double down on Jeff's comments and list a few more requirements that we (as a CMP) have for our regular web/in-app SDKs and that are actively used by publishers today: 1) Ability to control the workflow (when to collect consent, what the different banners / popups look like and how the user goes from one to the other, different languages, etc.). As you mentioned, being able to provide a ready-to-go template per CMP would go a long way vs the existing situation where every publisher needs to rebuild the consent UI. 2) Use our own list of vendors and purposes (we rely heavily on the list provided by the IAB but also need to support custom vendors and purposes that are not part of the IAB). This might also require being able to store more information than just the IAB consent string. 3) Be able to condition the loading of tags at the vendor level for a given set of purposes so this can be driven by the consent string and does not necessarily need to be a CMP-specific approach. It can rely on the existing data-block-on-consent attribute, assuming it gets the adequate level of granularity. 4) Reporting of all events to our back-end (consent status, clicks on various elements of the UI, etc.) 5) Support for different ways of giving consent. Depending on the country and the DPA, publishers might want to collect consent on scroll, navigation or click on the banner.

Also happy to help review any documentation/design that would be targeting CMPs.

[zhouyx]

@jawadst Thank you for the information. At this point we are leaning to let CMP implement the logic within an iframe, and then AMP runtime can help to store and pass the consent string to components on the page.

Since you mentioned that you will be supporting vendors that's not necessary of of the IAB list. Could you please provide more information on what exactly are you storing, and passing to vendors. My concerns are

1. AMP has very strict rules on the size of storage that can be used for this purpose. I want to make sure the consent string you are going to store is space efficient.
2. Your consent string can be passed to every vendors that integrates with AMP. I want to make sure that all vendors can understand the response from CMP.

Thanks

[jawadst]

@zhouyx Thanks for your answer!

The iframe option would work great. We are currently using the existing <amp-consent> with an iframe and that works pretty well.

For non-IAB vendors, we usually store the consent information for each of them with a structure that looks like this:

{
  enabled: ['custom_vendor_1', 'custom_vendor_2'],
  disabled: ['custom_vendor_3']
}

While that structure is less efficient than the IAB one, it is usually only used for a small number of vendors (I think our publishers use ~10 custom vendors on average).
The IAB specification allows using a format similar to the consent string for publisher-specific vendors (https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework/blob/master/Consent%20string%20and%20vendor%20list%20formats%20v1.1%20Final.md#publisher-consent-stored) so that might be used as well. To be fully transparent, we like to have string IDs for those vendors but that might be very specific to our CMP and allowing a second consent-string that would be CMP/publisher-specific and can be referenced for blocking/unblocking tags might be enough.

Those custom vendors are important because the IAB mostly covers advertising vendors but not all others (social, analytics and A/B testing are the main ones we are seeing). It's especially important to have a way to cover at least all the different categories that have custom AMP components (so not just advertising but also analytics and social).

As to what we pass to these vendors, there are two cases:
1) If they support receiving consent information through their own API, we provide that (that's the case for DFP or AdX, for instance: https://support.google.com/admanager/answer/7678538). That's similar to what the current AMP consent does I think so there might not be anything special to do except being able to keep the consent at the vendor level. 2) If they don't support receiving consent information at all, we simply block the tag. That's the current behavior of AMP consent if we could somehow have a condition on consent specifically for a custom vendor with the data-block-on-consent

Hope this helps, happy to provide more details or examples if needed. Our documentation is public if that can help better understand the use case: https://developers.didomi.io/privacy-management/consent-management/consent-notice/vendors-and-purposes

[jawadst]

@zhouyx Another point that is usually pretty important to publishers is being able to load non-personalized ads without consent and then -sometimes- reload ads after consent has been given (it usually has an impact on viewability so not every publisher makes that choice). That's usually a configuration that is specific to advertising tags.

DFP, for instance, has support for non-personalized ads (although consent for cookies is still required) and I think that a lot of vendors are going to start offering more options for non-personalized ads eventually.

I think that the current AMP consent somewhat supports that with the combination of a fallback action with "dismiss" status but:
1) I am not sure what the interpretation of "dismiss" is for vendors and if an extra status "unknown" would not be useful. Maybe "dismiss" is enough though but that could be made more explicit in the documentation. 2) I don't think there is an option to reload some tags after consent is given. The scenario "load ads without consent then reload after consent" would be a combination of fallback=0 and "reload after consent".

[zhouyx]

Thank you @jawadst for all the details.

For non-IAB vendors, we usually store the consent information for each of them with a structure that looks like this: { enabled: ['custom_vendor_1', 'custom_vendor_2'], disabled: ['custom_vendor_3'] }

Due to the fact that AMP page can be cached and not served from the origin domain. There's very strict usage on the storage space. I understand that your customized vendors number is small, but still I am worrying about the storage usage. Could you provide me an estimate on the maximum usage of storage in bytes? Currently we are thinking about a maximum of 300 bytes storage upper limit. Not sure if that will work for you.

I am not sure what the interpretation of "dismiss" is for vendors and if an extra status "unknown" would not be useful. Maybe "dismiss" is enough though but that could be made more explicit in the documentation.

It's up to the vendor that how they interpret 'dismiss'. We have vendors that decide to not serve ad on dismiss, but serve npa on 'reject'. The different to AMP is that, with 'dismiss' AMP will never store the decision.

I don't think there is an option to reload some tags after consent is given. The scenario "load ads without consent then reload after consent" would be a combination of fallback=0 and "reload after consent".

You're correct! There's no such reload option for AMP now. Publishers can choose to serve a npa while asking for consent. And when someone visit the site next time a personalized ad will be served. But that's depends on the publishers' configuration.

[zhouyx]

I wonder if it is even possible for CMP to store some related information on server end?

[consentmanager]

@zhouyx The IAB consent string is usually less than 60 or 70 bytes. Anyhow, with the next versions of the consent string there are some additions that can extend the consent string up to 700-800 bytes.

[zhouyx]

@consentmanager Thank you for the heads-up. I didn't know the next version of the consent string expands its size so much : ( Is there a doc that I can refer to? Thank you!

[consentmanager]

https://docs.google.com/document/d/1dYFuZQfw2hqdwv-1fI6B3QyzcDTVvCd0sqEdq2z_9cM/edit

[consentmanager]

@zhouyx storing info on server end: probably possible (e.g. give the user a cookie with an id and save the consent string on servers) but not yet supported by the iab framework

[jawadst]

@zhouyx Storing consent on the server-side is possible independently from the IAB framework but most CMPs would rather not have to store anything server-side for availability and speed questions. Especially in AMP, it is extremely important to guarantee that everything is stored directly on the device to avoid extra HTTP requests and increased latency for the user (especially when the CMP views are likely to be blocking for the user).

[jeffnordlund-ghost]

the biggest problem with storing info server-side as a CMP is identifying the user the information belongs to. without some kind of specific user identifier it isn't possible to map a server request to the consent information and your standard browser request doesn't contain enough information to accurately identify a specific user from random sites on the internet. that, and performance, are the main reasons we use either cookies or local storage for this information.

[zhouyx]

Especially in AMP, it is extremely important to guarantee that everything is stored directly on the device to avoid extra HTTP requests and increased latency for the user (especially when the CMP views are likely to be blocking for the user).

@jawadst This is a great point. I know some would still want to make a request for additional information even with stored value. Is that true for your case? Can I assume that once there's previous stored consent string, AMP runtime can simply use that without sending request to CMP?

without some kind of specific user identifier it isn't possible to map a server request to the consent information

@jeffnordlund-ghost You're correct it's impossible to identify user on server side right now. But it can be supported if we feel server side storage is the way to go. What concerns me more is that do you feel it's OK for you to store an user's identifier as a CMP in general?

[mobiluck]

@zhouyx the amp runtime should/must load the cmp code every time on every page, even if the value is already stored. the code will be providing the functionality/API for vendors, hence its necessary to load it even in cases where consent is already given

[jeffnordlund-ghost]

No, that's kind of the point of not doing things server side. If we do that then we are forced to store some kind of user identifier and we have no desire to do that. All we want to do is provide users a mechanism to exercise their choices and then communicate those choices to the publisher.

[hhhjort]

Good to see AMP moving in the direction of the IAB consent framework. One use case I haven't seen explicitly mentioned is an RTC call to a vendor that will then proxy additional calls to additional vendors. (Example: prebid) That will require passing the IAB consent string (or the same information in some other format) in the RTC call so the vendor can properly support the GDPR requirements in whatever it does with the request.

[jeffnordlund-ghost]

I was under the impression that forwarding the consent string through to downstream calls is the responsibility of the vendor. So an auction vendor would be responsible for sending the string to the auction winner. I don't think there is anything special that would be required in AMP to support that.

[hhhjort]

Well, the vendor needs the consent string to be able to forward it. If the vendor is being called via AMP RTC, then AMP needs to be able to provide the consent string in the RTC call for the vendor to have it.

[jasti]

@keithwrightbos are there plans for DoubleClick to pass the consent string to RTC vendors?

[jdelhommeau]

Hi @jasti ,

Is there any update on supporting consent string passed in RTC config? Seems to be a blocker for prebid support in AMP.

[zhouyx]

Hello @jdelhommeau I talked to @jasti today on this topic. We can definitely add support to pass consent string along with the RTC request.

Trying to understand your requirement better? Are you working with the prebid server, that can consume the consent string along with the RTC request.

[hhhjort]

I imagine the best way to support this is to have a macro that will be populated with the IAB consent string if it is available. Ideally also a GDPR flag to signify if the viewer is in a location where the GDPR applies. If these two can be placed in the query string, it will not be difficult to modify prebid server to read and make use of them.

[Facens]

@jasti what's the status of this task? Was there any progress?

[zhouyx]

The support is already in prod. Please refer to the doc here for integration guidelines. Closing the issue.

[mgriego]

This is still in experimental status?

[zhouyx]

No it's not. Good point, I'll update the doc. Thank you. #23049

[jeteve]

I don't see why this is closed? @zhangsu the document you're referring to simply shows how to delegate the ternary 'google only personalised ads' consent to a publisher's own interface (via the iFrame). It doesn't help implementing AIB's TCF's granularity of Purpose and Vendors choices, nor the page level API compatible vendors would look for.

Am I wrong?

[zhouyx]

Hello @jeteve Thank you for your question.

AMP only provides a tool for publisher along with consent management platform and vendors to collect and maintain their user consent. As you mentioned, the support is provided via iframe. AMP helps store and pass consent information from the iframe. But it's up to the publisher or the CMP to provide IAB's granular support within the iframe.

We do have API for vendors to retrieve the stored consent information. However because AMP decides to not enforce any format on the stored consent information. The API we provide couldn't handle more granular information, and that's by design. It's up to the CMP and vendor to agree on a standard, which could be IAB standard or any other format.

The AMP's support works when the CMP decides to collect granular consent information from user. AMP will pass and stored the information to vendor, where they can interpret the info. Does this make sense.

Thanks.

[adamread]

I also don't understand how this FR has been addressed. According to the docs, it's still not possible to enable ads or analytics on a granular basis. Both can only use data-block-on-consent and it's an all or nothing toggle. Collecting granular consent data is only half the requirement and it's not only useful if you can apply the user's decisions to the current page. From interactions with potential AMP developers, a way to set and retrieve key-values with amp-consent and then use them with data-block-on-consent to selectively allow amp-analytics components is the missing part of this FR.

[martinschierle]

+1 we still hear many requests for a better integration here.

[zhouyx]

@adamread @martinschierle, you're right data-block-on-consent only supports the a global decision. Granular decision is passed via the consent string, and then AMP assumes that it's the 3rd party vendor's responsibility to interpret the consent string and behave accordingly.

Can I ask if you're working with a CMP or you're customizing the page behavior as a publisher. Thank you

[zhouyx]

Saw #26735, let's move the conversation there. Thank you