I2I: Support CMP Integration with <amp-consent>

zhouyx commented 6 years ago

To #15653

Overview

The following sections will try to answer these questions:

What to include in the prompt dialog and how to allow CMP to customize it?
What is the blocking behavior with CMP?
What the data flow looks like between different parties?
What the CMP needs to do to integrate with AMP?
What does a publisher need to do to include a CMP solution on their AMP page?

Background

<amp-consent> provides a solution for publishers to collect user consent and block or unblock any AMP component in an AMP page. Currently, this solution only allows the publisher to collect a boolean consent for all vendors in the consent UI.

To support per purpose and per vendor based consents and allow for third party integration from Consent Management Platforms (CMP) with publisher AMP pages, AMP will enhance <amp-consent>. It will allow for richer interaction between the publisher page, vendors, CMP and ad extensions on AMP pages that need to know about the consent status. AMP must make such an API for CMPs because AMP doesn’t allow custom JavaScript to run on the page and therefore a CMP must integrate with the AMP framework to integrate with a publishers page.

Data Flow Desing

The following section explains each step in terms of data structure and API.

Read data from localStorage (Step 1)

Data includes the stored consent string and CMP stored information.

Pass information to CMP endpoint (Step 2)

The following data with be passed through a POST request

   {
     'consentInstanceId': string,
     'consentStr': string,
     'pubConfig': object,
     'cmpStoredData': string,
   }

CMP endpoint response (Step 3)

Endpoint returns a JSON object

   {
     'promptIfUnknown': boolean,
     'sharedData': object,
     'forcePrompt': boolean, //optional
   }

Initiate the UI iframe (Step 4)

Data will be stringified and passed via the iframe name attribute

   JSON.stringify({
     'publicConfig': object,
     'cmpStoredData': string,
     'consentStr': string,
   })

Iframe returns user decision (Step 5)

UI iframe will pass the consent information back to <amp-consent> via postMessage

window.parent.postMessage({
 type: 'consent-response',
 cmpStoredData: '',
 consentStr: '',
 action: 'accept/reject/dismiss'
}, '*');

consentStr is required to pass the value to vendors.

action is optional. When specified, it instructs components that don’t understand consent string to load or not based on the publisher’s configurations.

cmpStoredData value will be stored per the CMP request

Store consent information (Step 6)

Store the consentStr and the cmpStoredData to localStorage

Pass consent information (Step 7)

AMP components on the AMP page (vendors in the format <amp-ad type=${vendorName})> can access consent information with function call

getConsentPolicySharedData(ampdoc, policyId)

Where the function returns an object that merge the value of consentStr and sharedData

   {
     'consentStr': string,
     'sharedData': object
   }

Blocking Behaviors

Default Blocking Behaviors

When a CMP is present on the page, AMP runtime behavior will default to:

Block all <amp-ad> components on the page from rendering
Send request to the CMP endpoint to ask whether consent is required
- If no -> Stop here
Check if there’s a response stored in the AMP runtime, or the CMP endpoint returned with a response
- If yes -> Unblock <amp-ad> and pass the response to them
- If no -> Prompt the consent collect dialog in a foreign iframe
Get the response from the prompted iframe Store the response as instructed.
Unblock all <amp-ad> components (unless the publisher specifies not to by adding the data-block-on-consent attribute) and passesthe response to them.

Special Cases

Publishers can override the default behavior and customize the blocking behaviors by tagging the components manually.

If a publisher wants to avoid ad blocking with a CMP,: Use <amp-ad data-block-on-consent=”_never”>
If the publisher wishes to block a component other than <amp-ad>: Use the existing data-block-on-consent attribute. However, the value of the returned action (accept/reject/dismiss), depends on the agreement between the publisher and the CMP.

Prompt Window

External Iframe

AMP consent will load and render the CMP’s prompt UI in a cross domain iframe, where the CMP will have full control. However, AMP runtime will control the life cycle of the iframe, dictating when to load or unload it.

The iframe will be loaded if a user consent is required but missing, or if a user wants to manage their consent at any point.
The iframe will be unloaded once a user decision has been made and the prompt UI needs to be hidden.

Default placeholder

AMP will apply a placeholder before the CMP iframe loads to improve user experience. The placeholder will not block the user from interacting with the rest of the page. UX design will be added once ready.

Iframe Size and Placement

AMP will make the prompt UI interstitial by default. CMP and publishers will NOT be able to customize the size and the placement of the iframe. AMP will create a modal layer and place the iframe with default aspect width and height ratio in the center.

Consent String

Consent string contains the user decision and will be stored and passed to vendors. Vendors will interpret the consent string and respond accordingly.

Format

Because different CMPs want to store different consent strings for customized vendors in addition to the IAB consent string, AMP will not enforce format on the consent string, or attempt to interpret the consent string before passing the information to vendors. AMP would require CMP to indicate their string as IAB so that vendors can understand. Please note in MVP, not API will be provided to vendors, and vendors need to interpret consent string themselves.

Storage Limit

When the cached AMP page is served from the google.com. All information will be stored under the google.com domain. Because of it, AMP consent has to be very cautious of the storage usage.

In order to be space efficient, AMP restricts the size of total stored value to an upper limit of 300 bytes. That includes the consent string and the CMP private stored string, if one exists. CMPs should work with their partner vendors to compress their consent string.

In the case that a CMP asks AMP to store information that exceeds the size limit, AMP will:

Erase the current stored value
Log an error
Unblock <amp-ad> components and make the consent string information globally accessible to all AMP components the consent string information but w/o storing the value for next visit.

Access

The consent string is made available to all AMP components on the page and to the CMP.

To AMP components and vendors

AMP components can call where the policyId can be interpreted from the data attribute.
```
getConsentPolicySharedData(ampdoc, policyId)
```
3P ad that integrate with AMP can find the string at (similar to supported sharedData)
```
window.context.consentString
```
To CMP
- CMP will have access to the consent string. Per their request a CMP will also have access to the CMP stored data. Note that passing CMP stored data is not in scope with the initial implementation.
- The stored value will be divided into two parts.
- Standard Consent string
- Private CMP stored string
  
  Where consent string can be shared to all components, stored string will only be shared with the CMP.

CMP Integration

A CMP named foo_bar will integrate with AMP in the following way.

Check in configuration

Add the configuration as part of the <amp-consent> codebase. This will allow AMP to recognize the CMP, perform steps to check if consent is necessary, handle storage and UI.

 'foo_bar': {
   ‘storageKey’: ‘cmp_foo_bar’,
   'checkConsentHref': 'https://foo-bar.com/amp/endpoint',
   'promptUISrc': 'https://foo-bar.com/cmp.html',
  }

storageKey instructs AMP runtime to store the consent information under this unique key name. Note: AMP will append prefix to the key. If not specified, AMP will assume CMP does not require consent string to be stored client side.

Unlike the current DIY approach with <amp-consent>, CMP will be able to specify the storage key value instead of publishers. This is easier for CMP to bump up versions because they have control of the storageKey.

checkConsentHref is a field AMP consent already supports. It allows the AMP consent component to ask the CMP if consent is required for a user. This field is required, as it is the key to instruct AMP runtime to ask for consent and block other components on the page. For more details, please refer to the section here

Note:the existing promptIfUnknowForGeoGroup for CMP integration won’t be provided as AMP expects CMP to host their own endpoint instead of depending on publishers configuration on the <amp-geo> component.

src is the source for the foreign iframe. It is also a required field. Once AMP consent decides to prompt, it will create a foreign iframe with the src given.

Host endpoint

AMP consent will rely on the host endpoint to determine if the prompt is required with the checkConsentHref request.

Consent Dialog

The CMP will need to modify their prompt logic to function in a cross domain iframe. And communicate with the parent host AMP page through the proposed communication APIs.

Publisher’s Configuration

To work with a CMP, publisher’s configuration will need to stay minimal.

   <amp-consent id="ABC" type="foo-bar" layout="nodisplay">
     <script type="application/json">{
       “cmp”: {
         “token”: xxx
       }
     }</script>
   </amp-consent>

Type Attribute

A type attribute with a valid CMP name value instructs the <amp-consent> component to load the CMP config.

CMP Object

All configuration within the cmp object will be passed to the CMP, both with checkConsentHref and iframe (added to the name attribute).

PostPromptUI

CMP’s support to postPromptUI has not been discussed yet. Right now publishers can still put their own postPromptUI element on the page and reference the element.

   <amp-consent id="ABC" type="foo-bar" layout="nodisplay">
     <script type="application/json">{
       "postPromptUI": "postPromptUI"
     }</script>
     <div id="postPromptUI">
       Post Prompt UI
       <button on="tap:ABC.prompt" role="button">Manage</button>
     </div>
   </amp-consent>

Other topics

Below is a list of topics we have talked about but not covered in the current design. We may look into supporting it in the future. Allow publishers to still use the <amp-geo> component to reduce one roundtrip of latency Provide support for CMP to store consent server side but not client side

jasti commented 6 years ago

Hi @jeffnordlund-ghost @jawadst @janwinkler, @waschnick and @consentmanager. Since you've previously expressed interest in amp-consent's ability to integrate with CMPs, we'd love to have you participate in our design review this Wed @ 1PM PT where we'll go over this component and it's design. Details in #17357. Please feel free to leave feedback on this issue in the mean time.

jawadst commented 6 years ago

@jasti Thanks for pinging me, I'll definitely join!

zhouyx commented 6 years ago

Few thing I recall from the design review

The concern about blocking analytics vendors
- There's no good way to support that with the current design
- Would require analytics offline mode to store events before user giving consent
- To follow up with some analytics vendors and see if they wish to be blocked.
The concern about vendor list changes
- To introduce dirty bit to force ask user consent the next time they visit the page
Decided to support the integration with <amp-geo> with CMP integration as it provides faster experience for users that doesn't require consent.
storage limit concern
- CMP use the client side storage as their primary source of storage.
- We'll start with 300 bytes limit and work on the LRU/LRR to expand the storage limit.

lannka commented 6 years ago

Hi @jeffnordlund-ghost @jawadst @janwinkler, @Waschnick and @consentmanager

we are looking for a CMP to integrate with the amp-consent API. Please let me know if you are interested and we can schedule a VC to go over integration details.

consentmanager commented 6 years ago

yes please :-)

oliverfernandez commented 6 years ago

Hi @lannka

We also have an IAB certified CMP, deployed in several sites, what we'd like to integrate with AMP. I'd appreciate a lot if you could keep me in the loop regarding this

Thanks!

jawadst commented 6 years ago

@lannka Happy to help! We already have a first setup running with the existing amp-consent at Didomi so happy to see how the new version would perform with that.

jeffnordlund-ghost commented 6 years ago

Sure, we would like to explore a better integration with the amp-consent api.

lannka commented 6 years ago

Thanks all. @consentmanager @oliverfernandez @jawadst @jeffnordlund-ghost If you have a launched CMP product, could you share a link?

Also what's your timezone so we can schedule a VC.

consentmanager commented 6 years ago

Its http://www.consentmanager.net Timezone is CET

oliverfernandez commented 6 years ago

You can find and example of our CMP working here (mobile only): https://nospensees.fr/

My timezone is CET as well

zhouyx commented 6 years ago

@consentmanager @oliverfernandez @jawadst @jeffnordlund-ghost Thank you all for the interest. We scheduled a meeting on Thursday. Time: 2018-09-13 15:30 UTC (add to Google Calendar) Location: Video conference via Google Hangouts

To #17983 for meeting details. Please let us know if you plan to attend or suggest a different time. Thanks.

oliverfernandez commented 6 years ago

@zhouyx works for me

lannka commented 6 years ago

@consentmanager @jawadst @jeffnordlund-ghost in case you didn't see. are you able to join us tmr?

consentmanager commented 6 years ago

Yes, will be there

jawadst commented 6 years ago

@lannka That works for me as well

jeffnordlund-ghost commented 6 years ago

Sorry, I’ve been out of the office for a bit and just catching up.

I might be able to join (if it hasn’t happened yet). My timezone is MST (in the US). I thought I saw a mail with the date/time earlier but I’m not seeing it now. When are you looking to schedule this?

Thanks!

Jeff

From: Hongfei Ding notifications@github.com Reply-To: ampproject/amphtml reply@reply.github.com Date: Wednesday, September 12, 2018 at 7:59 PM To: ampproject/amphtml amphtml@noreply.github.com Cc: Jeff Nordlund jeff.nordlund@crownpeak.com, Mention mention@noreply.github.com Subject: Re: [ampproject/amphtml] I2I: Support CMP Integration with (#17742)

@consentmanagerhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fconsentmanager&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C238b63efda5f4c0ce4ff08d6191c8797%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636724007657104766&sdata=FhEa0FQB1GAo274LLa7aoFrA4JYEfjbKQp2CiUA5e0s%3D&reserved=0 @jawadsthttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjawadst&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C238b63efda5f4c0ce4ff08d6191c8797%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636724007657104766&sdata=93UCkKp%2BfaLczXwEepx2z1pnCa5xtPOBdgs8Er3oR04%3D&reserved=0 @jeffnordlund-ghosthttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjeffnordlund-ghost&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C238b63efda5f4c0ce4ff08d6191c8797%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636724007657114774&sdata=CotWh9hfJRYk6FyXcih5dn7LPY9EYyI5auTbQoSGS3c%3D&reserved=0 in case you didn't see. are you able to join us tmr?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fampproject%2Famphtml%2Fissues%2F17742%23issuecomment-420857254&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C238b63efda5f4c0ce4ff08d6191c8797%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636724007657124782&sdata=utwcKcqXU5G3AceOkBp6OjjtPQwttlG%2BOSGPONV3g6c%3D&reserved=0, or mute the threadhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIP0R6Xu9gS6KxkFQthsnMuxDlUDMQbKks5uabv7gaJpZM4WOiEN&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C238b63efda5f4c0ce4ff08d6191c8797%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636724007657124782&sdata=6YtJsqegpX%2B674jw1irTeMZN9Gll8M%2F9pSF57yzDJ3Y%3D&reserved=0.

Empower - September 26-27 2018 In New York

jawadst commented 6 years ago

@zhouyx There is an IAB Techlab call at that time that a lot of us are likely to attend. Can we do Friday instead?

zhouyx commented 6 years ago

Thank you @jawadst for the heads up. Would the same time on Friday, 2018-09-21 15:30 UTC work for everyone? If so I'll go ahead and change the invitation.

Please let us know. Thanks

jawadst commented 6 years ago

Works for me

consentmanager commented 6 years ago

works

consentmanager commented 6 years ago

re: "CMP and publishers will NOT be able to customize the size and the placement of the iframe. AMP will create a modal layer and place the iframe with default aspect width and height ratio in the center." --> why the downgrade? currently we are able to resize the iframe, which is necessary when many information is shown ...

zhouyx commented 6 years ago

Cool! Thank you everyone. The invitation was moved to Friday.

Time: 2018-09-21 15:30 UTC (add to Google Calendar) Location: Video conference via Google Hangouts

zhouyx commented 6 years ago

@oliverfernandez @jeffnordlund-ghost in case you didn't see, we scheduled a meeting Friday to talk discuss the integration work. Thanks

Time: 2018-09-21 15:30 UTC (add to Google Calendar) Location: Video conference via Google Hangouts

jawadst commented 6 years ago

@zhouyx Apologies as I missed the call. Let me know if you need anything from us.

jeffnordlund-ghost commented 6 years ago

Sorry, looks like I missed the call. Please let me know if I can provide any information or answer any questions from how we are using the IAB.

From: Yuxuan Zhou notifications@github.com Reply-To: ampproject/amphtml reply@reply.github.com Date: Friday, September 21, 2018 at 12:36 AM To: ampproject/amphtml amphtml@noreply.github.com Cc: Jeff Nordlund jeff.nordlund@crownpeak.com, Mention mention@noreply.github.com Subject: Re: [ampproject/amphtml] I2I: Support CMP Integration with (#17742)

@oliverfernandezhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Foliverfernandez&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C8970fa8fabce4ae638f408d61f8c9e04%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636731086139597876&sdata=YXnWFqhtmkaBq0SEG3RU5bdR1la5T5rsB883a8sAd1Q%3D&reserved=0 @jeffnordlund-ghosthttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjeffnordlund-ghost&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C8970fa8fabce4ae638f408d61f8c9e04%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636731086139607884&sdata=jwj6FVj0vzqYRActs6AV1pWYA2WD5KABdjOtfEyDUIk%3D&reserved=0 in case you didn't see, we scheduled a meeting Friday to talk discuss the integration work. Thanks

Time: 2018-09-21 15:30 UTChttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.timeanddate.com%2Fworldclock%2Fmeeting.html%3Fyear%3D2018%26month%3D09%26day%3D21%26iv%3D0&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C8970fa8fabce4ae638f408d61f8c9e04%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636731086139617897&sdata=507HOoK4QGhnTsQgLK3LiCJ7SKz1MqTuqTDiIWw%2BlxI%3D&reserved=0 (add to Google Calendarhttps://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.google.com%2Fcalendar%2Fevent%3Faction%3DTEMPLATE%26text%3Damp-consent%2520CMP%2520integration%26dates%3D20180921T153000Z%2F20180921T160000Z%26details%3Dhttps%253A%252F%252Fhangouts.google.com%252Fhangouts%252F_%252Fgoogle.com%252Ftgg-rpqh-ttp&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C8970fa8fabce4ae638f408d61f8c9e04%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636731086139617897&sdata=YDw80zn22PhXsYP6r52vPR9o%2Bnrne%2B5z50kNJ2771NI%3D&reserved=0) Location: Video conference via Google Hangoutshttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhangouts.google.com%2Fhangouts%2F_%2Fgoogle.com%2Ftgg-rpqh-ttp&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C8970fa8fabce4ae638f408d61f8c9e04%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636731086139627905&sdata=xSBddpw2bkvINvZ6%2Fw2SnywQQ0Unboje%2FYJWzAqYGSo%3D&reserved=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fampproject%2Famphtml%2Fissues%2F17742%23issuecomment-423429283&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C8970fa8fabce4ae638f408d61f8c9e04%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636731086139637918&sdata=5H3omEoT29SD4Q36j3itE1I7kB%2FKIvQVeDwwKkTqgDo%3D&reserved=0, or mute the threadhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIP0R70RzeU58pKlE3OrVNx13j7jvwExks5udIkDgaJpZM4WOiEN&data=02%7C01%7Cjeff.nordlund%40crownpeak.com%7C8970fa8fabce4ae638f408d61f8c9e04%7Ca5a244c1af69471fb4c7ae6e94586df5%7C0%7C0%7C636731086139647926&sdata=kcuHg%2FN32tk584LksA3Go73M7mZmmF%2BjNDfAw1fZX1M%3D&reserved=0.

Empower - September 26-27 2018 In New York

zhouyx commented 6 years ago

Thank you all for attending the meetings.

Here's the notes from the meetings we have

9/13 Understand the requirement

Consent string standard
- Supposedly all ad vendors will eventually support the IAB standard. Analytics vendors however don’t have incentive to do so.
- Vendor configuration is done on CMP server side
- AMP support the IAB use case. And also the global boolean consent decision (accept/reject) so publishers can work with CMP to block vendors that doesn't support IAB.
UI standard:
- All CMP provides blocking and non blocking version of the UI. (Blocking UI is an interstitial that takes the full screen. Non blocking UI stick to the bottom of the page)
- Publishers have the final word on UI design. And the configuration is from client side.
- Don’t have good data on which one is preferred by publishers today.
- AMP will by default support UI, that is non blocking before iframe load.

zhouyx commented 6 years ago

9/21 Discuss technical details

UI
- AMP will provide iframe ready API that show the iframe and hide placeholder.
- Initial size is always 30vh, and can’t be changed Is this OK with everyone?? (Feedback from the meeting: 30 vh may not be enough because clients can defined the text. Accept button is usually below the text. Also text on the button can be very long with certain languages. But 30vh is fine to start with. Also there's not much extra work to support dynamic sizing from CMP side).
- AMP offers the requestFullScreen API, that expand the iframe to fullsize for "learn more session".
- When postPromptUI is clicked to trigger the UI again. AMP will toggle the iframe to the initial 30vh size again because not all publishers allows custom settings and it doesn't make sense to jump to fullscreen.
Remote Config

Data from AMP: Is this sufficient to make the decision??

consentInstanceId. Which is defined by CMP and checked in to AMP code base
Cmp configuration that is defined by publisher. Note AMP won’t enforce format to the configuration.

Response from CMP

If consent is required or not.
To force clean the stored consent the NEXT time the user visit the page
sharedData that can be accessed by vendors, but won’t be stored

PromptUI Iframe

To render the iframe all data will be written to the name attribute

Data from AMP Is this sufficient to render the UI??

Stored consent string
Cmp configuration that is defined by publisher. Same to Remote Config

APIs

UI APIs
Consent-response API (pass back a consent string that will be stored and an optional global boolean decision).

Update on timeline from AMP

We're in the middle of the UI support
Configuration support is already there
Need to add API support afterwards

We're expecting to have CMP develop in parallel with us and have an end to end integration flow soon. Please let us if you have any question or concerns on the design. And would love to start email thread or github thread to track the development process. Thank you!

oliverfernandez commented 6 years ago

@zhouyx I'm really sorry I missed the last meeting (I was on holidays)

The technical details discussion sounds really great for me. Some comments

30vh for the initial height seems really small in my opinion, specially if you want to show the user a clear initial text and obtain his consent. The success of a CMP is measured by the ratio of consent granted over all consent requests, so the initial screen is critical. I'd suggest at least 50vh, but as you said it's something that can be changed easily in the future
Does the remote config substitutes the current checkConsentHref? Does it work in the same way?
Do you plan to substitute the current promptUI by this, or both will coexist?
Will it be needed to specify a list of CMP implementations somewhere in the AMP runtime (somehow like RTC of Amp4Ads), and then reference which one you want to use from AMP-CONSENT configuration?
Is going to be some kind of timeout to present the CMP?
Will both consent string and global boolean consent be stored by AMP? Is it up to the vendor AMP implementation (e.g. Doubleclick, Comscore, ...) which want to use?

I'd love to follow up your implementation progress, and do a first implementation from our side if you need validation.

Thanks!

ampprojectbot commented 5 years ago

This issue seems to be in Pending Triage for awhile. @zhouyx Please triage this to an appropriate milestone.

zhouyx commented 5 years ago

Hi all, Happy New Year.

We're close to the code complete of the feature on AMP side. Would like to look for early adopters who would integrate with us. Please let us know if you're interested.

Implementation Progress from AMP

[x] Stored consent string
[x] Cmp configuration integration
[x] Consent-response API to pass consent string back to AMP runtime
[x] Render iframe provided by cmp to collect consent.
[x] Pass stored consent string and client config to iframe
[x] Full screen API
[x] Enable ad vendors to retrieve consent string passed from CMP

One last piece of work we are still working on is to block all amp-ad on the page by default. The work can be tracked at #20253. We've decided to introduce a meta tag and require that from publishers in order to block all ads by default. Please let us know if there's any feedback. Thank you.

to @consentmanager @oliverfernandez @jawadst @jeffnordlund-ghost

jawadst commented 5 years ago

@zhouyx Happy to start the integration with the Didomi CMP whenever you are ready. Do you plan to release a test version before the final production release?

zhouyx commented 5 years ago

@jawadst great! We'll keep you updated once the last piece of work is done.

We are protect the feature with an amp-consent-v2 experimental flag. And plan to flip that flag once we believe it's ready.

zhouyx commented 5 years ago

@consentmanager @oliverfernandez @jawadst @jeffnordlund-ghost The AMP side work has completed. We expect all feature to be live in canary next Tuesday.

I've created a draft doc for you to start with. Please let us know if there's any questions. I'm happy to sync offline if necessary.

zhouyx commented 5 years ago

All features should be live in with developer opt-in now.

I've created an example page to demonstrate. It's a simplified cmp service, with

example remote endpoint logic
consent prompt ui iframe https://raw.githubusercontent.com/ampproject/amphtml/master/test/manual/diy-consent.html
and configuration in AMP consent https://github.com/ampproject/amphtml/blob/master/extensions/amp-consent/0.1/cmps.js#L30-L36

Please let me know what I can help with the integration. Thank you.

@consentmanager @oliverfernandez @jawadst @jeffnordlund-ghost

jawadst commented 5 years ago

@zhouyx Thanks a lot for all your hard work, this looks great! I'll try it out next week and send any feedback I have.

christian-felix commented 5 years ago

Great, thank you @zhouyx. We are also very impatient to use this new solution and adapt it soon.

zhouyx commented 5 years ago

Great! @jawadst @christian-felix please let me know if there's any feedback or question.

lannka commented 5 years ago

hi @jawadst @christian-felix did you get chance to look at the integration yet? we're looking for an announcement in the coming AMP Conf in April. It will be good if we could show case a couple of CMP integrations.

Let us know if you have any questions.

christian-felix commented 5 years ago

Hi @lannka, we are currently evaluating the idea and hopefuly absorbing it soon. In case of successfully integration I will provide you with the appropriate link for the coming AMP Conf.

jasti commented 5 years ago

Hi @jawadst @consentmanager @oliverfernandez @jeffnordlund-ghost we are looking to exist the beta relatively soon and would love your feedback before we do so. Do you have an ETA for when you might be ready with your AMP CMP integrations?

jawadst commented 5 years ago

@jasti @lannka @zhouyx We are finishing up our tests and should be able to provide feedback some time next week

jawadst commented 5 years ago

@zhouyx @jasti We have a few questions for you following our initial tests:

1) It seems that the function {amp-consent-id}.prompt() can be used to re-open the prompt UI, is that correct? Is that documented somewhere yet? (I am assuming it is new with your v2 update)

2) For the info field that gets returned by a CMP in the consent-response postMessage, we are not completely sure of what you expect in there. I understand it is a general field for a CMP to store data and:

Is it expected to be an IAB consent string or is it a free-form field for a CMP to use?
Is it the value that vendors get when they ask for the consent string?
Is it the value that gets passed back in the consentString field in the iframe name? If yes, you might want to rename the fields to have the same name (consentString or iabConsentString are good).

3) What is your recommended solution for testing the prompt UI iframe and checkConsentHref locally before adding our CMP to https://github.com/ampproject/amphtml/blob/master/extensions/amp-consent/0.1/cmps.js? (for local development and testing basically). We are able to replace your URLs when using your test file with a browser extension but is there a way to locally define a CMP or override the settings for your _ping_ CMP?

4) Regarding the co-existence of the consent string and the consent state, is our understanding correct that:

the consent state is used for blocking behaviors as described on https://www.ampproject.org/docs/reference/components/amp-consent#blocking-behaviors
the consent string will be passed to IAB vendors as needed but is not leveraged by AMP itself for on-page blocking

5) Do you have any documentation on how vendors are expected to query/receive the IAB consent string? Are you in discussion with any vendor on that topic?

6) One open question that I know we discussed on our calls and you decided not to include in this first update to AMP: do you have plans to add more granularity to the consent state and the blocking behaviors on the page? As it is defined today as a binary state (agree/disagree), it is impossible to really be compliant with GDPR and the latest information given by DPAs like the CNIL at the end of 2018, as they require granular choices (by purpose and by vendor) to be exposed to the user. A global yes/no choice is not enough from that perspective. While we might be able to achieve that for IAB vendors with your current implementation, that will not be doable for non-IAB vendors like analytics vendors.

Thanks a lot for all your hard work on this subject: we get daily requests from our publishers for AMP support so be assured that this will be used as soon as it is released! I think we'll have our first AMP CMP released some time in March so let me know if you want to coordinate something for your April conference, I am happy to provide some material as needed.

cc @MaximePlancke (who is also working on our AMP integration at Didomi)

zhouyx commented 5 years ago

@jawadst Thank you for the feedback.

1.

It seems that the function {amp-consent-id}.prompt() can be used to re-open the prompt UI, is that correct? Is that documented somewhere yet? (I am assuming it is new with your v2 update)

This is something we support in v1 : ) Document is here. It is designed for users to manage their consent after initial interaction with the prompt.

2.

Is it expected to be an IAB consent string or is it a free-form field for a CMP to use?

AMP expect it to be a free-form field. Whether to follow the IAB consent string format or not is up to the CMP.

Is it the value that vendors get when they ask for the consent string?

Exactly

Is it the value that gets passed back in the consentString field in the iframe name? If yes, you might want to rename the fields to have the same name (consentString or iabConsentString are good).Is it expected to be an IAB consent string or is it a free-form field for a CMP to use?

You're right! There are other client information that get passed to the iframe in the iframe name. Which includes clientConfig, consentState and consentString. And we pass the stringified object. Which field do you want to rename?

What is your recommended solution for testing the prompt UI iframe and checkConsentHref locally before adding our CMP to https://github.com/ampproject/amphtml/blob/master/extensions/amp-consent/0.1/cmps.js? (for local development and testing basically). We are able to replace your URLs when using your test file with a browser extension but is there a way to locally define a CMP or override the settings for your _ping_ CMP?

It is not documented, but you can use inline config to override any CMP provided vendor config. In this case I assume you can do

<amp-consent type='_ping_'>
  <script type="application/json">
    {
      'consentInstanceId': yourid,
      'checkConsentHref': ....,
      'promptUISrc': xxxx,
    }
  </script>
</amp-consent>

let me know if this helps with your local testing.

Regarding the co-existence of the consent string and the consent state, is our understanding correct that:
- the consent state is used for blocking behaviors as described on https://www.ampproject.org/docs/reference/components/amp-consent#blocking-behaviors
- the consent string will be passed to IAB vendors as needed but is not leveraged by AMP itself for on-page blocking

Exactly

Do you have any documentation on how vendors are expected to query/receive the IAB consent string? Are you in discussion with any vendor on that topic?

Thank you for asking this. Currently we haven't documented on how vendors are expected to query/receive the IAB consent string yet. But it's definitely we will be adding soon. We have not discussed the consent string topic with any vendor yet, given AMP is purely only going to pass consent string between CMPs and vendors.

One open question that I know we discussed on our calls and you decided not to include in this first update to AMP: do you have plans to add more granularity to the consent state and the blocking behaviors on the page? As it is defined today as a binary state (agree/disagree), it is impossible to really be compliant with GDPR and the latest information given by DPAs like the CNIL at the end of 2018, as they require granular choices (by purpose and by vendor) to be exposed to the user. A global yes/no choice is not enough from that perspective. While we might be able to achieve that for IAB vendors with your current implementation, that will not be doable for non-IAB vendors like analytics vendors.

This is a great question. And as you've pointed out this is achievable for IAB vendors. To support more granular consent blocking for non-IAB vendors, it requires AMP to come up with a consent string standard and it's not something we are planning to do. However if that becomes a popular request we will definitely consider adding more granularity.

Thanks again for the questions and feedbacks. Let me know there's more I can help with.

janwinkler commented 5 years ago

One question regarding the examples above:

window.parent.postMessage({ type: 'consent-response', cmpStoredData: '', consentStr: '', action: 'accept/reject/dismiss' }, '*');

Is this example still correct or does the message need to be a json string instead?

zhouyx commented 5 years ago

@janwinkler Good point. The example still applies, except we change the consentStr to info and the cmpStoredData is currently not supported.

dmdabbs commented 5 years ago

We'll start with 300 bytes limit and work on the LRU/LRR to expand the storage limit.

Consent strings are likely to be larger than this within the next year as the IAB TCF version 2 begins deploying.

Currently we haven't documented on how vendors are expected to query/receive the IAB consent string yet. But it's definitely we will be adding soon.

Perhaps I'm misunderstanding here. (Ad tech) vendors need access to the consent string to send in their ad fetch calls so that they know whether the proper disclosures were made to the user and whether they have consent (if they're relying on consent). Not sure I understand the point of a CMP that generates consent strings to which vendors have no way to access.

jawadst commented 5 years ago

@zhouyx Thanks for your answers! A few extra comments/questions for you:

On the format that is stored by AMP in the consentString field (consent string vs CMP-specific string): if what is stored gets passed to vendors then it forces CMPs to store the IAB consent string there and prevents us from storing consent for non-IAB vendors. I think we'll have to assume that non-IAB vendors are not supported in this first version but please keep them in mind for future evolutions. They are a big part of what we do with our publishers. One option would be to open another field (cmpConsent) that would only contain CMP-specific information and would allow us to store extra information as needed.
Do you have a way to exit the fullscreen mode? We sometimes display a banner notice (so the consent UI is not fullscreen initially) with a link to open more information in a pop-in. The pop-in uses the fullscreen mode when it opens which works great. When the user closes the pop-in and if they have not made a consent choice yet, we want to exit the full-screen mode and leave the banner notice open. It does not seem like there is an action to exit the full-screen mode.
CSS for iframe Would it be possible to remove the CSS that has a visual impact (border, for instance) on the prompt UI iframe? I think that there should not be anything else than positioning/height applied. Right now, it prevents us from styling the UI exactly how we want.
Consent re-collection There are a few cases when consent needs to be recollected and the consent UI needs to be shown again after the user gives consent:
- Consent is more than 13 months old (ie consent expiration)
- A new vendor is added to the list of vendors And publishers/CMPs usually add extra logic there (a minimum window to never display the consent UI more than once every X days, for instance). How are those cases managed currently? Ideally, the promtpUI iframe would always be loaded in a hidden state and it would have a special action to let AMP know if consent needs to be collected or not. The iframe/CMP would then be in charge of deciding whether consent needs to be collected or not (which would also allow us to apply extra logic as needed). That would effectively replace the checkConsentHref call with communication with the iframe. It could also be done with the checkConsentHref call if that call was done on every single page load (and not just when consent is unknown) and that call included the same information as what is passed to the iframe (consentString, consentState, etc.)

To summarize, I think the following two points are blockers at this point:

We need more clarity on how IAB vendors will get passed the IAB consent string and we need some vendors buy-in. I think this might be the right timing to join one of the IAB Techlab meetings for a presentation to get some vendors involved. Happy to help facilitate that if needed.
- Consent recollection is absolutely required. Publishers add a lot of new vendors regularly and we need a way to define when the consent UI should be shown again.

zhouyx commented 5 years ago

@dmdabbs

Consent strings are likely to be larger than this within the next year as the IAB TCF version 2 begins deploying.

Thanks for bringing it up. It's definitely something we keep in mind. We are happy to explore ways to loose the restrictions then.

Perhaps I'm misunderstanding here. (Ad tech) vendors need access to the consent string to send in their ad fetch calls so that they know whether the proper disclosures were made to the user and whether they have consent (if they're relying on consent). Not sure I understand the point of a CMP that generates consent strings to which vendors have no way to access.

I'm completely with you on this point. The support is already in, the documentation work is deferred until we have the CMP support.

zhouyx commented 5 years ago

@jawadst Thank you for the additional feedback.

On the format that is stored by AMP in the consentString field (consent string vs CMP-specific string): if what is stored gets passed to vendors then it forces CMPs to store the IAB consent string there and prevents us from storing consent for non-IAB vendors. I think we'll have to assume that non-IAB vendors are not supported in this first version but please keep them in mind for future evolutions. They are a big part of what we do with our publishers. One option would be to open another field (cmpConsent) that would only contain CMP-specific information and would allow us to store extra information as needed.

If I understand it correctly, what you want is to store a piece of information that's only for CMP use. This is something we can support as long as the string size under the cap. What confuses me a bit is How will this string work with non-IAB vendors?

Do you have a way to exit the fullscreen mode?

Short answer: No, but we can definitely add support

CSS for iframe

Sounds reasonable to me. @torch2424 who is the owner of amp-consent UI can help answer the question better.

Consent re-collection

Thanks for bringing this up. AMP decides to support a one behind model that allows the endpoint to erase the stored consent and force prompt next time. This will be supported with the 'forcePromptOnNext' feature. However it is something the team decides to de-prioritize until CMP integration is added. I'm happy to prioritize if you find it to be a big blocker.

We need more clarity on how IAB vendors will get passed the IAB consent string and we need some vendors buy-in. I think this might be the right timing to join one of the IAB Techlab meetings for a presentation to get some vendors involved. Happy to help facilitate that if needed.

I wonder in the non AMP case, how CMP passes the IAB consent string to vendors? I assume it will be very similar in AMP pages. Happy to discuss this further. cc @jasti

ampproject / amphtml