Add Web Custom format

[snianu]

Closes #165

For normative changes, the following tasks have been completed:

• [ ] Modified Web platform tests (link to pull request)

Implementation commitment:

• [ ] WebKit (https://bugs.webkit.org/show_bug.cgi?id=)
• [X] Chromium (https://bugs.chromium.org/p/chromium/issues/detail?id=)
• [X] Gecko (https://bugzilla.mozilla.org/show_bug.cgi?id=)

---

Preview | Diff

[snianu]

@annevk @mbrodesser This PR has the web prefix changes. PTAL. Thanks!

[annevk]

Upon doing code inspection in Gecko we found that the legacy clipboard API might have some support for custom types as well. @EdgarChen is investigating this. We don't really want to end up with a situation where we have two incompatible custom types APIs.

It might be that other browsers have something similar as there were at least some hints of cross-browser collaboration?

[snianu]

@annevk Chromium and Safari have browser specific custom type that is undocumented and doesn't interop with other apps. The format parsing rules are convoluted since it uses browser specific pickling types for serialization/deserialization. This proposal aims to standardize the web custom formats and the format structure so its well documented and is interop across apps on a platform. This doesn't affect the existing pickling types.

In Firefox, I see that it allows writing custom types in setData, but I don't see any restriction on the number of custom types that can be registered. This is a major security issue on Windows and Linux because of the restriction on the number of clipboard formats that can be registered with the OS. This proposal addresses this issue by introducing a fixed set of web custom formats and a map that maps the mime types to native web custom formats.

[EdgarChen]

This proposal aims to standardize the web custom formats and the format structure so its well documented and is interop across apps on a platform. This doesn't affect the existing pickling types.

Do we care about the interop across legacy clipboard API and async clipboard API for custom type? E.g. write custom type via legacy clipboard API and read via async clipboard API, or vice versa.

[snianu]

Do we care about the interop across legacy clipboard API and async clipboard API for custom type?

@edgarchen The async clipboard API model is fairly different, so I'm not sure if we can interop with legacy types. The web custom types are stored inside a JSON map structure, with fixed number of native web clipboard formats, and a well-defined format structure that helps the native apps parse the content of the format. At least in Chromium, the parsing of the custom types requires browser specific pickle types to serialize and deserialize the content, so it's not possible to add support for this format to the new web custom format.

[EdgarChen]

The web custom types are stored inside a JSON map structure, with fixed number of native web clipboard formats, and a well-defined format structure that helps the native apps parse the content of the format.

@snianu We could also write arbitrary type with "web" prefix via legacy API, e.g.

document.addEventListener("copy", function(e) {
  e.preventDefault();

  let clipboardData = e.clipboardData;
  clipboardData.setData("web text/foo", "foo");
  clipboardData.setData("text/bar", "bar");
});

I would expect the type with "web" prefix to follow the well-defined format/structure, too. So it could be interop with async clipobard API/browsers/native apps. for example, if we write the data into clipboard via above example and read clipboard data via async clipboard API

const items = await navigator.clipboard.read();
items[0].getType("web text/foo"); // get "foo"
items[0].getType("text/bar"); // promise reject 

(and vice versa)

Yes, this introduce a behavior changes in legacy API, but I think this also make web custom type less surprise to user? And the impact in legacy API should be small?

[snianu]

@EdgarChen While it's true that you could write custom types with "web " prefix using DataTransfer APIs, those custom types do not follow the same model - there isn't a restriction on the number of custom format types that can be written by the web authors, no web custom JSON map for mapping the web custom types to a fixed number of native web clipboard formats etc. We don't want to make any changes in the legacy DataTransfer APIs as it has high regression risks and the model is fairly different there(at least in Chromium). The native apps on Windows do not know how to parse Chromium specific custom types, so it's not possible for them to exchange formats between web and native apps. Chrome doesn't expose the custom mime types exactly as is - it exposes it via a Chromium specific custom format that uses browser specific parsing types, and it is not standardized, so the native apps don't understand how to parse this structure.

[polx]

Hello @snianu ,

I've tried to review the current state after I have seen your presentation at the TPAC.

Here are my rather nitty gritty comments:

• sec 7.2 (ClipboardItem Interface): should is custom really be written in two words and not in one? The link in the version I could build did not work but that might be a local issue.
• the spreadsheet example close to that is nice, could it not also include a custom type? (e.g. containing text/csv alternative?)
• A web custom format has is custom set to true. => A representation in a web custom format has is custom set to true ?
• 7.3.3: write algorithm: f type is not in the mandatory data types list and has "web ": did you not mean “or has...” ?
• write blobs and option to the clipboard:
  • step 1: why is the variable called webCustomFormat ? It should be formats or something universal, right?
  • substeps under 2.2: "If formatString is empty then follow the below steps:": the steps here does not define a (non-empty) formatString; a value is however needed in order for the recipients to read (suggestion: “set formatString to a randomly chosen format name”)
  • step 2.3: Let payload be the result of UTF-8 decoding item’s underlying byte sequence. : this must be a misunderstanding as it would only apply to text. Maybe the whole section was about text? Then it was not readable
• the steps of “write web custom formats” sound quite solid to me but why is it independent from the previous?

Tell me if I should make my comments more precise (e.g. with line numbers).

I am quite excited by the implementation found in Chromium (and blogged about it). I suspect that we are getting close to acceptance, at least provided that Mozilla will consider implementing as it seems it said at the start.

thanks and sorry to be slow to have reviewed.

Paul

[snianu]

Thanks @polx for the review and apologies for the delay!

A web custom format has is custom set to true. => A representation in a web custom format has is custom set to true ?

Note that in the spec it's referred to as [=representation/isCustom=]

step 1: why is the variable called webCustomFormat ? It should be formats or something universal, right?

This is referring to the custom format with a web prefix, so it can be anything. In the below step, we strip out the web prefix: If |webCustomFormatString| has "web "("web" followed by U+0020 SPACE) prefix, then remove the "web " prefix and store the remaining string in |webCustomFormat|'s {{Blob/type}}'s [=MIME type/essence=], else abort all steps.

step 2.3: Let payload be the result of UTF-8 decoding item’s underlying byte sequence. : this must be a misunderstanding as it would only apply to text. Maybe the whole section was about text? Then it was not readable

This is by-design that we write raw bytes sequence to the clipboard for the custom formats so the reader has to know how to parse its content (which is also by-design).

the steps of “write web custom formats” sound quite solid to me but why is it independent from the previous?

This is because we need to perform an additional step to produce a native clipbaord format and insert the key-value pair into the webCustomFormatMap. This step is not performed for mandatory data types.

[snianu]

@sanketj could you please take a look at this PR so we can merge it into the clipboard repo? Thanks!

[snianu]

@garykac Thank you so much!