Open anaskim opened 1 year ago
Tagging few people who would be interested in this proposal. @annevk @whsieh @a-sully @smaug----
Adding @EdgarChen too
2023-01-12: ACTION: everyone to take a look at the existing API and check if it can be used for the delayed rendering use case.
2023-02-09 call:
anasollanokim (Microsoft): presented delayed clipboard rendering last time, resolving feedback: does existing API work? anasollanokim: been working on that but no resolution yet, since we're unsure how the promises are going to be resolved anasollanokim: can't guarantee when the promise will be resolved browser-side anupam (Microsoft): need signal that system is trying to read data, exposed to web whsieh (Apple): that's when promise is called text/html => promise1 image/png => promise2 promise2.when() johanneswilm: what to do when navigating away? johanneswilm: when you navigate away, the promise is gone whsieh: (or the promises are all resolved — there are multiple ways to solve this problem) anupam: We will work with our partners and see if the promise approach works for them. If not, then we can discuss further.
FWIW, I find "rendering" a bit confusing here, but the description is relatively clear fortunately. I tend to agree with the feedback in the call that the existing API allows for this, but might need some refinements.
Details from 2023-01-12 call:
[torsdag 12 januari 2023] [17:04:49 CET]
We did some investigations into whether the existing read/write APIs are enough to implement delayed generation of clipboard data. Here are our findings:
Problems with the existing API
Current ClipboardItemData
type that is a Promise<Blob or DOMString>
doesn't provide the web authors an option to only generate the Blobs for specific formats when it's actually requested by the target app where the paste operation is performed. The executor function in the Promise
runs immediately, so it defeats the purpose of delayed rendering by not allowing the web authors to only generate the expensive formats when it's requested by the system clipboard.
e.g.
function generateExpensiveHTML(resolve, reject) { const blobInput = new Blob([ '<p>Some HTML</p>'], {type: 'text/html'}); resolve(blobInput); }
const promiseBlob = new Promise(generateExpensiveHTML); const clipboardItem = new ClipboardItem({'text/html': promiseBlob}); navigator.clipboard.write([clipboardItem]);
Here the executor function generateExpensiveHTML
runs immediately, so the web authors don't have a way to delay the execution of this function until the system clipboard asks for the data for this particular format.
There are couple of solutions that solves the issues mentioned above:
Map of MIME type to promise in ClipboardItem constructor Here the web authors can provide a map of callbacks to formats that are delay generated. That way browsers know which formats to write immediately to the clipboard and which ones to mark as delay generated in the system clipboard. Cons: Redundant format info need to be provided in the callback map that confuses the browser if a format is present in both the callback map and in the ClipboardItem constructor. This also affects ergonomics of the API usage.
Provide a callback argument in the ClipboardItem constructor that returns a Blob when executed.
callback ClipboardDelayedCallbackBlob = Blob();
callback ClipboardDelayedCallbackString = DOMString();
constructor(record<DOMString, Promise<DOMString or Blob> or ClipboardDelayedCallbackBlob or ClipboardDelayedCallbackString> items;
In this case the web author can decide which formats to delay generate and which ones to resolve immediately without having to provide the format and callback info in a separate map. This provides better developer ergonomics and easier to reason about as to which formats should be written immediately to the system clipboard and which ones should be marked as delayed generate. We want to pursue option 2, so please let us know if you have any feedback on the API design. Note: We tried option 2 in Chromium, but we are suspecting that the IDL compiler has a bug as it is not able to compile this syntax for some reason. @sanketj opened https://github.com/whatwg/webidl/issues/1278 to seek clarification. @anaskim @inexorabletash @annevk @whsieh @a-sully @evanstade
Ah, I see — it seems like DOM promise callbacks are always invoked immediately upon construction per the JS spec, rather than lazily. In that case, I agree that a callback is a reasonable way forward.
Slight update to option 2:
Per the clarification provided in whatwg/webidl#1278, the syntax proposed above won't work as is. The syntax would need to be:
typedef (Blob or DOMString) ClipboardItemValue;
callback ClipboardItemValueCallback = ClipboardItemValue ();
typedef Promise<(ClipboardItemValue or ClipboardItemValueCallback)> ClipboardItemData;
constructor(record<DOMString, ClipboardItemData> items);
Discussed at editing meeting 3/9/2023
[08:02:27]
A comment I made that didn't make it into the notes: UAs could consider integrating with the Reporting API for the scenario where a site places deferred content on the clipboard, the user navigates away from the site, and the user attempts to paste. This could act as a signal to sites that they should consider doing something in "beforeunload" handling, e.g. prompting the user that clipboard data will be lost.
I think I've seen native Excel do something like this at least on macOS; I think it has heuristics and makes the clipboard content deferred if over a certain size, and warns on app shutdown?
ETA: To clarify: someone else in the call suggested the beforeunload approach, which would be good to document as part of the proposal. I was suggesting that if a site doesn't do that, the Reporting API might be a way for the UA to provide a signal to web developers that it's something they should consider implementing.
From the Adobe perspective, speaking on behalf of the PSWeb team we would indeed have use for a delayed clipboard rendering interface. Our underlying native code already supports and uses this on OS's that support it. So we are definitely in support of having similar capabilities on Web.
We would however need clarification on the path to ensuring that a pending clipboard copy is fully rendered before the window/tab is closed (beforeunload might be sufficient, but I'm unclear on how this would work).
It would also be interesting if the API could support streaming results, since PS can be dealing with very large data copies, and unnecessary buffer allocations may make that prohibitive.
Kind of a tangent, but I'm a bit surprised to not see any mention of the previous attempt of including this kind of functionality in the spec (which still exists in the working draft) in the explainer, i.e. the https://www.w3.org/TR/clipboard-apis/#dom-clipboarditem-createdelayed method.
I don't think anybody every tried actually figuring out how that method should work, but I would expect to at least see it mentioned as a considered alternative API shape?
I believe https://docs.google.com/document/d/1lpi3-9vBP_1b7hZc2xBs0s_HaACJ6UigZZqHlJSNeJg/edit#heading=h.cuyqt05mqd5i was the explainer for that feature at the time.
@mkruisselbrink
Kind of a tangent, but I'm a bit surprised to not see any mention of the previous attempt of including this kind of functionality in the spec (which still exists in the working draft) in the explainer, i.e. the https://www.w3.org/TR/clipboard-apis/#dom-clipboarditem-createdelayed method.
I thought I removed this from the spec, but looks like we haven't sync'd the latest changes to the spec with the working draft for some reason? We didn't specifically mention this(which we should have), but we have considered a solution that looks very similar to this one. We'll add this to the list of alternate solutions. Thanks for mentioning this!
drive-by: I noticed that there are some discussions about what to do when the document is navigated away, but it seems like it assumes that the document will be destroyed after that, but that's not always true: the document can get BFCached and stay in a non-fully active state, and later restored. So we can still get the contents of the document for that case, although I'm not sure if we want to behave differently from the non-BFCache case (maybe there's a privacy problem if we do that too?). See also this section and this section of the BFCache guide.
Thanks for raising this. I think it's reasonable to make pages that own the clipboard with lazy data non-bfcacheable. It shouldn't come up very often.
Edit: to clarify, I don't think a page holding the clipboard can be allowed into bfcache because if the user tries to paste, we we have no way to resurrect the page so the paste will fail. This is just like a page having an unload handler, which I presume prevents deactivation.
@benjamind what does the adobe desktop application do if the user requests shutdown? Does it block until the clipboard content has been produced?
Doing the same thing on the web is (generally) no OK. We do not trust pages to behave well. We have a limit on the time they can keep computing after being closed. We do not want pages that no longer have a visible UI to consume resources indefinitely as it's much harder for the user to identify and kill them.
One possibility is for apps that use this API to install a beforeunload
handler to say "you have data in the clipboard that isn't ready, are you sure you want to leave?", similar to unsubmitted forms.
Thanks for raising this. I think it's reasonable to make pages that own the clipboard with lazy data non-bfcacheable. It shouldn't come up very often. Edit: to clarify, I don't think a page holding the clipboard can be allowed into bfcache because if the user tries to paste, we we have no way to resurrect the page so the paste will fail. This is just like a page having an unload handler, which I presume prevents deactivation.
New APIs shouldn't make pages non-bfcacheable. It will either make the user experience worse by making history page load times worse (as it could've been instant), or it will make web developers not want to use the API because they want to avoid losing the instant load. We've disabled BFCache on old APIs, but that's to prevent existing pages from breaking because they didn't expect the page to be BFCached and the API to behave differently.
I think the API here should behave the same way on navigation, regardless of whether the document got bfcached or not. Whether we drop the contents on normal navigations or immediately do the work then, I don't see why we can't do that on navigations where we bfcache instead.
@fergald I have the same concern and I think it's an important detail to work out before proceeding with the API. At least for Chromium we cap the duration of beforeunload
handlers because we don't want tab shutdown to be slow. Also I don't think onbeforeunload
handlers are allowed to specify custom text.
@rakina sure, in a world where navigating away eagerly fetched the clipboard contents, we could do that whether or not the page was going into bfcache. I thought you were suggesting that we wouldn't need to grab the data because the page might come back. If developers are worried about snappy navigations they probably won't use this API regardless, since this proposed work-on-shutdown model is going to be slow one way or another.
The executor function in the Promise runs immediately, so it defeats the purpose of delayed rendering by not allowing the web authors to only generate the expensive formats when it's requested by the system clipboard.
Yes it does run immediately, but you don't have to call resolve()
right away. You'd only call resolve(blob)
once you're actually ready, which can be later on in the event loop based on some event. Am I misunderstanding something or were promises not understood?
I guess what you're saying is that there's currently no event for such a request?
@annevk So, we don't want to just delay the write of the format's data for x seconds, we don't want to write the data for that format at all if the target app (where the paste is happening) doesn't need it.
Here are couple of scenarios where we think this feature would be useful:
User copies cells from the native Excel apps:
With just text in the cells, we see 22 different formats on the clipboard. Native Excel uses delayed clipboard rendering, so we don’t have the data for all the formats in the clipboard. The data for a particular format gets populated when the user pastes the content in an app that supports that format. E.g. When the user pastes this content in MSPaint, image formats are being read from the clipboard, but not the other formats (like CSV, HTML, Link, etc.) In this scenario, we can see that since the destination app is not known during copy, the native app has to produce all the formats it supports for paste operation. The cost for serialization of data for each of these formats is really high, so the app delay renders the most expensive formats (such as XML Spreadsheet, Bitmap, Embed Source, etc.), and populates the common ones that are relatively cheaper to produce (such as text, Unicode Text, etc.)
On the web, we support web custom formats that apps can use to copy/paste high fidelity content between web-to-web, web-to-native or vice versa. These formats are expensive to produce, and only the app that supports the custom format can parse its content, so these formats are ideal candidates for delay rendering.
For Excel online specifically, the model lives on the server, so copy-paste involves data transfer between the client and the server. This leads to a lot of COGS due to server-side processing and large amounts of data being transferred over-the-wire, particularly for large payloads. With delayed clipboard rendering, Excel online app is looking to efficiently handle the web custom formats (that are expensive compared to text & html) when it’s not requested by the target app where the user pastes the content copied from Excel online.
Scenario 2 (Adobe PS use case) From the PS perspective, copying a layer or object has two behaviors, it creates an internal clipboard copy which remains inside the PS engine (and is a pointer to immutable data structures and virtual memory backed data) and is used for internal copy and paste, and on web surface now also has to create a png encoded rendition and pass this to the clipboard API. Delayed clipboard rendering would mean the app could avoid that additional rasterization and encode until the user pastes externally to the app. This also plays into the fact that PS documents can be massive – encoding a 16k x 16k image that is never used is prohibitive to the UX, and may cause memory issues to boot. CPU time savings on copy, memory saving would be some of the benefits of delaying rendering of a format.
To be pedantic, this could be done with promises with the addition of another signal, e.g. an event as @annevk mentions. I think this would be a poor developer experience compared to the callback approach proposed above. Here's what it could look like, just to demonstrate:
navigator.clipboard.write(new ClipboardItem({
'text/html': new Promise(resolve => {
// somehow this event target is scoped to this clipboard item?
// event will only be fired once, even if paste happens again?
some_target.addEventListener('some_event_type', async e => {
// are we the type handler that's actually desired here?
if (e.requestedType === 'text/html') {
// do a bunch of stuff here, probably async
resolve(results);
}
};
}),
/* repeat the above for every supported type, but we'll only ever call resolve() for one Promise */
}));
There may be a cleaner way, but IMHO a callback seems much cleaner.
A callback definitely seems like the way.
Is there any value in having the clipboard item data ever be a Promise? In other words, rather than this:
typedef (Blob or DOMString) ClipboardItemValue;
callback ClipboardItemValueCallback = ClipboardItemValue ();
typedef Promise<(ClipboardItemValue or ClipboardItemValueCallback)> ClipboardItemData;
constructor(record<DOMString, ClipboardItemData> items);
I might expect this:
typedef (Blob or DOMString) ClipboardItemValue;
callback ClipboardItemValueCallback = (ClipboardItemValue or Promise<ClipboardItemValue>) ();
typedef (ClipboardItemValue or ClipboardItemValueCallback) ClipboardItemData;
constructor(record<DOMString, ClipboardItemData> items);
That signature lets the value be provided immediately, deferred, or deferred and resolved asynchronously.
Is there any value in having the clipboard item data ever be a Promise?
Yes, because we want to allow sites to construct the data asynchronously, but eagerly. (Which is how the API currently behaves.) I believe what we actually would like is:
typedef (Blob or DOMString) ClipboardItemValue;
callback ClipboardItemValueCallback = Promise<ClipboardItemValue>();
typedef (ClipboardItemValueCallback or Promise<ClipboardItemValue>) ClipboardItemData;
constructor(record<DOMString, ClipboardItemData> items);
However, it's apparently not possible for a Promise to be part of a union. Thus I think the next best option is to be able to provide additional data to the ClipboardItem
with setter methods rather than in the ctor.
Is there any value in having the clipboard item data ever be a Promise?
Yes, because we want to allow sites to construct the data asynchronously, but eagerly.
Right, that makes sense. Four options, then:
DOMString
or Blob
Promise
of a DOMString
or Blob
DOMString
or Blob
Promise
of a DOMString
or Blob
However, it's apparently not possible for a Promise to be part of a union.
If that's the case, then the callback will have to return a promise. That seems okay, but I think it'd be more ergonomic for the application developer to return a value synchronously if they want to.
Yes, because we want to allow sites to construct the data asynchronously, but eagerly.
Right, that makes sense. Four options, then:
- Eager: a
DOMString
orBlob
- Eager, but asynchronous: a
Promise
of aDOMString
orBlob
- Deferred: a callback returning a
DOMString
orBlob
- Deferred, but asynchronous: a callback returning a
Promise
of aDOMString
orBlob
Are these suggested alternative ctor param types? If so, they can't be or'd because of the promise. But I agree with the premise they could all be useful.
I agree it would be nicer not to have to wrap your argument with Promise.resolve()
but it also doesn't seem too awful, and it's what the API was launched with, presumably because of the Promise union problem.
Not alternative params, no, just trying to enumerate the usage scenarios.
@snianu 2 general comments
To support streaming, and async data fetch, could ClipboardItemData be Promise<(DOMString or Blob or ReadableStream)>? If ReadableStream was used, UA could pull the data when needed. Streams spec has an example. If needed, clipboard could have its own kind of controller which would let one to enqueue not only raw data but also blobs or strings.
If needed, clipboard could have its own kind of controller which would let one to enqueue not only raw data but also blobs or strings.
Should be doable, yes, but not with a custom controller but with a custom read request perhaps. But I'd rather expect the caller to do the needed conversion with TextEncoderStream
and blob.stream()
.
Note that we have a significant privacy concern with callback-based production of clipboard data. In particular for non-built-in-types this could allow websites to determine the target application. (See https://github.com/WebKit/standards-positions/issues/144#issuecomment-1483754712.)
Delayed clipboard rendering is the ability to delay the generation of clipboard data until it is needed by the target applications. It is especially useful when the clipboard formats supported by target applications are expensive to produce. Delayed clipboard rendering enables web authors to specifically mark delayed rendered payloads in the source application and avoid producing them if they’re not requested by target applications. Our goal is to leverage the existing Async Clipboard API to allow websites exchange large data payloads and improve performance by only producing clipboard payload when it’s needed by target applications.
We detail the proposal in this explainer. We would love to get feedback on our proposed solution, considered alternatives, and open questions.
Related discussion: https://github.com/w3c/clipboard-apis/issues/41
@snianu @sanketj