Seeking feedback on delayed clipboard rendering proposal

anaskim commented 1 year ago

Delayed clipboard rendering is the ability to delay the generation of clipboard data until it is needed by the target applications. It is especially useful when the clipboard formats supported by target applications are expensive to produce. Delayed clipboard rendering enables web authors to specifically mark delayed rendered payloads in the source application and avoid producing them if they’re not requested by target applications. Our goal is to leverage the existing Async Clipboard API to allow websites exchange large data payloads and improve performance by only producing clipboard payload when it’s needed by target applications.

We detail the proposal in this explainer. We would love to get feedback on our proposed solution, considered alternatives, and open questions.

@snianu @sanketj

snianu commented 1 year ago

Tagging few people who would be interested in this proposal. @annevk @whsieh @a-sully @smaug----

smaug---- commented 1 year ago

Adding @EdgarChen too

johanneswilm commented 1 year ago

2023-01-12: ACTION: everyone to take a look at the existing API and check if it can be used for the delayed rendering use case.

johanneswilm commented 1 year ago

2023-02-09 call:

anasollanokim (Microsoft): presented delayed clipboard rendering last time, resolving feedback: does existing API work? anasollanokim: been working on that but no resolution yet, since we're unsure how the promises are going to be resolved anasollanokim: can't guarantee when the promise will be resolved browser-side anupam (Microsoft): need signal that system is trying to read data, exposed to web whsieh (Apple): that's when promise is called text/html => promise1 image/png => promise2 promise2.when() johanneswilm: what to do when navigating away? johanneswilm: when you navigate away, the promise is gone whsieh: (or the promises are all resolved — there are multiple ways to solve this problem) anupam: We will work with our partners and see if the promise approach works for them. If not, then we can discuss further.

annevk commented 1 year ago

FWIW, I find "rendering" a bit confusing here, but the description is relatively clear fortunately. I tend to agree with the feedback in the call that the existing API allows for this, but might need some refinements.

sanketj commented 1 year ago

Details from 2023-01-12 call:

[torsdag 12 januari 2023] [17:04:49 CET] topic: https://github.com/w3c/editing/issues/417 [torsdag 12 januari 2023] [17:04:49 CET] * github-bot Because I don't want to spam github issues unnecessarily, I won't comment in that github issue unless you write "Github: | none" (or "Github issue: ..."/"Github topic: ..."). [torsdag 12 januari 2023] [17:06:07 CET] Link to Delayed Clipboard Rendering explainer: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/DelayedClipboard/DelayedClipboardRenderingExplainer.md [torsdag 12 januari 2023] [17:07:59 CET] ana: goal is to improve performance. Going through the proposed solution [torsdag 12 januari 2023] [17:09:31 CET] q+ [torsdag 12 januari 2023] [17:10:56 CET] whsieh: wondering if the the spec already fulfills the use case [torsdag 12 januari 2023] [17:11:37 CET] anypam: we want to wait until the target app needs the data [torsdag 12 januari 2023] [17:11:38 CET] Join AlexK (~AlexK@d9560e24.public.cloak) has joined this channel. [torsdag 12 januari 2023] [17:12:18 CET] https://www.w3.org/TR/clipboard-apis/ [torsdag 12 januari 2023] [17:12:30 CET] typedef Promise ClipboardItemData; [torsdag 12 januari 2023] [17:13:35 CET] whsieh: UA could only call the promise when data is needed [torsdag 12 januari 2023] [17:16:54 CET] sanket: wondering is there are limitations with resolving the promise with a blob [torsdag 12 januari 2023] [17:18:36 CET] we can discuss more about promise vs callback in the issue async. Need some time to think through what all cases might not work with the promise approach. [torsdag 12 januari 2023] [17:19:34 CET] johanneswilm: how does the proposal work if the browser closes [torsdag 12 januari 2023] [17:19:44 CET] or native app [torsdag 12 januari 2023] [17:21:31 CET] smaug: could also consider some generic API for async Blobs or such [torsdag 12 januari 2023] [17:21:50 CET] sanket: are there any concerns about the concept? [torsdag 12 januari 2023] [17:21:58 CET] whsieh: no concerns [torsdag 12 januari 2023] [17:22:10 CET] smaug: no concerns [torsdag 12 januari 2023] [17:22:48 CET] whsieh: would it be left up to UA whether to do delayed rendering? [torsdag 12 januari 2023] [17:23:14 CET] sanket: should probably be MAY in the spec [torsdag 12 januari 2023] [17:24:28 CET] ACTION: everyone to take a look at the existing API and check if it can be used for the delayed rendering use case.

snianu commented 1 year ago

We did some investigations into whether the existing read/write APIs are enough to implement delayed generation of clipboard data. Here are our findings:

Problems with the existing API Current ClipboardItemData type that is a Promise<Blob or DOMString> doesn't provide the web authors an option to only generate the Blobs for specific formats when it's actually requested by the target app where the paste operation is performed. The executor function in the Promise runs immediately, so it defeats the purpose of delayed rendering by not allowing the web authors to only generate the expensive formats when it's requested by the system clipboard.

e.g. function generateExpensiveHTML(resolve, reject) { const blobInput = new Blob([ '<p>Some HTML</p>'], {type: 'text/html'}); resolve(blobInput); }

const promiseBlob = new Promise(generateExpensiveHTML); const clipboardItem = new ClipboardItem({'text/html': promiseBlob}); navigator.clipboard.write([clipboardItem]);

Here the executor function generateExpensiveHTML runs immediately, so the web authors don't have a way to delay the execution of this function until the system clipboard asks for the data for this particular format.

There are couple of solutions that solves the issues mentioned above:

Map of MIME type to promise in ClipboardItem constructor Here the web authors can provide a map of callbacks to formats that are delay generated. That way browsers know which formats to write immediately to the clipboard and which ones to mark as delay generated in the system clipboard. Cons: Redundant format info need to be provided in the callback map that confuses the browser if a format is present in both the callback map and in the ClipboardItem constructor. This also affects ergonomics of the API usage.
Provide a callback argument in the ClipboardItem constructor that returns a Blob when executed. callback ClipboardDelayedCallbackBlob = Blob(); callback ClipboardDelayedCallbackString = DOMString(); constructor(record<DOMString, Promise<DOMString or Blob> or ClipboardDelayedCallbackBlob or ClipboardDelayedCallbackString> items;

In this case the web author can decide which formats to delay generate and which ones to resolve immediately without having to provide the format and callback info in a separate map. This provides better developer ergonomics and easier to reason about as to which formats should be written immediately to the system clipboard and which ones should be marked as delayed generate. We want to pursue option 2, so please let us know if you have any feedback on the API design. Note: We tried option 2 in Chromium, but we are suspecting that the IDL compiler has a bug as it is not able to compile this syntax for some reason. @sanketj opened https://github.com/whatwg/webidl/issues/1278 to seek clarification. @anaskim @inexorabletash @annevk @whsieh @a-sully @evanstade

whsieh commented 1 year ago

Ah, I see — it seems like DOM promise callbacks are always invoked immediately upon construction per the JS spec, rather than lazily. In that case, I agree that a callback is a reasonable way forward.

sanketj commented 1 year ago

Slight update to option 2:

Per the clarification provided in whatwg/webidl#1278, the syntax proposed above won't work as is. The syntax would need to be:

typedef (Blob or DOMString) ClipboardItemValue; callback ClipboardItemValueCallback = ClipboardItemValue (); typedef Promise<(ClipboardItemValue or ClipboardItemValueCallback)> ClipboardItemData; constructor(record<DOMString, ClipboardItemData> items);

megangardner commented 1 year ago

Discussed at editing meeting 3/9/2023 [08:02:27] Issue: https://github.com/w3c/editing/issues/417 [08:03:29] Johnanneswilm: Talking a bout this issue, there've been some updates [08:03:41] jsbell (~jsbell@a11e3e9c.public.cloak) joined the channel [08:03:41] Sanket: Could we use the exsting API? [08:03:59] sanketj The thing that delays that is that there is only one place to pass a promist [08:04:08] sanketj so there is not way to actually delay the execustion [08:04:16] sanketj based on that we need to make an upstae [08:04:26] alexk (~alexk@a11e3e9c.public.cloak) joined the channel [08:04:29] sanketj One idea is a map of callbacks [08:04:43] sanketj a smaller change is to pass in an option [08:04:54] sanketj the 2nd option is it's better egonomics [08:05:04] estade (~estade@a11e3e9c.public.cloak) joined the channel [08:05:07] sanketj and it make is so we don't have to have the same thing specified in two places [08:05:11] sanketj other thoughs? [08:05:40] johanneswilm Option 1 is linked to an explainer, there is an option 2 on that page, is that the 2nd option? [08:05:49] sanketj no, that's a variation on option 1 [08:06:13] apologies for interruption - I got a late invite - is there a voice call as well as IRC? If so can someone point me to connection details? [08:06:34] whsieh yes, option 2 seems the better options [08:06:48] jsbell yes - https://meet.jit.si/W3CEditingCall [08:07:19] sanketj it needs to be a promise to a blob, or a promise to a funciton [08:07:45] sanketj in this case, we'd pass a promise to a function, and it would hold onto it [08:08:05] johanneswilm from a web author's perspective, why do we have a promise that is immediatly resolved [08:08:30] whsieh that is how it works, there's no opportunitu to make it lazy [08:08:43] sanketj ergonomically it's still convientent [08:09:23] sanketj any other thoughts? Josh, Evan others? [08:10:11] estade maybe I'm not understanding, but there doens't seem to be a change to the api [08:10:41] jsbell there was an update that the promise would be egarly resolved, so a callback is needed [08:10:52] estade it doesn't need to be a callback that returns a promise? [08:11:04] sanketj we want to delay the execution of that function [08:11:24] estade it's good to have more options, and its good to nudge authors in the right direciton [08:11:47] estade even if we introduce it, there's nothting that compels an author to use it [08:11:56] estade I see two big issues [08:12:15] estade we'd hve to do something when we have to forward as written the data, copy, close tab, then paste [08:12:25] estade it expose to the user that something is going on under the hood [08:12:39] estade if we go with some with a dialog, then that is. more useable [08:13:04] estade the second thing, if you immediately start an async task, but then it's not expensvie to the user [08:13:18] estade by the time they paste, then it's done by the time the user pastes [08:13:47] estade if you delay at paste, then the user doens't think that it is responsive, and will paste again [08:14:15] sanketj what is expensive - the partner team has a seneriou where they write several formats tot he clipboard [08:14:41] sanketj some are custom formats, to generate the payload, they reach out to a server, populate the data, and the it comes back [08:15:14] sanketj on paste, some users just need one or two of those formates, and they do all that work for no reason [08:15:27] sanketj if it's delayed, then they only need to do that call when it's asked form [08:16:11] snianu if excell online generates a custom format, only excell understands that format [08:16:35] snianu web custom formates are very expensive to produce, it's very heavy, very expensive [08:16:52] snianu even if it's async, it's still expensive and may not be even needed [08:17:08] estade what is expsnvies? cpu, bandwith, etc? [08:17:31] snianu it's all of those, the cost if very expensive for the webauthors, in the case for Excell 9 [08:18:06] sanketj for CPU, you will pay that cost on paste, but because it's so expsnsive it seems worth to delay [08:18:46] jsbell We need to make sure that we say what we mean by 'expensive' [08:19:07] jsbell we should be clear why we delay, why this is better for the user [08:19:19] sanketj sorry for not updating the explainer [08:19:26] sanketj we'll update the background there [08:19:52] sanketj if we look at the native implementation, they do use delayed clipboard renederend because some of the formats are expensive [08:20:30] sanketj if we close the tab, what we notice from the apps that we've tried, then the app will only keep around the cheap formats, and not keep around the expensive formats [08:20:55] sanketj the apps that we're using have their own document model, they have their own wya to know if there was a change in the range that was copied [08:21:13] sanketj the current API will allow them to make changes, or make a 2nd write [08:21:28] estade do aps have the ability to write whenever they want to? [08:21:52] sanketj if the app notices that it's being closed, they can generate some basic formats and write them in a non-deferred manner [08:22:05] estade they can only do that if there's a user action [08:22:32] sanketj yes, only if there's a user actions, but the apps have their own way of determining if there's been a change in the area that has been copied [08:23:01] estade I think that when we're making decsion about web apis, we need to be first concerned about the user [08:23:36] estade and if we're encougaing apps to do something to not have the experience that the user expects, that seems wrong [08:24:06] sanketj yes, but many native apps have delayed rendering, there are situations where certian formats are not going to be there, and they won't be available [08:25:25] snianu technically in delayed rendereing, web authors don't write to the full format, when the callback is executed, if there's a callback associated with it, and the browswer retieves that call back, during that, the web authors produce the payload, then we set that data for the formate, there's not user gesture restriction, if you have the callback, you get the payload [08:25:36] snianu but it doesn't answer your content change questions [08:25:54] snianu the webauthor's don't have to write this [08:26:18] johanneswilm is there's cliboard history, can you go back three items on the shipping OSes? [08:27:13] snianu the system cliboard keeps track of all the formats, so if you copy three formats, and the 2nd copy there's four formats, the first is verision 0, and the second is version 1, the OS knows how to keep track for everything [08:27:26] snianu the OS keeps track of all that [08:27:42] johanneswilm if we go back three items, we don't call the functions [08:28:09] sanketj the window's cliboard history feature, they only save the text format [08:28:22] sanketj if you do that, it will do it at render [08:29:01] jsbell speaking of text, do we want to special case text as a format, the UA could specify the text foram [08:29:14] jsbell require that at least one of the formats not be delay rendered? [08:29:34] jsbell if on shutdown if everything is delayed rendered [08:30:20] estade I can think of one situation when you would want to delay renerding for text, chrom is not very good at selecting megabytes of text, it's slow [08:30:42] estade even when we add this functionaltiy, we should direct to websauthoers that this isn't the best way to optomise [08:31:14] estade say there's 5 formats, if you could just download once and converge [08:31:26] estade if the user never pastes, that doesn't seem a big deal to me [08:31:55] estade what's the ration of copiest to paste? probably not 10 to 1, probably more like2 to 1 [08:32:06] estade if there's 5 things that need to be generated on the suerver [08:32:27] snianu [via chat] Just a small correction: Currently, the built-in clipboard history only supports text, HTML, and images less than 4 MB in size. [08:32:30] estade it would be good to go back to see why native apps use this [08:33:27] estade I rememebr when we were first building chrome, we did not find this to be a good tradeoff, it could be deferred, at the time, we determined it wouldn't be a good tradeoff [08:33:50] estade native apps don't have the problem, it wouldn't be a big issues to use local CPU cycles [08:34:04] estade in short, is it not possibel for the computation to be pushed to the client? [08:34:11] estade is that not a possibility? [08:34:24] estade are we really painted into a corner in order to need to implemnent this API [08:34:43] estade I think it will be a worse user experience, even if it's necessary [08:34:59] sanketj even if you do all the optimiattions on the server, it can still be expensvie [08:35:09] sanketj if you're a very popualr web app, [08:35:28] sanketj I dn't think we are necessariy trying to encourge or discourage delayed renedering [08:35:43] sanketj from the user, the not delayed experinece, is bettter [08:36:01] sanketj but delayed clipboard rendering does help the apps sometimes [08:36:14] sanketj we aren't forcing the use of delayed rendering [08:36:23] sanketj but this allowes the app to make that tradeoff [08:36:36] sanketj we're also only modlifying the async clipboard api [08:36:46] sanketj not the other one that would have more impact [08:37:03] estade the platform should be somewhat oppinionated [08:37:25] estade we don't want to say this is optomised when there are tradeoffs [08:37:40] estade what's the status, are they slow, or are they holding features back? [08:37:59] sanketj features are being held back becaue it's just too expenive without delayed clipboard rendering [08:38:19] sanketj the expenives formats are also not used, so it seems like an acceptable tradeoff [08:38:50] snianu on excell online, if it takes too long, the paste fails, it shows a message that the app is unable to render the data [08:39:01] johanneswilm is that happening server side? [08:39:19] snianu yes if they server isn't able to make the payload quickly enough [08:39:34] snianu and if this with the normal API [08:39:55] snianu at least with promises, we wait for all the promises to resolve [08:40:12] snianu if we're not able to product the data is x seconds, then it fails, [08:40:24] snianu it's not worth all the back and forth and they timout the paste [08:40:51] estade I know there's optimizaiton in chromium to make the async clipboard api [08:41:14] estade a webapp should work more and more like a native app, it should still be able to work like a native app [08:41:24] estade you should still be able to copy and paste quickly [08:41:44] estade is this only relegvant to web apps that have given up on offline experineces [08:42:34] sanketj no, it's the other way around, even some of these native apps, have become more and more of a connected experience, syncing your documents from one place to another, there's a good amount of data that's processed ont he server rather than on the client even in native apps [08:42:50] sanketj not everything can be done int he client, in both a native and a webapp [08:43:17] estade I find that disheartening, it would be nice to be able to use the app even if I'm on an airplane [08:43:43] estade maybe we'll solve internet connectivity, but for now, all things should be moveing towards working better with a shakey or not internet connection [08:44:03] estade what's the copy paste esperience when you're teathered to your cell phone and you ahve a bad connection? [08:44:28] estade if every copy opperations requires a heavy sever componenet [08:44:38] estade as an app authoer I'd be trying to offload more of that to the client [08:44:58] sanketj basic functionality will not stop working, but it's more about the complext formats [08:45:10] sanketj more and more of those things will be powered by a web experience. [08:46:29] sanketj to summerise our point, I think we, I think the current API surface is to not forcet he web in one way or ther other. if there's more way to be explicit in how we delay render. there's a deisre to make things work more like native [08:46:58] johanneswilm make it would make sense of having some real life examples of where this is needed. having real examples that cannot be transformed in client, where you need these server calls. [08:47:26] johanneswilm if we have such examples, then maybe we can move forward, other wise it's just back and forth on whether this is needed [08:47:38] sanketj we can find out more information on where this is needed [08:48:02] sanketj the native apps do use delayed clipboard rendering, and there is no way to do this on the web today [08:48:21] estade more concrete examples help make the case [08:48:40] estade a way to make the case even stronger, something that preserver some of the delay [08:48:57] estade your custom format can be jsut enough data for the other app to fetch the data itself [08:49:09] estade this means there isn't a probelm with closing the app [08:49:16] estade this is the one though [08:49:28] estade adding that, or other alternatives can really help [08:49:43] estade besieds the arguemtn of this is how it's been done in the paste [08:49:50] estade you don't want to reinvent the wheel [08:50:16] johanneswilm sanketj can you find some examples [08:50:30] sanketj estade could you provice more exmaples of a better experinece? [08:50:46] estade the directive we follow is to put the end user first [08:50:52] estade then the webapp auther [08:50:57] estade then user agent laast [08:51:18] estade we've identified in some ways that a async but eager copy can make for a better exerinece [08:51:41] estade if you can give some sort of token where the receiivne app can get the date themselves [08:52:12] estade it's potentially a better end user experience that can be explored [08:52:21] estade that's what I'm focusing on, is the user [08:52:39] estade and of course it's not a good user experinece for them to pay for downloading data when they paste [08:52:57] johanneswilm anyone else? [08:53:20] sanketj we are limited a bit based on what the OS cliboard offers, if there's not a way to put tokens, then we can't do that. [08:53:40] sanketj we can think of some other solution, but we need to integrate with the OS clipboard. [08:54:13] johanneswilm ok, until next time, sanketj try to find examples for this, and we recovien in a month. [08:54:43] alexk we have a pr in review for and have a comple more questions [08:55:04] alexk good news for the project, google has more recources they can give us [08:55:21] alexk #33 in the edit contect repo [08:55:22] -Github- https://github.com/w3c/editing/issues/33 : Should execCommand be spec'd to do nothing in cE=typing? [08:55:47] johanneswilm this is still continuing but slow [08:55:48] ? [08:56:06] alexk that's it for edit context [08:56:17] estade (~estade@a11e3e9c.public.cloak) left IRC (Ping timeout: 180 seconds) [08:56:38] sanketj I just wanted to say that given that the next working group call is a month away, is there any way we can resolve and make some progress in the issue [08:57:13] johanneswilm some history, we just did things on an e-mail list, we can do that in the meantime [08:57:33] johanneswilm there's nothing that says we have to do things in these calls, the e-mail list can make things more faster [08:57:47] johanneswilm put your things on github, and then send and e-mail saying you want feedback [08:58:09] sanketj sounds great, even, josh, and wenson, anyone from firefox? [08:58:14] sanketj Ali right? [08:58:22] https://lists.w3.org/Archives/Public/public-editing-tf/

inexorabletash commented 1 year ago

A comment I made that didn't make it into the notes: UAs could consider integrating with the Reporting API for the scenario where a site places deferred content on the clipboard, the user navigates away from the site, and the user attempts to paste. This could act as a signal to sites that they should consider doing something in "beforeunload" handling, e.g. prompting the user that clipboard data will be lost.

I think I've seen native Excel do something like this at least on macOS; I think it has heuristics and makes the clipboard content deferred if over a certain size, and warns on app shutdown?

ETA: To clarify: someone else in the call suggested the beforeunload approach, which would be good to document as part of the proposal. I was suggesting that if a site doesn't do that, the Reporting API might be a way for the UA to provide a signal to web developers that it's something they should consider implementing.

benjamind commented 1 year ago

From the Adobe perspective, speaking on behalf of the PSWeb team we would indeed have use for a delayed clipboard rendering interface. Our underlying native code already supports and uses this on OS's that support it. So we are definitely in support of having similar capabilities on Web.

We would however need clarification on the path to ensuring that a pending clipboard copy is fully rendered before the window/tab is closed (beforeunload might be sufficient, but I'm unclear on how this would work).

It would also be interesting if the API could support streaming results, since PS can be dealing with very large data copies, and unnecessary buffer allocations may make that prohibitive.

mkruisselbrink commented 1 year ago

Kind of a tangent, but I'm a bit surprised to not see any mention of the previous attempt of including this kind of functionality in the spec (which still exists in the working draft) in the explainer, i.e. the https://www.w3.org/TR/clipboard-apis/#dom-clipboarditem-createdelayed method.

I don't think anybody every tried actually figuring out how that method should work, but I would expect to at least see it mentioned as a considered alternative API shape?

mkruisselbrink commented 1 year ago

I believe https://docs.google.com/document/d/1lpi3-9vBP_1b7hZc2xBs0s_HaACJ6UigZZqHlJSNeJg/edit#heading=h.cuyqt05mqd5i was the explainer for that feature at the time.

snianu commented 1 year ago

@mkruisselbrink

Kind of a tangent, but I'm a bit surprised to not see any mention of the previous attempt of including this kind of functionality in the spec (which still exists in the working draft) in the explainer, i.e. the https://www.w3.org/TR/clipboard-apis/#dom-clipboarditem-createdelayed method.

I thought I removed this from the spec, but looks like we haven't sync'd the latest changes to the spec with the working draft for some reason? We didn't specifically mention this(which we should have), but we have considered a solution that looks very similar to this one. We'll add this to the list of alternate solutions. Thanks for mentioning this!

rakina commented 1 year ago

drive-by: I noticed that there are some discussions about what to do when the document is navigated away, but it seems like it assumes that the document will be destroyed after that, but that's not always true: the document can get BFCached and stay in a non-fully active state, and later restored. So we can still get the contents of the document for that case, although I'm not sure if we want to behave differently from the non-BFCache case (maybe there's a privacy problem if we do that too?). See also this section and this section of the BFCache guide.

evanstade commented 1 year ago

Thanks for raising this. I think it's reasonable to make pages that own the clipboard with lazy data non-bfcacheable. It shouldn't come up very often.

Edit: to clarify, I don't think a page holding the clipboard can be allowed into bfcache because if the user tries to paste, we we have no way to resurrect the page so the paste will fail. This is just like a page having an unload handler, which I presume prevents deactivation.

fergald commented 1 year ago

@benjamind what does the adobe desktop application do if the user requests shutdown? Does it block until the clipboard content has been produced?

Doing the same thing on the web is (generally) no OK. We do not trust pages to behave well. We have a limit on the time they can keep computing after being closed. We do not want pages that no longer have a visible UI to consume resources indefinitely as it's much harder for the user to identify and kill them.

One possibility is for apps that use this API to install a beforeunload handler to say "you have data in the clipboard that isn't ready, are you sure you want to leave?", similar to unsubmitted forms.

rakina commented 1 year ago

Thanks for raising this. I think it's reasonable to make pages that own the clipboard with lazy data non-bfcacheable. It shouldn't come up very often. Edit: to clarify, I don't think a page holding the clipboard can be allowed into bfcache because if the user tries to paste, we we have no way to resurrect the page so the paste will fail. This is just like a page having an unload handler, which I presume prevents deactivation.

New APIs shouldn't make pages non-bfcacheable. It will either make the user experience worse by making history page load times worse (as it could've been instant), or it will make web developers not want to use the API because they want to avoid losing the instant load. We've disabled BFCache on old APIs, but that's to prevent existing pages from breaking because they didn't expect the page to be BFCached and the API to behave differently.

I think the API here should behave the same way on navigation, regardless of whether the document got bfcached or not. Whether we drop the contents on normal navigations or immediately do the work then, I don't see why we can't do that on navigations where we bfcache instead.

evanstade commented 1 year ago

@fergald I have the same concern and I think it's an important detail to work out before proceeding with the API. At least for Chromium we cap the duration of beforeunload handlers because we don't want tab shutdown to be slow. Also I don't think onbeforeunload handlers are allowed to specify custom text.

@rakina sure, in a world where navigating away eagerly fetched the clipboard contents, we could do that whether or not the page was going into bfcache. I thought you were suggesting that we wouldn't need to grab the data because the page might come back. If developers are worried about snappy navigations they probably won't use this API regardless, since this proposed work-on-shutdown model is going to be slow one way or another.

annevk commented 1 year ago

The executor function in the Promise runs immediately, so it defeats the purpose of delayed rendering by not allowing the web authors to only generate the expensive formats when it's requested by the system clipboard.

Yes it does run immediately, but you don't have to call resolve() right away. You'd only call resolve(blob) once you're actually ready, which can be later on in the event loop based on some event. Am I misunderstanding something or were promises not understood?

I guess what you're saying is that there's currently no event for such a request?

snianu commented 1 year ago

@annevk So, we don't want to just delay the write of the format's data for x seconds, we don't want to write the data for that format at all if the target app (where the paste is happening) doesn't need it. Here are couple of scenarios where we think this feature would be useful: User copies cells from the native Excel apps:

With just text in the cells, we see 22 different formats on the clipboard. Native Excel uses delayed clipboard rendering, so we don’t have the data for all the formats in the clipboard. The data for a particular format gets populated when the user pastes the content in an app that supports that format. E.g. When the user pastes this content in MSPaint, image formats are being read from the clipboard, but not the other formats (like CSV, HTML, Link, etc.) In this scenario, we can see that since the destination app is not known during copy, the native app has to produce all the formats it supports for paste operation. The cost for serialization of data for each of these formats is really high, so the app delay renders the most expensive formats (such as XML Spreadsheet, Bitmap, Embed Source, etc.), and populates the common ones that are relatively cheaper to produce (such as text, Unicode Text, etc.)

On the web, we support web custom formats that apps can use to copy/paste high fidelity content between web-to-web, web-to-native or vice versa. These formats are expensive to produce, and only the app that supports the custom format can parse its content, so these formats are ideal candidates for delay rendering.

For Excel online specifically, the model lives on the server, so copy-paste involves data transfer between the client and the server. This leads to a lot of COGS due to server-side processing and large amounts of data being transferred over-the-wire, particularly for large payloads. With delayed clipboard rendering, Excel online app is looking to efficiently handle the web custom formats (that are expensive compared to text & html) when it’s not requested by the target app where the user pastes the content copied from Excel online.

Scenario 2 (Adobe PS use case) From the PS perspective, copying a layer or object has two behaviors, it creates an internal clipboard copy which remains inside the PS engine (and is a pointer to immutable data structures and virtual memory backed data) and is used for internal copy and paste, and on web surface now also has to create a png encoded rendition and pass this to the clipboard API. Delayed clipboard rendering would mean the app could avoid that additional rasterization and encode until the user pastes externally to the app. This also plays into the fact that PS documents can be massive – encoding a 16k x 16k image that is never used is prohibitive to the UX, and may cause memory issues to boot. CPU time savings on copy, memory saving would be some of the benefits of delaying rendering of a format.

inexorabletash commented 1 year ago

To be pedantic, this could be done with promises with the addition of another signal, e.g. an event as @annevk mentions. I think this would be a poor developer experience compared to the callback approach proposed above. Here's what it could look like, just to demonstrate:

navigator.clipboard.write(new ClipboardItem({
  'text/html': new Promise(resolve => {
    // somehow this event target is scoped to this clipboard item?
    // event will only be fired once, even if paste happens again?
    some_target.addEventListener('some_event_type', async e => {
      // are we the type handler that's actually desired here?
      if (e.requestedType === 'text/html') {
        // do a bunch of stuff here, probably async
        resolve(results);
      }
    };
  }),
  /* repeat the above for every supported type, but we'll only ever call resolve() for one Promise */
}));

There may be a cleaner way, but IMHO a callback seems much cleaner.

tilgovi commented 1 year ago

A callback definitely seems like the way.

Is there any value in having the clipboard item data ever be a Promise? In other words, rather than this:

typedef (Blob or DOMString) ClipboardItemValue;
callback ClipboardItemValueCallback = ClipboardItemValue ();
typedef Promise<(ClipboardItemValue or ClipboardItemValueCallback)> ClipboardItemData;
constructor(record<DOMString, ClipboardItemData> items);

I might expect this:

typedef (Blob or DOMString) ClipboardItemValue;
callback ClipboardItemValueCallback = (ClipboardItemValue or Promise<ClipboardItemValue>) ();
typedef (ClipboardItemValue or ClipboardItemValueCallback) ClipboardItemData;
constructor(record<DOMString, ClipboardItemData> items);

That signature lets the value be provided immediately, deferred, or deferred and resolved asynchronously.

evanstade commented 1 year ago

Is there any value in having the clipboard item data ever be a Promise?

Yes, because we want to allow sites to construct the data asynchronously, but eagerly. (Which is how the API currently behaves.) I believe what we actually would like is:

typedef (Blob or DOMString) ClipboardItemValue;
callback ClipboardItemValueCallback = Promise<ClipboardItemValue>();
typedef (ClipboardItemValueCallback or Promise<ClipboardItemValue>) ClipboardItemData;
constructor(record<DOMString, ClipboardItemData> items);

However, it's apparently not possible for a Promise to be part of a union. Thus I think the next best option is to be able to provide additional data to the ClipboardItem with setter methods rather than in the ctor.

tilgovi commented 1 year ago

Is there any value in having the clipboard item data ever be a Promise?

Yes, because we want to allow sites to construct the data asynchronously, but eagerly.

Right, that makes sense. Four options, then:

Eager: a DOMString or Blob
Eager, but asynchronous: a Promise of a DOMString or Blob
Deferred: a callback returning a DOMString or Blob
Deferred, but asynchronous: a callback returning a Promise of a DOMString or Blob

However, it's apparently not possible for a Promise to be part of a union.

If that's the case, then the callback will have to return a promise. That seems okay, but I think it'd be more ergonomic for the application developer to return a value synchronously if they want to.

evanstade commented 1 year ago

Yes, because we want to allow sites to construct the data asynchronously, but eagerly.

Right, that makes sense. Four options, then:

Eager: a DOMString or Blob

Eager, but asynchronous: a Promise of a DOMString or Blob

Deferred: a callback returning a DOMString or Blob

Deferred, but asynchronous: a callback returning a Promise of a DOMString or Blob

Are these suggested alternative ctor param types? If so, they can't be or'd because of the promise. But I agree with the premise they could all be useful.

I agree it would be nicer not to have to wrap your argument with Promise.resolve() but it also doesn't seem too awful, and it's what the API was launched with, presumably because of the Promise union problem.

tilgovi commented 1 year ago

Not alternative params, no, just trying to enumerate the usage scenarios.

fergald commented 1 year ago

@snianu 2 general comments

Those examples are great. I think the explainer would benefit a lot if you moved them in there.
There are a bunch of different concerns in this issue and in the open questions. It might be best to open separate issues (maybe in your own repo) for each of them, CCing people who've expressed an opinion on them, as this issue is getting quite large.

smaug---- commented 1 year ago

To support streaming, and async data fetch, could ClipboardItemData be Promise<(DOMString or Blob or ReadableStream)>? If ReadableStream was used, UA could pull the data when needed. Streams spec has an example. If needed, clipboard could have its own kind of controller which would let one to enqueue not only raw data but also blobs or strings.

saschanaz commented 1 year ago

If needed, clipboard could have its own kind of controller which would let one to enqueue not only raw data but also blobs or strings.

Should be doable, yes, but not with a custom controller but with a custom read request perhaps. But I'd rather expect the caller to do the needed conversion with TextEncoderStream and blob.stream().

annevk commented 1 year ago

Note that we have a significant privacy concern with callback-based production of clipboard data. In particular for non-built-in-types this could allow websites to determine the target application. (See https://github.com/WebKit/standards-positions/issues/144#issuecomment-1483754712.)

w3c / editing

Seeking feedback on delayed clipboard rendering proposal #417