search

beeware / toga

A Python native, OS native GUI toolkit.
https://toga.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
4.35k stars 672 forks source link

Design discussion for Clipboard functionality #1192

Open t-arn opened 3 years ago

t-arn commented 3 years ago

I would like to have a cross-platform functionality in Toga which allows us to copy data into the system clipboard and paste data from the system clipboard.

I already made a working proof-of-concept for the data type "Text" for Windows and Android (see PR #1191) as a base for discussion. The PR implements the Clipboard class with following methods:

It is not clear, though, how to handle data types other than Text because there are a lot of differences between Windows and Android, and other platforms will probably add to this diversity.

On Windows, you can only have 1 clipboard item, but you can store this item in several different data types at the same time (see https://docs.microsoft.com/en-us/dotnet/api/system.windows.forms.dataformats?view=net-5.0): Bitmap, CommaSeparatedValue, Dib, Dif, EnhancedMetafile, FileDrop, Html, Locale, MetafilePict, OemText, Palette, PenData, Riff, Rtf, Serializable, StringFormat, SymbolicLink, Text, Tiff, UnicodeText, WaveAudio So, you could have an image in the clipboard in the formats Bitmap, EnhancedMetafile and Tiff.

On Android, you can have several clipboard items, but every item has just 1 specific data type being one of: Text, URI or Intent (https://developer.android.com/guide/topics/text/copy-paste) When you use URI, you must provide a Content Provider which will be responsible to actually access the data. The data itself is NOT in the clipboard. In the clipboard, you only have the URI and the Content Provider.

So, the question is: How should we design the Clipboard feature, so that it is usable on all platforms with a uniform interface?

saroad2 commented 3 years ago

Love what you did here! I think it can be a great use for all of us. Absolutely amazing!

I think that the platform-specific data saving strategy is not that important. You can simply add methods such as set_image and get_image etc., and the user should know that in Windows it will override the last saved item and that in Android it will add it.

freakboy3742 commented 3 years ago

@t-arn Thanks for taking on this project. Reiterating what @saroad2 has said - this is going to be of great use.

My first question is about the overall access pattern. In the example app in #1191, the end-user is responsible for creating an instance of Clipboard. That suggests that an end user could create multiple clipboard instances. On the other hand, the native APIs on both Winforms and Android are "global" clipboard objects. That suggests to me that the Clipboard constructor shouldn't be exposed to the end user - but should instead be exposed as a property of the app. That way, every app would have a clipboard attribute, effectively making the API a "global" API (since the App is, by definition, a singleton object).

My second question is about Android's clipboard in particular, and how we would support it. Android clipboards can hold multiple items... but what's the UX for accessing those items? Does the user have options other than a simple "Paste" that access the most recent item?

Interestingly, the macOS/iOS Clipboard (internally - NSPasteboard) can support multiple items. The end-user's API is a simple "copy" or "paste"; but programatically, the paste request specifies a "preferred" datatype, and looks over multiple items on the pasteboard to find objects that match that datatype. Is this similar to what happens in practice on Android? The API (at least as you've used it on #1191) makes it look like you need to request a specific index when pasting.

Third, I have slight concerns about forward compatibility of the API for accessing the clipboard. The get_text()/set_text() API is definitely an obvious API option; however, given the nature of the problem, I wonder if it might be limiting. As you've noted, clipboards can support different formats (and different formats on different platforms). The existence of get_text/set_text implies that when we support new content types in the future, the API for those types would be get_image/set_image, get_html/set_html, or similar. This means the only content types available would be the ones that we internally support; and in the fullness of time, that list could get quite long.

An alternative would be a generic get_content(content_type) API, along with a set_content(content) API that can accept arbitrary types of content. We could use MIME types as the argument (possibly even defaulting to text/plain, so self.clipboard.get_content() would be a valid call to retrieve plain text). The major benefit I see to this approach is that it allows for a 'pluggable' model of content handling - essentially we'd end up registering "handlers" for a given content type, rather than needing to explicitly expose entry points. This handler registration process could even be exposed to end users, so I could write and install custom clipboard handlers for my own apps for content types that Toga itself doesn't support.

I guess we could even support both - with get_text() being a wrapper around get_content('text/plain') (or the other way around). Any thoughts on this?

Lastly - and perhaps most importantly - there's one aspect of clipboard use that has already come up in bug reports, and relates to the example app you've provided. Your example app includes a button on the app content to trigger copy and paste. This is great for testing purposes, but misses what I would consider the obvious point of interaction: cut/copy/paste menu items. These are standard items on every desktop platform; and while there aren't menus on iOS and Android apps, there are standardised mechanisms for invoking copy/paste functionality. #796 describes a bug related to this - text fields on macOS apps don't allow you to copy or paste, because there isn't a default keybinding for cut/copy/paste that attaches to the clipboard.

I'd consider resolving how the clipboard interacts with the default menu items to be a key part of the design of a clipboard API. In fact, for most purposes, I'd suggest that the end user shouldn't ever need to write their own cut/copy/paste buttons/handlers that are interacting with the clipboard - they'll be relying on the system-provided cut/copy/paste commands that are installed by default when an app is created.

t-arn commented 3 years ago

@freakboy3742 Thank you for thoughts and comments on this topic!

Regarding 1. I completely agree with you - we should expose the clipboard as a property clipboard of the app.

Regarding 2: I have never seen an app that uses the multi-item feature of the Android system clipboard - so, I suggest that we ignore this feature and just support 1 item in the clipboard

Regarding 3: I myself thought about implementing generic methods like get_content() / set_content() where you would pass the data type into it. But I think that using MIME types for defining the data type will not cover all use cases. AFAIK there is no MIME type for Android's Intent, nor for URI. I think that every platform needs to define what data types it supports (see for example the Clipboard.access_types variable in the Android implementation of #1191 At the interface level, we would then have a super-set of all data types supported by the platforms. This could be just for documentation, or, the interface could already reject data types which are not supported by at least 1 platform implementation. All the same, I would definately keep the simple get_text() and set_text() methods because they are the main use case. As you suggested, we could change them into just a wrapper for the generic methods. I like your idea of registering handlers for content types - this might be an elegant solution to the problem that Android expects a content resolver for the URI data type. On the other hand, it sounds like a rather complex way and I would like to keep things as simple as possible. But I don't have a simpler idea, so using handlers might in fact be the most simple way to handle it...

Regarding 4: My view on this issues is this: Toga controls are native controls, so they should also behave like native controls. Which means that if the native control has a standardized way to expose the clipboard commands, then the Toga control should also have them. So, the problem #796 is not a problem of the "Clipboard" - it must be handled by the Toga TextInput implementation for macOS. On Windows and Android, this already works this way: in Windows' TextInput for example, you can use Ctrl-C / Ctrl-V and also a mouse right-click to work with the clipboard (in the MultilineTextInput, the keyboard methods work, but not mouse right-click which I think should be fixed...). There are some controls that do not have a direct native implementation (like DetailedList on Android). Here, I suggest that these implementations provide a way of interacting with the clipboard that fits the platform's similar controls.