Smart Paste - So much potential

[Philo]

Having recently discussed with colleagues the feasibility of use AI as a "form assistant" and wondering how best to go about it I see this little beauty appear, I managed to integrate a PoC into an existing product in perhaps 30 minutes that worked like magic. After cautiously showing this to a couple of product people they immediately drew up a wish list!

Is it feasible for the smart paste component to also work from an image in the clipboard?

For example a user may screenshot a table of data or form and wish for that to be the source of the data extract, I realise it would be better to copy the content as text but we would have use cases where the content is for example a scanned document that maybe an image file or pdf that can't have text copied.

Is it feasible that smart paste could also be expanded to other data source than the clipboard?

For example, a user uploaded a PDF in step 1 of a process, that PDF is then used as the source of default form input values on the subsequent page? Sounds a bit like traditional OCR but of course with the power of an OpenAI GPT LLM to intelligently understand the source context and destination. i.e. a PDF proforma with user details that when uploaded can presented a pre-populated form of user details back to the user to validate, enhance and submit back.

I'm guessing this is straying away from this being a component and more a back-end service, but assume there is a lot of similarity in how the implementation would work.

A lot of our use cases would be healthcare related where the number of PDFs and faxed/scanned paper documents is quite amazing, even in 2024 so we would be happy to contribute to enhanced functionality if/when source was available.

Thanks in advance

[ADefWebserver]

I think extracting information from an image can be considered something 'general' that a lot of us could use.

[SteveSandersonMS]

This sounds like a great idea.

My guess is that you'd really need the UX to be different though, since people won't "copy" a PDF or image to their clipboard (even if that's technically possible, it would be impossible to explain to end users how to actually do that).

So instead I think the UI would need to look more like an "upload file/image" button they click and then pick a file. Perhaps that would make most sense as a separate smart component called <SmartUpload> that lets you configure things like:

• Which form to auto-fill based on the uploaded data
• Or, some instructions on what kind/shape of data you want to extract from the files, and then it invokes an arbitrary callback with that data so you can do whatever your application needs with it

What do you think?

[Philo]

I agree that a "smart upload" component would suit a number of requirements related to data extraction from files or even images and is something I and others in my profession would bite your hand off for :)

The image example is quite niche, a closer to real world example from healthcare...

A clinical user would typically need to open multiple applications (web or desktop) in order to complete a task, let's say booking an appointment for a patient. The patient information maybe held within "SystemX", this is a desktop application and the data is presented in a form view, while the appointment system maybe web based. As it not untypical these system's are not integrated so data needs to be re-keyed. If the user was able to snapshot the patient information form and present that to the web ui via a "SmartX" type of component this would be beneficial in:

• Reducing time and re-keying effort
• Reducing chances of typos and inaccurate information transfer
• Remove or provide a stop gap to the full integration of the 2 systems (which would usually take several months in healthcare)

[ADefWebserver]

As it not untypical these system's are not integrated so data needs to be re-keyed.

You could also take a picture of a paper form and then paste that information into several applications.

[RyoukoKonpaku]

Maybe allow the smart paste to accept a text input instead of just using the clipboard would be nice.

Something like a C# interface method of the @ref instance (e.g. interactive Blazor) or JS function from the DOM instance (non-interactive scenarios) that can accept a string and infer from that input instead of the clipboard, or maybe even a value instance property where you can update via Blazor's @bind functionality or DOM APIs. This allows more flexibility on how we can retrieve such text (e.g. API call, OCR, file system etc...) without needing specialized implementations of smart paste.

[pierslawson]

We were just discussing at work the potential for smart paste if it could work against an image as well... e.g. a quick screen capture of another screen or part of an image etc.

Regarding UX support for files, perhaps it could also be a drag / drop target for dropping a file (jpg, pdf, docx, audio etc etc). Last idea... perhaps it could capture an image live from the camera or listen to the microphone! ;-)