[ImageEditor] - Gradio support for ordered points as an output to sketch.

[ysig]

• [x] I have searched to see if a similar issue already exists.

Describe the solution you'd like
I want to use sketch inside gradio but I would like the output to be a poly line or with points ordered as given instead of a mask.

I think this is easy to implement, but it will open new possibilities that can be quite useful.

[abidlabs]

This is a cool idea @ysig! Not sure if it fits into our near-future roadmap, but would be happy to have a contribution towards this end.

[ysig]

I think the main question that arises from this is how easy it is for a user to implement new widgets (tools). Is there a tutorial for this?

[dawoodkhan82]

@ysig we have a guide on how to contribute a new component. https://gradio.app/creating_a_new_component/

[pngwn]

@ysig What is the use case for this? Do you have an example?

Is this a request for an additional drawing 'style' in addition to the current 'freehand pen/ brush'? A line or poly line tool.

In terms of the functionality of a line/poly line tool, would it be desirable for the line tool to have options such as closed/open the line (to ensure that it is a 'complete' polygon when needed)?

[ysig]

@pngwn It's exactly a line or poly tool that returns the order of the points. It doesn't at all need to be closed in my case.

[pngwn]

Ah, so you wouldn't want this as an image, you'd want the path data?

[ysig]

Exactly, I want the path data that could be defined over an image background.

On Tue, Aug 30, 2022 at 12:41 PM pngwn @.***> wrote:

Ah, so you wouldn't want this as an image, you'd want the path data?

— Reply to this email directly, view it on GitHub https://github.com/gradio-app/gradio/issues/1922#issuecomment-1231425008, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGY7H2NFEZ5LI4XNE6AZQULV3XJN5ANCNFSM55I24SXA . You are receiving this because you were mentioned.Message ID: @.***>

[johko]

What is the status on this feature? I just built a space where I could also have used a ordered list of coordinates as sketch output (https://huggingface.co/spaces/johko/homography-warping). As I have a fixed number of points I did a workaround where I use k-means clustering on the sketch output mask to get the centroids and order them by hand, which is working, but still is very hacky :sweat_smile:

I would also volunteer to work on that feature if you guys are short on time

[pngwn]

The difficulty is figuring out how this fits into the API. With a sketch that has lots of features, some people want it all as a single composite raster image, others as separate raster image layers, others want images as data, or as some combination of images + data. When you start to introduce multiple shapes + lines (as a sketch tool is typically capable of) how do you figure out which parts of the image should be raster images (possibly flattened) and which should be data.

It is possible to figure this out but it needs some careful consideration. The team will get around to this but it isn't the highest priority right now.

For reference there is a larger tracking issues for the Image component here: #466.

[dasayan05]

@pngwn @abidlabs any update on this please ? there is a community that works on sketches/vector graphics applications etc. who will immensly benefit from this. I started planning some cool vector-based demos but realized gradio doesn't have good support for this.

[abidlabs]

Hi @dasayan05 this is a really nice use case and we'd like to support it. However, it would add some complexity that we are not ready to support in the core library at the moment. We are working on ways to make it possible for users to easily customize components, which would enable this, I would estimate that it's a couple months out.

[abidlabs]

Hi! We've now made it possible for Gradio users to create their own custom components -- meaning that you can write some Python and JavaScript (Svelte), and publish it as a Gradio component. You can use it in your own Gradio apps, or share it so that anyone can use it in their Gradio apps. Here are some examples of custom Gradio components:

• A "Rich Textbox" that allows you to write bold/italics/colored text
• A "Folium Map Viewer" component that allows you to use interactive maps

You can see the source code for those components by clicking the "Files" icon and then clicking "src". The complete source code for the backend and frontend is visible. In particular, its very fast if you want to build off an existing component. We've put together a Guide: https://www.gradio.app/guides/five-minute-guide, and we're happy to help if you'd like to put together a custom component for this issue

[pngwn]

Still unconvinced we should add this, needs discussion. It isn't an issue in principle but hard to know exactly what kind of data users expect from brush strokes.

[abidlabs]

Agree with @pngwn this is outside the core gradio library, but would be a nice custom component. Please see https://github.com/PhyscalX/gradio-image-prompter as an example of a similar custom component!