Image editor

[pngwn]

user facing

This is the new ImageEditor, which is a replacement for 3.x's Image with tool=.... I have linked the issues that it fixes below (both bugs and enhancements), so i wont go over every detail but the high level is as follows:

This component is a simple + streamlined image editor that handles basic image manipulations. It should be at least equivalent to the old Image with various tools set but less bad and more good. Additional features include:

• Layers

  • The new component had a concept of layers. Certain operations are constrained to a single layer and do not impact other layers. Details on each tool.
  • Each layers is returned individually to the user function.

• Background

  • has a concept of backgrounds.
  • A background is a special type of layer than cannot be drawn on or erased.
  • default is a plain white background which can be drawn on.
  • can also accept an image as the 'backgrounds'
  • upload will upload a file from the users computer
  • webcam will open a webcam capture window which will be set as the bg when it is captured.
  • paste will paste from clipboard

• Draw

  • users can draw on top of any background.
  • drawing is constrained to a single layer and does impact others.
  • The size and color can be selected by the user from a set of swatches and optionally an arbitrary 'picker'
  • The picker is only present if the app author allows it.
  • The size picker is a range input (slider)
  • The color picker is a custom color picker that users will be somewhat familiar with. The color picker also allows users to build their own swatches.
  • I think the color picker is very nice.

• Erase

  • Erase is basically draw but erase.
  • It only affects drawings on the currently selected layer
  • It has only size which operates the same as draw's size

• Crop

  • Crops the entire canvas
  • Crop is non destructive, you can re-enter crop mode and uncrop however you like.
  • i think crop is very nice too.

• Undo

  • as in 3.x we have undo, except this time is actually works

• Redo

  • Because of the architecture (or the fact that it even has one), we can also easily 'redo' any operation.

• Performance

  • The new image editor is much more performant in my experience. It seems to work well with pretty large images and lots of operations. There are definitely opportunities to optimise though, history is not cheap.

technical

The new Image Editor is built on top of pixi.js, a javascript library for creating 2d webgl experiences. It is an exceptional library if a little confusing but it sits at an almost perfect level of abstraction. High level enough so you don't need to think too much about the inner workings, low level enough that you can dive in when you need to. It is not a sketch library in any way but has all of the correct primitives. It is also really fast.

The image editor has a bunch of code but its a lot less than I thought it would need to be. The image editor has been designed to be flexible and maintainable, it should be relatively easy to add new functionality without needing to touch the whole thing. For example, modification to the crop tool, should only require changes to the cropping part of the code.

The editor uses the command pattern which is a little complex but incredibly powerful. Every editing action is a 'command'. A command knows how to perform itself and how to undo itself. We have a command manager that takes in every command, and adds it to its history. When we want a new feature, we create a new command that is dedicated to just that thing. crop is a command, draw is a command, 'add bg image` is a command.

When we want to undo something we ask the command manager to undo it, it will then call the current history node's undo method and move a point back. And repeat.

When we want to redo something it will call the 'next' commands execute method, and move the history point forwards. The history is implement as a doubly linked list so it is simple to traverse and very difficult to skip nodes.

The 'command' interfaces look like this:

interface Command {
  start?: (...args: any[]) => any;
  continue?: (...args: any[]) => any;
  stop?: (...args: any[]) => any;
  execute: () => void;
  undo: () => void
}

Each command can decide how to implement all of these methods, it doesn't matter as long as they 'work'. Only execute and undo are required.

The reason start continue and stop exist is best explained with an example:

Lets take drawing. A single drawing operation is one line. So execute should draw an entire line onto the canvas (for example when we are 'redoing' a line). However, we can only call execute when the line has finished drawing because that will get added to the history as a single entity. Since this is an image editor we need real time updates. With this in mind, i added start, continue, and stop as optional methods that can be used to provide realtime updates to the canvas, while still giving us the option to undo/redo later.

In the case of drawing:

• we create a new command and trigger start when a user presses their mouse down. We set up some initial textures and snapshot the initial state.
• When the user moves their mouse we trigger continue. here we are interpolating between x/y points (to ensure a smooth line), drawing to the texture we set up before and rendering it.
• When the user releases their mouse button, we call stop. In stop we perform some final clean up and reset the textures. We reuse the textures for performance reasons, so this clean up is important.

After realtime drawing execute is basically a no-op (it always get called when we pass it to the command manager). However if no realtime drawing has been completed then it will draw the full line.

The reason it is important that the command manager is able to handle these realtime updates is because all drawing operations then just live in this one ovject which makes maintenance much easier. Since everything is passed into these commands as arguments, unit testing is also straight forward.

The command pattern is actually really nice for this use case and allows us to easily add complex features without drowning in a sea of complexity. It also unlock other possible features in the future (really fancy ones like macros/ batch processing).

jsdoc

I have made extensive use of jsdoc annotations throughout the code in the hope that it offers a little help for us all when navigating this component in the future.

This is a practice that I think we should experiment with across the rest of the code base (frontend). I personally find it very helpful, both when writing and reading code because it makes me think about my interfaces + what they are actually doing.

Closes #5645 Closes #5206 Closes #5152 Closes #5055 Closes #4931 Closes #4907 Closes #4842 Closes #4677 Closes #4653 Closes #4492 Closes #4413 Closes #4290 Closes #4252 Closes #4159 Closes #4120 Closes #4011 Closes #3810 Closes #3623 Closes #3535 Closes #3472 Closes #3305 Closes #3280 Closes #3138 Closes #3110 Closes #2649 Closes #2425 Closes #2314 Closes #1591 Closes #466

🎯 PRs Should Target Issues

Before your create a PR, please check to see if there is an existing issue for this change. If not, please create an issue before you create this PR, unless the fix is very small.

Not adhering to this guideline will result in the PR being closed.

Tests

1. PRs will only be merged if tests pass on CI. To run the tests locally, please set up your Gradio environment locally and run the tests: bash scripts/run_all_tests.sh

2. You may need to run the linters: bash scripts/format_backend.sh and bash scripts/format_frontend.sh

[gradio-pr-bot]

🪼 branch checks and previews

•	Name	Status	URL
	Spaces	ready!	Spaces preview
	Website	ready!	Website preview
	Storybook	ready!	Storybook preview
	Visual tests	all good!	Build review
:unicorn:	Changes	detected!	Details
:notebook:	Notebooks	matching!

---

Install Gradio from this PR

pip install https://gradio-builds.s3.amazonaws.com/0b5240eafd96b4138875c2c37a2a02cb54c16807/gradio-4.4.1-py3-none-any.whl

Install Gradio Python Client from this PR

pip install "gradio-client @ git+https://github.com/gradio-app/gradio@0b5240eafd96b4138875c2c37a2a02cb54c16807#subdirectory=client/python"

[gradio-pr-bot]

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
@gradio/app	minor
@gradio/atoms	minor
@gradio/icons	minor
@gradio/image	minor
@gradio/imageeditor	minor
@gradio/preview	minor
@gradio/statustracker	minor
@gradio/upload	minor
gradio	minor

With the following changelog entry.

⚠️ Warning invalid changelog entry.

Changelog entry must be either a paragraph or a paragraph followed by a list:

<type>: <description>

Or

<type>:<description>

- <change-one>
- <change-two>
- <change-three>

If you wish to add a more detailed description, please created a highlight entry instead.

⚠️ The changeset file for this pull request has been modified manually, so the changeset generation bot has been disabled. To go back into automatic mode, delete the changeset file.

#### Something isn't right?

- Maintainers can change the version label to modify the version bump. - If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can [update the changelog file directly](https://github.com/gradio-app/gradio/edit/image-editor/.changeset/few-tips-appear.md).

[abidlabs]

@pngwn I'm happy to review this, but it looks like the frontend code may have broken as a result of the v4 changes, and it can't build at the moment.

Reviewed the backend, looks very good. I'm excited about the Brush and Eraser classes, and will try to recreate some of the old demos (like the sketchpad demo) with this.

[pngwn]

Got a few v4 bugs to fix but I'll be picking this back up shortly and I'll take care of the issues and finish it off.

[pngwn]

hmm

[abidlabs]

Amazing work @pngwn! I just gave the image editor a quick spin and noticed a few potential bugs / UI suggestions:

1. When you're starting out, seeing all 7 buttons grouped together can be a bit overwhelming. I was thinking we could show only the 1st row of upload/webcam/paste buttons initially, and then show the others when you've provided an image. But I don't think that would work since you can sketch on a blank canvas without having provided any image. What if we moved the 2nd row of buttons to the side, in a vertical column underneath the "X" button. I feel like that might be a bit more intuitive than the current setup. (cc @hannahblair for your thoughts!)

2. Clicking the webcam icon opens a full-screen webcam capture. I assume not intentional?

3. The crop works beautifully. But clicking the rotate image icon has no effect.

4. Similarly, I can't get sketching or erasing working for me. Clicking the palette icon or the gray button has no effect and does not allow me to sketch on the base image

5. Running demo/image_editor/run.py and clicking the "Run" button does not do anything for me. I see this error in the console: Uncaught Error: Could not create blob

[pngwn]

1. I'd rather we didn't split the buttons up, you have to cover a lot of screen to get from one to the other, especially on larger screen sizes. Likewise on mobile having core controls in two different places will be irritating. It might be less visually overwhelming but it would be a worse UX. Buttons at the top are ones that are uses less frequently.

We could just start off with the brush selected, that way we have fewer buttons. We could even start off with nothing selected but again i think that is worse UX even though it might be OK visually. I don't really think users will be overwhelmed by 6 buttons though, most image editors have dozens on display at all times.

2. It is intentional but given your response, I'll tweak that.

3. I am going to remove rotate for now and implement later. Its complex (not the rotate, the UX).

4. I'll fix sketching / erasing. There is a bug atm (well 2). SO you have to crate a new layer and change the size of the brush for it to work. Should be sorted shortly.

5. Probably need to handle empty layers more gracefully. Will check that.

[pngwn]

2. is fixed
3. is fixed
4. should be fixed now.
5. is fixed

also:

• Can now draw with no image.
• can undo/ redo

[abidlabs]

Good stuff @pngwn! I was able to get it running locally just fine. Here are the issues that I noticed:

1. When you open the color palette, there doesn't seem to be a way to close it. Clicking outside the palette didn't close it:

2. Similarly, when choosing a custom color, I would have expected that clicking on the color (the purple here) would have closed the palette, but it did not:

3. The "X" icon to clear the image has no effect

4. I don't think the image data is being preprocessed correctly. Neither of these two demos work (no image output is shown when you click submit):

import gradio as gr

demo = gr.Interface(lambda x:x, gr.ImageEditor(), gr.ImageEditor())

if __name__ == "__main__":
    demo.launch()

import gradio as gr

demo = gr.Interface(lambda x:x["composite"], gr.ImageEditor(), gr.Image())

if __name__ == "__main__":
    demo.launch()

Some UI improvements would be nice (though we can do these in a separate PR afterwards):

• the brush radius selector

• not a huge fan of the two rows of buttons -- I think it creates too much unused white space at the bottom. I don't have a good alternative right now, perhaps @hannahblair or @gary149 do?

Cropping, brushing, layers, erasing, undo, and redo all work super well!

[pngwn]

@abidlabs I've fixed all of those issues now I think. Also fixed the change and input events.

The issue with interface wasn't the preprocessing but the static variant.

[aliabid94]

Incredible work. Playing around with this, the few points of friction I hit:

1. Like abidlabs said, closing the popups for the brush / color controls was a little annoying
2. Hadn't immediately realized that uploading an image resets the entire history. I was trying to upload a second image on another layer, and then realized that uploading sets that as the new source entirely. (not something we need to change, just what I encountered)
3. Once I've selected a color and brush size, it would be nice if my cursor became a circle with that size and color. Would make it a little easier to know what I'm about to draw.
4. Brush sizes show weird numbers:
5. Python doesn't work on 3.8 (think this may be what you added @abidlabs)

[pngwn]

Brush sizes show weird numbers

That's pretty hilarious. Will remove the default size swatches for now. Not sure how useful they are.

[pngwn]

I've tightened up the buttons a little now too. I have some ideas about keeping them one on line but will discuss first + try in another PR.

[hannahblair]

Everything works great!! Awesome work @pngwn

A few notes:

• Color Picker isn't completely mobile responsive from ~400px (this is albeit a relatively small mobile size)

• The download button in the output needs a background - has something changed with it? (I'll check if this is on main)

• For another PR, it'd be good to have an empty state before a file has been uploaded - e.g. the file upload box is shown on other components when they're empty

• Super tiny nit, I'd love a small bit of spacing around the eyedropper icon

  

The whole colour picker component is so nice!

[pngwn]

I think I broke the button background. Will check that.

I'll take a look at those other issues too. Dark mode looks a bit janky. Will fix.

[abidlabs]

Like the tightened buttons a lot. Noticed some small things:

1. It'd be good to know see visually what the boundaries of the canvas are for sketchpad inputs before you start sketching (perhaps a faint gray background on the areas that are outside of the canvas):

2. When you first see the ImageEditor, its not clear what you're supposed to do:

Can we put a "drop image here" text similar to gr.Image() when you're on the uploading step

3. After you upload an image, would it make sense for it to automatically go to the next step of cropping?

4. When you click on the paintbrush step, its not clear what the paintbrush color is before you start drawing. Can we make the color of the brush radius selector match the paintbrush color?

5. The ImageEditor doesn't have a progress indicator, e.g. try with:

import gradio as gr

demo = gr.Interface(lambda x:x, gr.ImageEditor(), gr.ImageEditor())

if __name__ == "__main__":
    demo.launch()

[abidlabs]

just fyi removed some console logs so that we could run the static checks without linter complaining

[hannahblair]

thanks for taking the time to add jsdocs btw!

[pngwn]

@abidlabs

1. Added background when there is no background image
2. Added some text when there is no bg + no history + we are in bg mode. We cant add the drop text because dropping doesn't work atm.
3. I think that could be annoying if you want to upload a new image or something. After you leave the upload option you can't re-upload without clearing. If a crop_size is set by the author it will automatically go into crop though.
4. Will look into this
5. Is this when the ImageEditor is an output?

[abidlabs]

For (5), yes when it is an output

[pngwn]

@abidlabs @aliabid94 @hannahblair

Thanks for the reviews!

I think i've addressed everything.

@hannahblair I added some edge collisions to fix the issues on mobile (although mobile needs looking at on its own) and I tweaked darkmode so the color picker looks much nicer. I've also fixed the icon button (i did break them).

@aliabid94 I've cleaned up the UX regarding controls and remove those brush size related issues. I've also added a 'pain' cursor that matches the brush size as a preview for both erasing and drawing. Everything should be working on 3.8 now too (thanks @abidlabs ).

@abidlabs I've added a message for the blank state to give some pointers. We can probably improve that in the future. I've also fixed the statustracker and added the color preview on the circle.

The main new thing is you can now pass 'crop constraints' via crop_size and this will force the crop to match that ratio on the frontend and be resized fully on the backend. You can pass either a fixed width and height or a string ratio "1:1".

[abidlabs]

Doing some testing right now. Generally everything is working great. Just very tiny nits that I've found so far:

1. I would have expected clicking the "Upload an image or select..." box to open the upload file dialog like it does for our other components.

2. When you use an ImageEditor as an input and trigger the event (e.g. click a submit button), it takes quite a while before anything actually happens.

E.g. running

import gradio as gr

gr.Interface(lambda x:x, "imageeditor", "imageeditor").launch()

I have to wait ~5 seconds before even just seeing the progress status tracker on the output component.

3. Pressing the "X" button does not bring back the "Upload an image or select..." message. It just shows an empty white box for me (not sure if this is intentional):

4. "Clearing" an ImageEditor does not seem to work. Running the same demo as above:

import gradio as gr

gr.Interface(lambda x:x, "imageeditor", "imageeditor").launch()

if you upload an image and then press the gray "Clear" button in the Interface, nothing happens.

5. If sources=[], then the placeholder text should be changed as to not mention uploading an image:

6. If sources=[], should we also remove the transforms by default? At the very least, we should set the initial state of the ImageEditor component to start on the sketch tool, not on the cropping tool, as that is confusing.

7. We may want to add a parameter in Brush to disable custom color selection. E.g. if you want to constrain users to only be able to draw a single-color mask.

8. On narrow screens, the color selection is cut off. At least we can increase its z index so that the color wheel is still visible.

[pngwn]

I'll take a look at 1,2,3,3 and 5.

6. I don't think we should remove transforms entirely but I'll change the default.

7. Should already be possible by setting color_mode in Brush to "fixed".

8. I thought I fixed this, will double check. It might need a resize to take effect.

[pngwn]

Okay, made a few changes.

I couldn't add click to upload yet due to how things are put together, we can do that later if it is really needed but feels off for an image editor imo.

• I've fixed up the 'empty' text so it is more appropriate depending what tools are shown
• Empty text now shows correctly when you clear the component
• Clearing works properly whether its done vis the x button or the Clear button
• Crop can now correctly be disabled.
• Eraser is only enabled if brush is enabled
• After uploading an image, crop will automatically be selected if a crop_size is set
• If sources=[] the draw tool will be selected.
• The status tracker now show on the uploading component (the editor) when we are preparing the data (uploading the files). Then after that is complete the normal process kicks off. We can improve the UX for this later but it should address the current lack of feedback (although 5 seconds seems a long time, i couldn't repro that).
• I've also cleaned up the python API a little. Previously it wasn't possible to disable the brush or eraser (because None cases get a default brush), so eraser and brush can now accept a boolean too.

[pngwn]

And we are done!

Thanks @abidlabs @hannahblair @freddyaboulton @aliabid94 for the help with various bits and pieces, the PRs, and the detailed reviews + tested. Very much appreciated!

[wamiq-reyaz]

Hey all,

thanks for the wonderful work. I am still wondering about a few things though

1. The brushes are not well documented. Like how does one create a custom swatch?
2. If one instead uses a colorpicker component, how does one update the brush color for an ImageEditor? The current events do NOT target the brush color.
3. The documentation for the ImageEditor does not actually showcase how the ImageEditor is actually used but instead has an unrelated rotation example