Support bulk editing

[fdw]

I haven't found a way to edit several documents at once (f.e. adding a document type or a tag to all of them). This would be very helpful, especially after importing or migrating a large number of documents.

[jonaswinkler]

I want this and I'm drafting the UI right now. I'm thinking about a Bulk-Edit dropdown button with all the operations listed, and popups for additional information / delete confirmations.

Bulk operations I can think of right now.

• Add/Remove Tags, Correspondents, Types
• Delete
• Re-assign metadata by matching rules
• Redo OCR (this will come in handy later, when OCRmyPDF is in the dev branch). This will allow users to embed OCR'ed text into existing documents directly from the front end instead of executing some management command.

I'll also need API support for this.

[jayme-github]

I want this and I'm drafting the UI right now. I'm thinking about a Bulk-Edit dropdown button with all the operations listed, and popups for additional information / delete confirmations.

Bulk operations I can think of right now.

* Add/Remove Tags, Correspondents, Types

What I did use a lot (had that patched into paperless) was a bulk action "Remove INBOX tag" .

[fdw]

I'm thinking about a Bulk-Edit dropdown button with all the operations listed, and popups for additional information / delete confirmations.

That sounds very good to me. Only thing that might be obvious to you, but not to me, is that I'd add some way to select the affected documents (for example, checkboxes).

Bulk operations I can think of right now.

• Add/Remove Tags, Correspondents, Types
• Delete
• Re-assign metadata by matching rules
• Redo OCR (this will come in handy later, when OCRmyPDF is in the dev branch). This will allow users to embed OCR'ed text into existing documents directly from the front end instead of executing some management command.

That should at the least cover the basics. I can't think of anything else yet, anyway ;)

I'll also need API support for this.

That's the thing I'm curious about as well. The trivial way would be to have one PATCH call per document, but I think bulk might be nicer. Would be a non-standard use of PATCH though, if it contains not just the changes, but also the targets. Do you have an idea already?

[jonaswinkler]

I'm thinking about a Bulk-Edit dropdown button with all the operations listed, and popups for additional information / delete confirmations.

That sounds very good to me. Only thing that might be obvious to you, but not to me, is that I'd add some way to select the affected documents (for example, checkboxes).

Something like this.

Checkboxes, of course. I'd show them in the top left corner of the preview image and have them visible only on hover / when checked to keep the list clean.

Edit: I think slightly altering the color of selected items would be super nice.

Bulk operations I can think of right now.

• Add/Remove Tags, Correspondents, Types
• Delete
• Re-assign metadata by matching rules
• Redo OCR (this will come in handy later, when OCRmyPDF is in the dev branch). This will allow users to embed OCR'ed text into existing documents directly from the front end instead of executing some management command.

That should at the least cover the basics. I can't think of anything else yet, anyway ;)

I'll also need API support for this.

That's the thing I'm curious about as well. The trivial way would be to have one PATCH call per document, but I think bulk might be nicer. Would be a non-standard use of PATCH though, if it contains not just the changes, but also the targets. Do you have an idea already?

Some operations aren't simple PATCH calls, since they'd fire some process in the back end instead of assigning new model properties, so I'm thinking about a new end point /api/documents/bulkedit/ that accepts POST requests. Each request would specify a set of document id's, a task, and optional parameters. The easy tasks will block, longer tasks will use the task queue.

[jonaswinkler]

This is now mostly working as pictured above. The only thing left to figure out is how to display check boxes on the small and large cards.

[jonaswinkler]

Oh, and I need to write some test cases.

[shamoon]

How’s this going UI-wise? Can I help? 🙂

[jonaswinkler]

Check out the feature-bulk-edit branch.

Pretty much the only thing left to do is the check boxes on the cards. No check boxes on the search results though (also uses large cards).

I've done that on the small cards already, but I'm not entirely happy.

• Not sure about selection by clicking on the preview image.
• The check boxes themselves are rather small and hard to hit.

I'd be happy for some help.

[shamoon]

• I think selecting by clicking the content preview is great, makes a big target area.
• I think thats ok with preview being clickable for cards and easy / obvious access to a select all button.

With that second thing in mind, here's what I was thinking (see gif attached). This UI slides out once an item(s) is selected and allows for more quickly doing things than the dropdown. In playing with it works pretty good—let me know what you think!

[jonaswinkler]

Here are my thoughts:

• I might add some more, less used operations in the future. They don't exactly need to be this accessible. Also, I think that these editing tools aren't going to be used very frequently, so having an additional click to get to them isn't so bad.
• I don't like the UI jumping around like this. Say you've accidentally selected some item, want to hit the checkbox on the top left, but that's now where the edit button of the document above is, roughly.
• I think it's too stacked, to many buttons. Reminds me of those days when everyone wanted to install some toolbar in your browser :)

[shamoon]

Of course, youre the boss! I will say:

• I agree its a lot of buttons, a lot of stacked. Im not totally sold on the buttons visually though I grew to like them but I do like the slide down, it sort of tells you "Im in bulk mode" or something.
• But! I actually think these may get more use than you realize (obviously there are a lot of different people out there but I have already found myself needing them) and the thing about using a slide down is they're totally gone when you dont need them.
• The sliding down pattern for bulk edit is a UI pattern I've seen before, I agree things moving can be annoying but in this case theres no real 'penalty' other than perhaps you have to move your mouse to un-select the item if you had no intention of doing bulk editing, I dont think theres a real risk of accidentally performing an action or something.

Anyway, again of course your call

[totti4ever]

Hey, I am really excited what you guys are performing here I have to say!

Reading this discussion, I'd just like to add my view in case it's of interest: I actually see your points, Jonas. But I am on Michael's side when it comes to the frequency of use, as especially new users will need them a lot to structure their documents. And also once that has been done, in the second you add another category or so, the bulk mode heavily helps again! This means, I like the instant availability

keep up this great effort, thanks a lot!

[shamoon]

Yea I know people have their own workflows. Attached is a screenshot of a smaller version (that's still pretty usable IMHO) if that makes any difference.

[jonaswinkler]

Is this working? Can I try that somewhere?

[shamoon]

Yes there’s a branch on my fork: https://github.com/shamoon/paperless-ng/tree/feature-bulk-editor (I don’t have a backend setup with the API version that includes bulk editing but the new UI component uses the same logic as your menu did so should be fine)

[jonaswinkler]

Alright, I see where this is useful now. Much of the editing work is just setting types tags and correspondents and that's where all the use of this feature will come from. I can see how having quick access to that will be useful.

I'm just not entirely sold on the layout yet. It's somewhat annoying on the list view when you try to select multiple items, the check boxes always move after selecting the first. Just had a look at how github does it. They replace much of the filter options with mass manipulation menus. Could we do the same?

• When a selection is active, replace the quick filter buttons with "Set correspondent" etc drop downs. That drop down could also have an option to unset the correspondent. same for types, and similar for tags.
• The title filter can stay.
• selection buttons to the left.

What do you think?

[shamoon]

Thats an interesting approach, I think it could work. In an effort to coordinate 😃 how about I take a crack at it and let you know how it goes?

[jonaswinkler]

Sure thing!

The idea behind that is that filtering does not work all that well together with mass editing anyway. Filtering removes the selection from filtered items. Don't want to modify invisible items. Therefore, I don't see any immediate concerns with disabling the filters when the user starts to select things.

If that works out usability wise, I'd probably move the bulk edit functions into the filter editor as well, hook that up with the document list service to get the current selection, and maybe rename that component to something that describes the combined functionality better.

I've also considered disabling the confirmation dialogs when only very few items are selected (< 5, or something, except for delete, of course). It would be ideal if we could use the same drop downs for selecting correspondents / types / tags that we also use for filtering. That way, we need to (select) -> (click the drop down) -> (click the thing) -> (optionally confirm for large selections) and that's it.

These are just some ideas that just came to mind.

[shamoon]

I'll have a think. I will keep it DRY by reusing the smaller components (and making them common) but I was thinking to just use a separate component, a BulkEditor vs FitlerEditor, I think it will end up cleaner.

[jonaswinkler]

Actually, yes.

You might want to pull dev before making too many changes, some changes to the large cards.

FitlerEditor, again. :)

[shamoon]

Phew, yea I did 1h ago but Im moving things around so hopefully no more big changes until I have something to show you...

[jonaswinkler]

Seems I forgot to merge that into the feature-bulk-edit branch, which would have been important. Anyway, I'll leave it as it is.

[shamoon]

My branch is based off that one so should be OK.

So Im pretty happy with how this is working, needs more testing but should be close. A couple things wanted to discuss:

• I implemented a mechanism where once documents are selected the menu items either get a ✓ if every item in the list or – if only some do. Then the user can toggle those either off / on (so – becomes ✓, ✓ becomes __, etc). This works well for correspondents & doc types and for tags this allows you to set multiple (e.g. check off a few), but I think the API currently does not allow setting multiple tags in bulk. Is that something you can look at? Maybe it will be easier if I complete the UI stuff before you do that.
• I dont actually have a backend running with bulk-editing (I just build the UI and connect it to a docker instance) but I didn't change the code you had for communicating with the backend (except tags TBD) so I think it should be OK.

Let me know about tags and hopefully will have something to look at soon!

[jonaswinkler]

I can't quite follow.

• I implemented a mechanism where once documents are selected the menu items either get a ✓ if every item in the list or – if only some do.

Which menu?

Do you mean:

• A correspondent gets a ✓ if every document with that correspondent is selected? And – if only some?
• This does not work, since you'll only ever have a fraction of all documents available (paging). This works when you have a handful of documents available and set the page size to something greater than that, but it does not work with thousands of documents. We also can't just download the entire document list, since that's slow. We also should not cache that, since updates can happen when consuming files.

Or

• A correspondent gets a ✓ if every selected item has that correspondent? And - if only some?
• Same as above, as soon as you switch pages, the documents (and thus the information about what correspondents and tags these documents have / had) from the previous page is gone and you just don't know anything about these documents anymore.

Or did you take that into consideration?

Regarding multiple tags: I can add that to the backend.

[shamoon]

Sorry my bad, its hard to explain without a photo or something. The ✓/– just refers to the selected documents. Please see the short video below to illustrate (essentially works the same for correspondents / doc types just not multiple).

• First 1 document is selected so the tags menu has a ✓ next to both tags because all items in the selected set have those tags.
• Then when another document is selected the tags menu shows a ✓ next to Inbox since they both have it but a – next to Auto and Test1. Closing the menu (which "applies" the changes i.e. triggers the modal confirmation) will only apply items with a ✓ (Test2).
• Removing all checked items is how you would remove all tags / correspondents.
• This works pretty well IMHO and is pretty compact & efficient. I think the confirmation dialog is a good idea to keep. If this looks good only a couple things remain to do (on my end)!

[jonaswinkler]

Looks pretty good! I have two concerns.

• When determining whether to show ✓, –, or nothing, did you take into consideration that the document list service only remembers the currently displayed documents? You don't know which tags/correspondents/types documents on other pages have, and these can be in the current selection.
• I don't like the action on closing the menu at all, that feels weird, mainly due to the confirmation dialog.

[shamoon]

Cool 😎

Regarding the paging issue I think the most obvious solution is to make it that when there are items selected on a different page we just hide all – / ✓ and it becomes "✓ something to apply". Only other way I can think of would be to build something additional into DocumentListViewService (probably would need to come from the API) that when given a set of selected documents returns the used tags and a count of documents within the set with that tag i.e. [{tag: "Inbox", count: 275}, {tag: "Auto", count: 121}], that would be enough to do – / ✓. But not sure if that would work performance-wise?

Im on the fence about the action on closing. At first I thought was weird but I actually think it works, especially because of the confirm dialog. It wouldn't make sense to do on clicking each item because for tags you want to wait until the user is done checking / un-checking things. Alternatives I can see would be:

1. Add a little "Apply" button at the bottom of the dropdown. In this case I would still vote we leave it to apply on close as well.
2. We could have one big "Apply" button in the toolbar. BTW, GitHub labels work like no. 1 without the confirmation dialog but I didn't like that because I really didn't even know it was going to do something. I think at the moment I prefer 1.

[shamoon]

Ok! I'm pretty happy with this. In practice I think just 'turning off' the – / ✓ when "All" is chosen is a good solution. I added:

• Updated layout (screenshot attached)
• Added small Apply button to bottom of dropdowns
• Hide – / ✓ when bulk editing selection spans across pages
• Only apply actions if something changed (this is also part of why I think leaving "apply on close" is OK)
• The count badges now update live when bulk editing

Only remaining question is about the UI for applying changes, let me know your thoughts. Or I can just create a PR and you can test it out and see how you feel?

[jonaswinkler]

Please make a PR.