Batch edit items - why not long running job?

[luku]

I'm facing some difficulties with batch edit of more than a thousand items at once:

1. standard PHP settings for max_input_vars is 1000 - that's my responsibility to change
2. Omeka is running the batch edit job in default mode, so it's timing out - and here I'm not really sure what is best...

I can see advantage of running the job in default mode, because I can throw and display exceptions to users and I don't need any page for showing the job status. But at the same time it's wrong to allow batch edit of few thousands items if the job cannot handle it. So I would like to get your feedback on this before creating some pull request. Do you think it's good idea to run the job in background? And if so, do we need some status page for job progress, or do you think silent processing with errors in log only is enough?

Also I think there are several ways for dispatching the job:

1. change the method for running job in ItemsController to $dispatcher->sendLongRunning(
2. add new option for it, i.e. items_batch_edit_job_longrun with bool value and decide based on that
3. decide how the job should run based on number of submitted items. For example =< 1000 = default and > 1000 = long running

Which one do you think is better, or other ideas??

As for status page, if it should be done, I have no idea where to put it... So any advice is appreciated. Thanks!

[zerocrates]

You're accomplishing this by just setting the page limit really high?

The initial plan, as far as I remember it, with the batch edit was to have two modes basically: one that operated just on the stuff visible on a single page, and another that would operate off a query. Obviously the single page version is what's in Omeka today, and the theory there was that the scale would be low enough that a long-running job wasn't required. The batch edit also dates to a time when we had serious problems with users running on hosts incompatible with background jobs, so we didn't want to tie a core feature to having background support.

As for which of your options would work best, I'd have to think about that a little more... I don't think changing the job to always be long-running is a good idea, as the vast majority of users will be doing only small batches that should fit easily. The "query-based" batch edit that never came to be would I guess be more suited as a plugin at this point (and to some extent I believe is fulfilled by UC Santa Cruz's Bulk Metadata Editor plugin).

[luku]

Thanks for the reply! Yes, it's possible to achieve very high items/page by per_page query param, or in settings. I agree the majority of batch edits are about smaller number of items, but the changes needed are rather simple. That's why Bulk Metadata Editor plugin feels like an overkill, where we would need to hide or disable too powerfull/harmfull features from users:) But I also understand that running background script is not always possible, so maybe it's really something that plugin should do... But please let me know if you come up with alternative that would fit Omeka core.

[luku]

Adding real life scenario (for better motivation): Imagine you import few thousands items. After import you do some random changes. Now you visit your public page and see it's empty. Well, it's because all those thousands of objects were imported as private. What can I do? I did some changes in the imported items, so I don't want/cannot undo import. I just need to change them all to public. (Bulk Metadata Editor plugin cannot help with this). So you change your settings for items/page in admin to 900 (to make sure it's working) and change all private items to public in let's say 30 batch edit jobs. Then you visit your public page again. OH, NO! Why are not those items in my collection? Did I forget it during import??.... I think there's no need to continue, the frustration could affect good mood here:) It's easy to do this in DB directly, but it would be big help if users could do this themselves in admin UI.

[zerocrates]

I totally get the motivations; they're legitimate and reasonable. Still, my ultimate preference would be for something that just operates on the active query, like some alternate button, so instead of applying the batch edit to the checked items, apply it to all X results you're looking at, even ones on other pages. That action would make sense to always be a long-running job, and it would also work around the max_input_vars issue, as we wouldn't be showing or processing thousands of checkboxes.

As for your less-radical alternatives, I'm still not sure which is best, if any. Note though, that for your own purposes, you can make "short-runnning" jobs run in the background as well, by changing the short running dispatcher in the config.ini.

[luku]

Thank you very much! I completely missed config.ini directive for this. It should be possible to change jobs.dispatcher.default on fly, so I could apply it just for the items batch edit. It'd still tell nothing to users about possible errors, but it'd never timeout. Good :+1: I also like the option of doing batch job for either selected/checked items or all (on all found results) if none item is checked or with new button. Status page for background job is the only missing part then, and I still don't see any fitting place for it... Anyway, since there is now clean way of changing how the batch edit is executed, I'm happy with it. If there is no better solution that would deserve pull-request, I have nothing against closing this issue as solved. Thanks again!

[Daniel-KM]

Hi,

I just added a button to batch all browsed items via a background process (see #694).

Sincerely,

Daniel Berthereau Infodoc & Knowledge management