search

moinwiki / moin

MoinMoin Wiki Development (2.0+), unstable, for production please use 1.9.x.
https://moinmo.in/
Other
307 stars 92 forks source link

Add celery as task manager? #1000

Open moloney opened 4 years ago

moloney commented 4 years ago

I am working on an issue right now, where certain PDF files can take 30+ minutes to be converted to text by pdfminer3. There are some improvements I am working on to help this issue, but for quality text extraction there will always be the potential it takes at least a couple minutes.

This got me thinking that the conversion should really be run as a background task, rather than just timing out on the http response. Using the celery package seems like a good fit for moin since it can use the filesystem as a message broker (doesn't require a redis instance).

Any thoughts?

RogerHaase commented 4 years ago

Current version does not support windows, but there may be a hack: https://www.distributedpython.com/2018/08/21/celery-4-windows/

Another use could be for email notifications when an item is saved.

ThomasWaldmann commented 4 years ago

When adding celery, do it in a way so it is optional to use it (use it only if there is configuration for it).

moloney commented 4 years ago

Sorry, I didn't think about windows support. An alternative could be something like huey, although it looks like the multiprocess worker option won't work in Windows. I guess being stuck with the multithreaded worker in Windows is still better than the current situation.

@ThomasWaldmann - if there is a pure python option that supports all platforms, would you be okay making it a hard dependency?