Download tasks don't seem to run simultaneously, causing delays

[deldesir]

Issue/Problem

The current task processing mechanism in Calibre-Web relies on apscheduler for managing various tasks, including format conversions, uploads, and metadata-related activities. It has been recently optimized to support downloads (see #9). While apscheduler is functional, there have been observations that tasks might not be processed concurrently in an asynchronous manner, causing potential delays in download task execution.

Proposed Solution

To enhance the processing of tasks within Calibre-Web, I propose either leveraging apsheduler.AsyncScheduler (https://apscheduler.readthedocs.io/en/master/api.html#apscheduler.AsyncScheduler) or integrating Redis Queue (RQ) as an alternative job queuing system (https://python-rq.org/).

[holta]

• How serious are delays?

• Can you give a few specific examples with details? (Whether or not there might appear to be patterns?)

Thanks @deldesir for explaining when you can!

Related:

• 9

• PR #10

• 12

• PR #18

• 20

• 29

• 58

[deldesir]

• How serious are delays?

Well, that depends. The delays happen when users try to download multiple things at once. Even though downloads are in separate threads, other tasks get lined up in a queue, so users have to wait for one task to finish before the next starts.

• Can you give a few specific examples with details? (Whether or not there might appear to be patterns?)
• Steps to Reproduce:
• Initiate the download of several videos simultaneously.
• Observe the "Tasks" page in Calibre-Web.

• Observations:
  • The first video download task starts promptly and progresses as expected.
  • Other tasks have status "waiting" while the first one is running, so they go one after the other, not at the same time.
  • Users experience delays as they await the sequential execution of download tasks.

[holta]

Excellent summary to help us all keep track, Thanks @deldesir! Annoying-but-tolerable for now.

Two much-higher-priorities in December 2023:

• Reliable download of "ALL" videos (to IIAB) + Reliable viewing/listening on "ALL" student devices) which likely means:
  • 1-to-2 Mbit/sec codecs or whatever's "Just Good Enough"(TM) to preserve the readability of all blackboard/whiteboard writing.
  • Avoid all HD and UltraHD codecs (unless absolutely necessary, to guarantee "ALL" needed Videos/Audio are available?)
  • DON'T require VLC Media Player on student devices (though we can mention it if absolutely necessary, e.g. for obscure corner cases like Clickable Video Chapters ?)
• Video/Audio/Book thumbnails a.k.a. posters/covers (that are not excessively distorted, even if initially imperfect!)

DESIGN PRINCIPLE 1: Honor Low-Income Teachers/Students with "Near-Zero Bureaucracy"(TM) deep learning experiences 📐🧭 🖼️

DESIGN PRINCIPLE 2: Ignore Geeky Corner Cases — keep focus on above — remove features to welcome the masses authentically!

[holta]

Tangentially related question is how educators/parents/curriculum designers can monitor their ongoing "Download to IIAB" downloads' progress:

• chapmanjacobd/library#26