Please implement queuing on the web interface

[wypiki]

Please implement queuing so it would show that another request is being processed and start as soon as this is finished when it is busy processing another request.

[Ananderz]

I think what would be even better is not to lock the write section with "processing query" but to add a cancel button and also if you try to prompt while it's prompting something else it's put into a queue

[wypiki]

In the meantime I've implemented a worker_status endpoint in ui.py which is queried by a corresponding JavaScript function which checks every second and disables the text field as soon as the worker is running for everyone, also shows the status in the title bar so people see when it's available. But it's far from perfect...

[drvenabili]

In the meantime I've implemented a worker_status endpoint in ui.py which is queried by a corresponding JavaScript function which checks every second and disables the text field as soon as the worker is running for everyone, also shows the status in the title bar so people see when it's available. But it's far from perfect...

@wypiki Would you be willing to share the code? Thanks

[wypiki]

In ui.py before def ui I added:

# A flag indicating whether the worker is busy.
worker_busy = False

Then within def(worker(... at the beginning I added:

global worker_busy
worker_busy = True

At the end of this part I added: worker_busy = False

Then within def ui I added:

    @app.get("/worker_status")
    async def worker_status():
        # Return the status of the worker.
        return {"busy": worker_busy}

In the index.html before </script> I added:

  // Check the status of the worker every second and disable the submit button if the worker is busy.
  function checkWorkerStatus() {
Promise.race([
    fetch('/worker_status'),
    new Promise((_, reject) =>
      setTimeout(() => reject(new Error('timeout')), 1000)  // 1 second timeout
    )
])
.then(response => response.json())
.then(data => {
    if (data.busy) {
      form.elements.query.style.color = 'red';
      form.elements.submit.disabled = true;
      if (form.elements.query.value === 'Processing your query. Please wait...') {
        document.title = '\u{1F7E0} answering: ChatDocs';
      } else {
        document.title = '\u{1F534} busy: ChatDocs';
      }
    } else {
      form.elements.submit.disabled = false;  // Only enable the submit button if the worker is not busy
      form.elements.query.style.color = 'white';  // Reset the color to the default
      document.title = '\u{1F7E2} available: ChatDocs';
    }
})
.catch(error => {
    console.error(error);
    form.elements.query.style.color = 'red';
    form.elements.submit.disabled = true;
    if (form.elements.query.value === 'Processing your query. Please wait...') {
      document.title = '\u{1F7E0} answering: ChatDocs';
    } else {
      document.title = '\u{1F534} busy: ChatDocs';
    }
})
.finally(() => {
    // Schedule the next check regardless of whether the fetch operation was successful or not.
    setTimeout(checkWorkerStatus, 1000);
});
}
  // Start checking the worker status as soon as the script runs.
  document.addEventListener('DOMContentLoaded', checkWorkerStatus);

This updates a red, green or yellow point in the title bar every second, so people now it if it runs in the background also.

During the processing of the prompt the webserver doesn't react to other requests, so that's why there is a short timeout which also triggers the busy state.

But as I said, it's not perfect. If someone closes the page while it is processing the prompt, the answer might get delivered to another user. It's the quickest change I could do without changing the architecture.

[drvenabili]

Many thanks!

[Ananderz]

@wypiki would this also give you the ability to run multiple instances. Let's say I prompt in one webbrowser and run it on another webbrowser and try to prompt there also (a que is present)

[Ananderz]

I couldn't understand the instructions fully mind sharing the ui.py and index.html ?

[Ananderz]

I figured it out! It works great!

I have been able to make it crash when I loaded up two prompt windows with content and more or less hit enter at the exact same time. I wonder if we can do something to fix that. What do you think @wypiki

[wypiki]

@wypiki would this also give you the ability to run multiple instances. Let's say I prompt in one webbrowser and run it on another webbrowser and try to prompt there also (a que is present)

No, there's no queue, it just blocks other requests for this time and shows a colored indicator in the titlebar so people know when it's ready. A queue would be a bigger change, didn't want to spend too much time with that yet in hopes @marella implements it in the future.

[wypiki]

I couldn't understand the instructions fully mind sharing the ui.py and index.html ?

Sorry, I'm still on vacation.

I figured it out! It works great!

I have been able to make it crash when I loaded up two prompt windows with content and more or less hit enter at the exact same time. I wonder if we can do something to fix that. What do you think @wypiki

Glad, you figured it out! Yes, that's a weakpoint of my implementation. It's unlikely that this happens though. The easiest but not most efficient change would be lowering the timeout in promise.race():

setTimeout(() => reject(new Error('timeout')), 80)

This would change the state to busy when the webserver is not replying because of being busy to 80 milliseconds instead of 1 second. In a local network with a fast cpu that would probably be fine. Over VPN in combination with a maxed out CPU this could lead to unwanted busy states. But usually responds should come in 5-30ms, so it should be fine.