Fix async communication between dane-asr-worker and dane-kaldi-nl-api

beeldengeluid / dane-asr-worker

DANE worker for processing ASR (optimised for Dutch)

GNU General Public License v3.0

0 stars 0 forks source link

Fix async communication between dane-asr-worker and dane-kaldi-nl-api #67

Closed jblom closed 2 years ago

jblom commented 2 years ago

Make sure the async mode implemented in the dane-asr-worker works. This to avoid timeouts

jblom commented 2 years ago

related PR (far from finished): https://github.com/beeldengeluid/dane-asr-worker/pull/68/files

jblom commented 2 years ago

Update the code in the PR has been merged and tested successfully in the following ways:

use the LOCAL_KALDI option in the asr-regtest
use the ASR_API option on the dit-is-de-dag workflow
use the LOCAL_KALDI option on the dit-is-de-dag workflow

Now the final and most important test is how it will perform on the long items from the "high prio" workflow (i.e. use-case-100-high-prio-items workflow, which use to fail right at the first item)

Shortly, I will rerun the first item of batch use-case-100-high-prio-items_0 and hopefully we'll see it succeed soon

jblom commented 2 years ago

Update I retried one of the items in the use-case-100-high-prio-items_0 batch, left it over night, but it seems to be still running. I noticed that the stdout of Kaldi is not logged (debug level, running code is configured to show info).

Now will redeploy with proper logging and try (another item) again. For this run I did not notice any zombie processes or dmesg errors, so that seems fine. Without Kaldi logging I really have no idea why it is running so long.

jblom commented 2 years ago

Update Ah I figured it out. The reason it fails is this rabbitmq exception occurring (already fixed by @gb-beng in the new rabbitMQ setup):

      (406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. 
      Timeout value used: 1800000 ms. 
      This timeout value can be configured, see consumers doc guide to learn more')

This exception causes the worker to stop/crash and k8s will restart it again after which the same ASR job is started again (now in the same container)

Will configure the proper timeout in the old cluster and retry

jblom commented 2 years ago

Update applied the same consumer_timeout setting used in the new k8s cluster via a new configmap for the rabbitmq pod

Now we wait to see if the same item does finish in time

jblom commented 2 years ago

the new consumer_timeout setting worked fine. This DANE document (with a 4 hour audio file) was processed successfully:

DANE doc ID = da831ecf70383a868ce46de3766bff0e5543868b
FILE = 2102203260336511231__2022032602RA1-RCR3000MVDF.mp3

Now to test the single export of this item, so that it is also available in the MS:

https://mediasuite-test.rdlabs.beeldengeluid.nl/tool/resource-viewer?id=2102203260336511231&cid=daan-catalogue-aggr&st=2102203260336511231

jblom commented 2 years ago

After manually triggering the ASR for the first batch (proc_batch_id = use-case-100-high-prio-items_0) in the high prio workflow I started the workflow again and the export went fine. In this program you can find the first two carriers of that first batch: https://mediasuite-test.rdlabs.beeldengeluid.nl/tool/resource-viewer?id=2102203260336511231&cid=daan-catalogue-aggr&st=2102203260336511231

Closing