Guidance for getting eoranged/rq-dashboard type functionality

[seandalaiocht]

I've built a fairly substantial flask application which uses arq for all of its background processing. I'm pretty happy with the solution but see one deficiency that I'd like to try to address. I'd love to have some clean way to integrate visibility/monitoring of the queues that my application is leveraging.

Since rq-dashboard was designed to monitor http://python-rq.org/ queues and since you have been an significant contributor to RQ, I hoped you might offer some pointers for how I could integrate such a web front-end to monitor to arq queues, jobs, and workers. I'm looking to empower my application's users with greater insights into what is happening behind the scenes.

I'm looking to know whether forking rq-dashboard and reworking it might be feasible for me to pursue, and I'm hoping for a little guidance.

Separately, I suppose I'm also pushing for arq to offer console based "rq info" type capabilities similar to what is described at http://python-rq.org/docs/monitoring/

[samuelcolvin]

Great to hear you're using arq.

Have you noticed about the complete rewrite in v0.16 which is not yet released? If not, hope that doesn't cause you too many problems.

I haven't looked into rq-dashboard, but my guess would be that it would be way more work get it to work with arq than to start from scratch, especially since v0.16 uses sorted sets not lists as it's main primitive.

Regarding arq info, I'd be very happy to include something like this. What would you like it to include? Would you consider submitting a PR?

Regarding monitoring more generally I have the heartbeat functionality which should print the most important information about queues and jobs to the console regularly.

For me there are 3 classes of monitoring (you don't necessarily need to use all of them):

• error reporting and alerts, for me this is always sentry, but you could setup another logging handler to do other things. This is well supported already I think, although you could regularly check queues haven't got too long/slow and log a warning if so
• real time "what's going on at the moment" (which I think is what rq-dashboard is about), it seems cool but in practice I don't use this stuff that much myself, beyond glancing at logs every now and again. It's no use to investigate errors since it's always too late.
• grafana style trend analysis, this can be very useful, but recording enough data for it is definitely outside the bounds of arq (and redis ultimate isn't a very good timeseries database, although it would probably do fine in many scenarios). I built the heartbeat functionality to allow this data to get sent to grafana/telegraf but never completely the work, mostly because arq "just works" :crossed_fingers: and I haven't need it.

[seandalaiocht]

Thank you for your detail response. I've been using the v0.16 software.

I am open to working on and contributing much of the arq info type functionality that I was requesting. I believe that I can dedicate a bit of time to this effort in a couple of weeks while I am away from my office attending PyCon 2019.

[leosussan]

Adding +1 to the interest in a dashboard offshoot project.

[teruguparthasaradhi]

@seandalaiocht Hi, Even I'm trying to build flask with arq for job queuing. And I'm facing issue with having both flask and arq. can you share the code if possible? even a simple flask with arq integrated is fine. Thanks

[samuelcolvin]

arq is asynchronous, flask is synchronous.

Unless you know exactly what you're doing, you're going to run into trouble.

Best to either:

• use flask with a synchronous queueing library like rq
• or use arq with an asynchronous web framework like starlette or aiohttp

[teruguparthasaradhi]

Thank you @samuelcolvin Currently we are using celery worker to queue all the jobs. we would like to replace it with arq. can you suggest how to achieve this ?

[samuelcolvin]

Read the docs...?

[teruguparthasaradhi]

I have gone through the docs and tried some examples. I don't have any experience in asynchronous in python. Recently I went through some articles and videos stuff. For more info I would post my code which I was trying to implement. This is a sample POC I was trying to make using flask. This may help you to understand what I'm trying to do here.

#flask_arq.py

from flask import Flask,jsonify,request
import json

import asyncio
from demo_arq import arq_task

app=Flask(__name__)

async def new_arq_task(notification):
    await arq_task(notification)

@app.route('/arq',methods=['POST'])
def generic():
    try:
        notification = request.get_json() or json.loads(request.get_data(as_text=True)) or request.form
        print("notification in generic: %s",str(notification))
        a=asyncio.run(new_arq_task(notification))
    except Exception as e:
        print(str(e))
        return 500
    return jsonify(status="ok"), 202

if __name__ == "__main__":
    app.run(host='0.0.0.0',port=5243) 

#demo_arq.py

`import asyncio
from arq import create_pool
from arq.connections import RedisSettings

async def arqdemo(ctx,notification):
    print("CTX:",ctx)
    print(notification)
    await asyncio.sleep(10)
    print("After Sleep")

async def startup(ctx):
    print("Starting the arq.....")

async def shutdown(ctx):
    print("Stopping the arq.....")

async def main(notification):
    print("Inside Main task ","%" * 40 )
    redis=await create_pool(RedisSettings(host='10.236.220.226',port=6379))
    await redis.enqueue_job('arqdemo',notification)

class WorkerSettings:
    redis_settings = RedisSettings(host='10.236.220.226',port=6379)
    functions=[arqdemo]
    on_startup=startup
    on_shutdown=shutdown

async def arq_task(notification):
    print("Inside arq Task ", "*" * 40)
    await main(notification)

[samuelcolvin]

This is not the right place to teach you about async python, there are many other resources for that.

Also, if you want help on github, please take the time to format your comments or questions so they're easily read

[teruguparthasaradhi]

Thank you for the feedback @samuelcolvin I also tried to format my code it was not happening as I was not doing it in the right way. Today I learnt how to do it 👍 .

[SlavaSkvortsov]

I did a small dashboard project for arq. It is based on Django coz it is the easiest way to have an admin dashboard. You can find it here https://github.com/SlavaSkvortsov/arq-django-admin

It is more like real-time "what's going on at the moment". It is relevant for the project I'm working on. For error reporting/alerts we use Sentry, for trend analysis Newrelic