search

LibreQoE / LibreQoS

A Quality of Experience and Smart Queue Management system for ISPs. Leverage CAKE to improve network responsiveness, enforce bandwidth plans, and reduce bufferbloat.
https://libreqos.io/
GNU General Public License v2.0
444 stars 48 forks source link

Can we use a database like mysql instead of a text file ? #65

Open marsalans opened 2 years ago

marsalans commented 2 years ago

I think it would be good to utilize a mysql database instead of a text file, it would be helpful to show statistics or to manipulate data easily. It would be great for in future if we are creating a web based management system for this to integrate with other vendors.

as @thebracket have created a web application for management and statistics viewing.

rchac commented 2 years ago

I mean right now with v1.1+ everything is fed into InfluxDB, which stores that data over time in a super efficient way. The only large csv file currently would probably be ShapedDevices.csv if you have a lot of subscribers - but the nice thing about having it as a CSV is that you can more easily manipulate that file manually as a CSV compared to a database. Are we sure a database would be helpful?

marsalans commented 2 years ago

I just have a thaught on that, by thy way utilizing csv file wouldnt be a hassle.

rchac commented 2 years ago

Gotcha. I like the idea though, it's definitely more standard to use a DB. If we get some more feedback here in the thread - lets definitely consider that direction. Thank you for contributing ways to improve the code.

thebracket commented 2 years ago

If it's any help, I replaced the csv with a serialized (Ron, could easily be JSON or any other structured format). It preserves the hierarchy more naturally than sql or a flat format - and is very easy to generate, read for debugging or send elsewhere.

I did that because I run a "generator" that periodically updates from uisp (with flat, ap only or deep hierarchy strategies) that emits the Ron file - which is then available as a "last known good", for building the queue tree, and for the web front-end - as well as debugging.

On Mon, Sep 5, 2022, 5:42 PM Robert Chacón @.***> wrote:

Gotcha. I like the idea though, it's definitely more standard to use a DB. If we get some more feedback here in the thread - lets definitely consider that direction. Thank you for contributing ways to improve the code.

— Reply to this email directly, view it on GitHub https://github.com/rchac/LibreQoS/issues/65#issuecomment-1237508994, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRU4346XCELS4AMBSKD47DV4ZZPFANCNFSM6AAAAAAQFISFNY . You are receiving this because you were mentioned.Message ID: @.***>

rchac commented 2 years ago

In v1.3, we've added queuingStructure.json which acts as the true register of queues. Much more readable and it's more easy to verify exactly what's happening inside the software.

dtaht commented 2 years ago

Still the overall thrust is the db question. Maybe we setup a bug with a list of defferred bugs?

rchac commented 2 years ago

We can definitely revisit this.

So for statistics we do use a DB - it's InfluxDB, which handles time series data pretty efficiently. The queue structure is better suited to be stored as a structured format like JSON or RON, which it is in v1.3-alpha+. ShapedDevices.csv is less meant as a database and more meant as a way to pass info from the CRM integration to LibreQoS, which transforms that to a queuingStructure.json (actual storage of queue config).

thebracket commented 2 years ago

Related question regarding 1.3:

If you want a purely flat structure, with every client object directly hanging from the root - is there a shortcut around having to put every client in ShapedDevices.csv AND the topology network.json (with each device listing its "parent" as an entry in network.json)? I think I'm reading the code correctly to say that you can put "none" as the parent node in ShapedDevices.csv; I haven't tried it yet.

On Thu, Oct 20, 2022 at 4:26 PM Robert Chacón @.***> wrote:

Reopened #65 https://github.com/rchac/LibreQoS/issues/65.

— Reply to this email directly, view it on GitHub https://github.com/rchac/LibreQoS/issues/65#event-7635112978, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRU432VENGR2NVL56HVR6DWEG2GRANCNFSM6AAAAAAQFISFNY . You are receiving this because you were mentioned.Message ID: @.***>

rchac commented 2 years ago

Great question. For flat networks - they no longer need to enter any parent node or network.json info. They can simply leave network.json with:

{}

and leave the Parent Node field blank in ShapedDevices.csv. It will automatically assign devices to Generated Parent Nodes (Generated_PN_X) and will balance out the load by CPU using the binpacking module based on each subscriber's max bandwidth rates.

dtaht commented 1 year ago

in the end csvs won't scale. The relative simplicity tho is useful for small isps.