search

bradmontgomery / django-redis-metrics

Metrics for django apps backed by Redis.
http://django-redis-metrics.readthedocs.io
MIT License
93 stars 24 forks source link

Dumping and loading redis data #59

Open psychok7 opened 7 years ago

psychok7 commented 7 years ago

@bradmontgomery i was reading up on https://github.com/bradmontgomery/django-redis-metrics/issues/35 but i still have some questions regarding:

I see the advantages of Redis reading and saving metrics regarding performance, but what happens when you accumulate metrics of 4 years, regarding how much RAM Redis will start using?

I ask this because to me it looks like dumping the data (after it becomes very big for Redis) in a DB (Postgres for example) and then being able to read it (historical data) would be a really nice feature.

From what i understand if you have 2GB of RAM (and cannot afford more) your metrics data is limited to that. You would have to delete what you have and start over.

Is this what happens? Do you have a solution for a problem like this?

Cheers

bradmontgomery commented 7 years ago

Hi @psychok7, thanks for opening this issue. This seems to be a fairly different request than #35, although I suppose it's tangential. I could see the use for a command that exports redis data as an archive of some sort (maybe a giant JSON field), and I think it could be worthwhile to some to have a util function or a management command to do that.

Since I'm using this project on personal and/or small-traffic projects, I personally don't care to keep data for more than a year, so I tend to let redis expire data after some period of time.

If you'd like to work on a data export feature, I'd be willing to merge it into the project.

psychok7 commented 7 years ago

@bradmontgomery i also don't have a high traffic project but i was thinking as a future feature.

Maybe you are familiar with https://github.com/p/redis-dump-load that will do the exporting.

Maybe django-redis-metrics can have a command that exports the json, and saves to the db (for a later query). Then a view that fetches the json (directly from the db without using redis maybe) and chooses between showing it all or up to a certain date.

What do you think? I don't have the time right now but maybe in the future i could help out with this.