search

aschn / drf-tracking

Utils to track requests to Django Rest Framework API views
http://drf-tracking.readthedocs.org/
ISC License
366 stars 95 forks source link

How to do drf tracking write in async thread #114

Open amankumarjain opened 6 years ago

amankumarjain commented 6 years ago

is it possible to write log with async tasking i dont want my view thread to get blocked with drf tracking operation or to write in db. I am using celery for async task.

triat commented 6 years ago

Hello @amankumarjain, That's actually a really nice question that I tried to solve some times ago. My conclusion was:

  1. Using celery is not really a solution (at least for me as we are using SQS as a queue system) because response time of the DB ~= response time of SQS. So not a solution
  2. Async python3 could be a way to do it, for I don't remember which reason, we could consider this solution so I don't know
  3. "Send a request to add data and do not wait for the answer" was our best solution. What it means is you do your call to what ever system is saving your data but you don't wait the answer or there is no answer. By doing that you avoid every latency or service unavailable disruption. The bad point of this solution is that you could potentially lose data, or at least, not register them and don't know about it.

Actually the solution we went for, with a nice scaling possibility, is the Elasticsearch integration. We have an ES cluster hosted on Elastic cloud and we are sending our data directly to ES. An improvement to do is the 3rd point that I told you before, to send without waiting for the answer.

So I don't think there is an easy or awesome solution but there is definitively things to try. Aside all the possibilities, I think that DRF is trying to keep the impact on the API very low. If I'm not mistaken, our tests are checking that we don't take more than 20ms to log something (which is not perfect but it's ok-ish)