search

avniproject / avni-etl

GNU Affero General Public License v3.0
0 stars 6 forks source link

Set up Newrelic and investigate disk usage issues #71

Closed vinayvenu closed 1 year ago

vinayvenu commented 1 year ago

Since release 4.0, we've been having consistently high disk usage on the prod db. ETL was shut down on Sep 15 and brought back up on Sep 18. During this time, EBS byte balance stayed high. This points to the ETL job being the cause of high disk usage. This story is to

Acceptance criteria

  1. Set up NewRelic on avni-etl so that we can identify these issues
  2. Identify and resolve bottlenecks causing high disk usage

Image

petmongrels commented 1 year ago

new relic config added to keeweb. this needs to be present in start.sh -Dnewrelic.environment=production -javaagent:/opt/newrelic/newrelic.jar (this doesn't seem be changed / managed by deploy scripts right now)

petmongrels commented 1 year ago

sync telemetry insert seems to be taking a disproportionate amount of time compared to value added by it.

vinayvenu commented 1 year ago

Missing index?

On Mon, Sep 18, 2023, 19:49 Vivek Singh @.***> wrote:

sync telemetry insert seems to be taking a disproportionate amount of time compared to value added by it.

— Reply to this email directly, view it on GitHub https://github.com/avniproject/avni-etl/issues/71#issuecomment-1723524338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACUBHRDNWNONYGYE72KMPTX3BKABANCNFSM6AAAAAA44JECHI . You are receiving this because you authored the thread.Message ID: @.***>

petmongrels commented 1 year ago

Created the index. There are no other queries of big concern as of now on ETL. There are few slow ones but they are less frequently run, so not very time consuming. The RDS graph looks better (not sure it is only because of this or if we made other changes as well).

Image