search

appsmithorg / appsmith

Platform to build admin panels, internal tools, and dashboards. Integrates with 25+ databases and any API.
https://www.appsmith.com
Apache License 2.0
32.85k stars 3.58k forks source link

[Bug]: [aforce] Memory leak issue in Appsmith's database integration causing website downtime. #34028

Open rohan-arthur opened 2 months ago

rohan-arthur commented 2 months ago

Is there an existing issue for this?

Description

slack link

Steps To Reproduce

no clear reproduction steps

Public Sample App

No response

Environment

Production

Severity

High (Blocker to building or releasing)

Issue video log

No response

Version

Self hosted

sneha122 commented 2 months ago

The root cause of this issue is still unknown as we tried reproducing this issue by:

  1. Creating MariaDB instance on AWS RDS and creating appsmith app that has multiple pages with multiple onload queries, observed memory usage on AWS dashboard, nothing suspicious when using the application, only thing observed was active memory trend was increasing even if the application was not in use, but this did not stop the app in any way or caused any lag.

    Screenshot 2024-06-10 at 7 06 21 PM

  2. For now, we have also raised a couple of questions with the user to understand their environment, awaiting user response to take further actions.

github-actions[bot] commented 2 months ago

This critical issue has not seen activity for a while. It will be closed in 7 days unless further activity is detected or the Critical tag is removed.

okletsov commented 1 month ago

Memory leak happens in our prod app (this github issue is the result of my communication with support). Let me know if there is any additional information we can provide to troubleshoot.

rohan-arthur commented 1 month ago

@okletsov Thank you for sharing more context and following up. Apologies for not being able to get to this because we are a small team, and have some more urgent issues to work on.

We have struggled to reproduce the error situation, so any clues or suggestions would help to make our debugging easier.

sneha122 commented 1 month ago

@okletsov Thank you so much for your patience. I have gone through the code to understand the potential causes of connections not being released and eventually causing memory leak, I wanted to understand the nature of datasources and queries you have in your application that are running on page load. My hunch right now is that you may have multiple datasources (connecting to same db) having queries on page load, this can cause creation of multiple connection pools to same database and end up exhausting connections on that database. Hence If you can let me know if the 11 queries that running page load, do they belong to same datasource or they belong to different datasources (connecting to same db)?

Further more, I am creating action item here to add more logs to observe the nature of connection pool to understand active connections, connection pool size etc

okletsov commented 4 weeks ago

Hello sneha,

I will have a chance to jump on it today. I believe we have one data source (in appsmith interface). Can send you some screenshots later today.

Thank you, Oleksii Kletsov

On Wed, Jul 24, 2024, 6:59 AM sneha122 @.***> wrote:

@okletsov https://github.com/okletsov Thank you so much for your patience. I have gone through the code to understand the potential causes of connections not being released and eventually causing memory leak, I wanted to understand the nature of datasources and queries you have in your application that are running on page load. My hunch right now is that you may have multiple datasources (connecting to same db) having queries on page load, this can cause creation of multiple connection pools to same database and end up exhausting connections on that database. Hence If you can let me know if the 11 queries that running page load, do they belong to same datasource or they belong to different datasources (connecting to same db)?

Further more, I am creating action item here https://github.com/appsmithorg/appsmith/issues/35158 to add more logs to observe the nature of connection pool to understand active connections, connection pool size etc

— Reply to this email directly, view it on GitHub https://github.com/appsmithorg/appsmith/issues/34028#issuecomment-2247609356, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLX4ZAVBN3HZGLUPHSO4VTZN6CIJAVCNFSM6AAAAABI4OWC2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBXGYYDSMZVGY . You are receiving this because you were mentioned.Message ID: @.***>

okletsov commented 4 weeks ago

Hi Snrha,

Please see the datasource configuration attached. The queries themselves are simple "select * from tableName" queries. "tableName" for many of them is the View created on the DB server.

[image: image.png]

Thanks, Oleksii Kletsov

On Sat, Jul 27, 2024 at 8:50 AM Oleksii Kletsov @.***> wrote:

Hello sneha,

I will have a chance to jump on it today. I believe we have one data source (in appsmith interface). Can send you some screenshots later today.

Thank you, Oleksii Kletsov

On Wed, Jul 24, 2024, 6:59 AM sneha122 @.***> wrote:

@okletsov https://github.com/okletsov Thank you so much for your patience. I have gone through the code to understand the potential causes of connections not being released and eventually causing memory leak, I wanted to understand the nature of datasources and queries you have in your application that are running on page load. My hunch right now is that you may have multiple datasources (connecting to same db) having queries on page load, this can cause creation of multiple connection pools to same database and end up exhausting connections on that database. Hence If you can let me know if the 11 queries that running page load, do they belong to same datasource or they belong to different datasources (connecting to same db)?

Further more, I am creating action item here https://github.com/appsmithorg/appsmith/issues/35158 to add more logs to observe the nature of connection pool to understand active connections, connection pool size etc

— Reply to this email directly, view it on GitHub https://github.com/appsmithorg/appsmith/issues/34028#issuecomment-2247609356, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLX4ZAVBN3HZGLUPHSO4VTZN6CIJAVCNFSM6AAAAABI4OWC2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBXGYYDSMZVGY . You are receiving this because you were mentioned.Message ID: @.***>

sneha122 commented 3 weeks ago

Hello @okletsov

Thanks a lot for the information provided. Unfortunately, I am not able to see the image that you have attached, it would be great if you can schedule call using this link. This can quickly help us to analyse database connections.

okletsov commented 3 weeks ago

Hi Sneha,

Done. Scheduled a call for Thursday. Talk to you then. The screen shot was just trying to show the fact there is 1 data source in our appsmith application. We can discuss more when we meet.

Thanks, Oleksii Kletsov

On Mon, Jul 29, 2024 at 6:07 AM sneha122 @.***> wrote:

Hello @okletsov https://github.com/okletsov

Thanks a lot for the information provided. Unfortunately, I am not able to see the image that you have attached, it would be great if you can schedule call using this https://calendly.com/d/cktt-vjj-4gz/30min link. This can quickly help us to analyse database connections.

— Reply to this email directly, view it on GitHub https://github.com/appsmithorg/appsmith/issues/34028#issuecomment-2255522748, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLX4ZGSO5ELGUO7SFRJMQ3ZOYH5JAVCNFSM6AAAAABI4OWC2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVGUZDENZUHA . You are receiving this because you were mentioned.Message ID: @.***>