search

baztian / jaydebeapi

JayDeBeApi module allows you to connect from Python code to databases using Java JDBC. It provides a Python DB-API v2.0 to that database.
GNU Lesser General Public License v3.0
365 stars 148 forks source link

Memory Leak Issue in Jaydebeapi Python Package #227

Open Yeyan-wzq opened 1 year ago

Yeyan-wzq commented 1 year ago

I have encountered a severe memory leak issue while using the Jaydebeapi Python package. Even after closing the cursor and connection, the program's memory usage does not decrease, and it continues to increase with each use of the package. After multiple uses, the program's memory usage reached half of the system's total memory, and only closing the program released the memory.

Upon investigation, I found that there is a memory leak in the data conversion process in the init file of Jaydebeapi, especially when using the getObject function. This issue is causing significant performance problems and memory usage in my application.

Steps to Reproduce:

Open a connection using Jaydebeapi package. Execute SQL queries and fetch data using the cursor. Close the cursor and connection. Repeat the above steps multiple times. Monitor the program's memory usage. Expected Results: The program's memory usage should not increase significantly with each use of the Jaydebeapi package, and closing the cursor and connection should release the memory.

Actual Results: The program's memory usage continues to increase with each use of the Jaydebeapi package, and closing the cursor and connection does not release the memory.

Additional Information: The issue is primarily occurring in the data conversion process in the init file of Jaydebeapi, especially when using the getObject function. I would appreciate any guidance or updates on how to resolve this problem, as it is significantly impacting the performance and memory usage of my application. Thank you for your attention to this matter.

elapouya commented 1 year ago

I confirm I have that problem too. I have to migrate a huge database, it is almost impossible because it uses too much RAM, even by closing/recreating cursor or connection.

Niivii commented 1 year ago

Confirming too,

Seems to be triggered by _java_to_py(java_method) method in my case and especially .getObject() as stated before.

sairam6267 commented 9 months ago

Hello jaydebapi maintainers, I need help from your end.

Im facing an issue regarding memory and my program is terminating automatically in the case when im fetching the data from the database usig chunk size in scheduled manner as a linux service.

  1. For first some cycles its working here frequently the merory usage is increasing linearly.
  2. Even thought i reduced the chuck size still its casing the same issue.

Can anyone please provide resolution:

  1. How to stop memory usage ?
  2. Is there any soultion in the jaydebapi where i can provide arguments in the execute query which helps to stop this kind of issue.

Tried solutions:

  1. chnaged the chunk size.
  2. closing the curose connection.
  3. Tried to increase heap memory of Java.