Closed haffla closed 4 years ago
Interesting!
Could you provide reproducible code? A snippet that we can copy, paste and run on our machines that show the problem. That will make it so much easier to debug it and solve it. Otherwise we have no idea where to start.
Reproducible code:
require "pg"
def generate_values(i)
String.build do |io|
500.times do |j|
io << ',' unless j == 0
io << "(#{i}, #{j})"
end
end
end
DB.open ENV["DATABASE_URL"] do |db|
db.exec "CREATE TABLE foo (a int, b int)"
1000.times do |i|
puts "Iteration #{i}"
db.exec "INSERT INTO foo (a, b) VALUES #{generate_values(i)}"
p! GC.stats
end
db.exec "DROP TABLE foo"
end
Heap seems to grow by about 140K per iteration.
This is a problem in crystal-db: the string passed to exec
is never released. Apparently crystal-db
caches query strings but never releases them. So if you keep passing different strings you'll get more and more memory consumed.
This problem is solved by using parameter placeholders, like "$1". But I still think this is an issue in crystal-db: there should be a pool of prepared statements, not an unbounded cache.
Please someone open this issue in crystal-db (I don't have more time right now, sorry).
Answered in crystal-db we can keep the discussion there. I agree is something that belongs in crystal-db.
I am currently writing a crystal script that performs an ETL process. I read data from one table in a MySQL database, perform some transformations and write the result into a Postgres database. The workload is quite big (9 million rows). I stream results from the source db, collect them in an array and after every 500 rows I insert them into the target DB.
So the problem now is that the above works but over time memory consumption grows and grows and grows, on my machine finishing with like 600 MB. So initially I thought maybe it's my code or maybe it's just Crystal (I am new to Crystal). I realise that I am creating a lot of huge strings and all but when I just skip the insertion into the Postgres DB (and still generate all that SQL) this runs with a very very low and stable memory profile.
I am opening the issue in crystal-pg and not in crystal-db because I also tested the same script with two MySQL connections, transferring data from one MySQL db into another and there memory consumption is quite stable at around 180 MB.