JuliaDatabases / ODBC.jl

An ODBC interface for the Julia programming language
https://odbc.juliadatabases.org/stable
Other
106 stars 62 forks source link

DBInterface.execute is taking a lot of time to pre-compile in Julia 1.9 #371

Open ronisbr opened 1 year ago

ronisbr commented 1 year ago

Hi!

After upgrading to Julia 1.9, I see that DBInterface.execute is taking a lot of time to precompile in Julia 1.9. The issue is that it is happening every time in a new session:

julia> @time DBInterface.execute(conn, query);
 35.153157 seconds (22.41 M allocations: 907.274 MiB, 0.39% gc time, 99.99% compilation time)

julia> @time DBInterface.execute(conn, query);
  0.012925 seconds (1.59 k allocations: 111.314 KiB, 80.52% compilation time)

I can reproduce this behavior in two machines (Intel Xeon and M1) when connecting to a MS SQL server.

I think everything can be highly improved if a precompile statement is added to the package.

ronisbr commented 12 months ago

I finally could isolate the problem!

The precompilation did not help because it takes the same amount of time to load the package. Hence, there is not point in adding precompile statements, AFAIK.

All the problem comes from this structure here:

https://github.com/JuliaDatabases/ODBC.jl/blob/a6e4fb726c7c34fdfa59c3741ef98ddc6f3b9e32/src/utils.jl#L83

It seems that Julia v1.9 is compiling a lot of signatures, leading to this huge slowdown.

However, I could not find any good workaround beside not specializing the buffer, which can lead to many problems.

ronisbr commented 12 months ago

Btw, these are the steps to reproduce the problem without requiring connection to a database:

Julia v1.6

julia> @time precompile(ODBC.Buffer, (ODBC.API.SQLSMALLINT, Int, Int, Bool))
  0.556863 seconds (698.22 k allocations: 33.957 MiB, 0.74% gc time, 100.00% compilation time)
true

Julia v1.9

julia> @time precompile(ODBC.Buffer, (ODBC.API.SQLSMALLINT, Int, Int, Bool))
 30.855483 seconds (21.02 M allocations: 814.780 MiB, 0.24% gc time, 100.00% compilation time)
true