search

postgresql-interfaces / psqlodbc

Other
16 stars 13 forks source link

Use mimalloc to improve performance and reduce memory allocation lock contention #6

Closed apgrucza closed 5 months ago

apgrucza commented 7 months ago

We have a multi-threaded Windows application that was experiencing delays due to high lock contention in memory allocations from the PostgreSQL ODBC driver. We tried modifying the driver to use mimalloc, which is a memory allocator with better performance characteristics. After deploying this change, the delays due to lock contention disappeared. It has been running on thousands of our production deployments for 9 months without issue.

I've created this pull request so that others can benefit from this change by building the driver with the _MIMALLOC_ symbol defined and linking to the mimalloc library. If building on Windows, BuildAll.ps1 accepts a -UseMimalloc argument that does this for you (requires a toolset of v141, v142 or later). Below is an example of how to build with mimalloc on Windows:

.\BuildAll.ps1 -Platform x64 -Toolset v141 -UseMimalloc

Currently the usage of mimalloc is off by default, but I'd like to get people's thoughts on whether it should be enabled by default.

apgrucza commented 5 months ago

I have a fix for the caching issue here: https://github.com/postgresql-interfaces/psqlodbc/pull/13

apgrucza commented 5 months ago

can you check these https://github.com/davecramer/psqlodbc/actions/runs/9036264469 ?

The x86 artifact still contains libpq 17, and looking at the GitHub Actions log reveals that it's because the "build postgresx86" step was again skipped (due to the caching problem). If you want to merge your changes then I can rebase #13 and the problem should go away.

apgrucza commented 5 months ago

The EDB installer takes 2m 30s to install. With your recent change, is the EDB installer now only used to run PostgreSQL (so that the psqlodbc tests can run)? If so, we could reduce the workflow duration by running the PostgreSQL Docker image instead.

davecramer commented 5 months ago

The EDB installer takes 2m 30s to install. With your recent change, is the EDB installer now only used to run PostgreSQL (so that the psqlodbc tests can run)? If so, we could reduce the workflow duration by running the PostgreSQL Docker image instead.

I didn't think docker ran on github windows images ?

apgrucza commented 5 months ago

I didn't think docker ran on github windows images ?

Oh, that surprises me. I haven't used GitHub Actions much before.

davecramer commented 5 months ago

I think you can run testcontainers which does run docker but I don't think you can specify the version of postgres. It seemed simpler to use EDB