torodb / testing-tools

A set of projects that make it easy to execute test, specially with docker. It is heavily used by ToroDB, but can be useful on third party projects.
Apache License 2.0
1 stars 3 forks source link

Try to improve testing performance by using tmpfs disks #9

Open gortiz opened 7 years ago

gortiz commented 7 years ago

As explained here by @vladmihalcea, integration test that use databases on docker can run faster if the container is configured to mount data folders on a tmpfs disk. It can be easily done with the argument --tmpfs at container creation time.

We need to check how this argument interacts with persistent volumes, as MongoDB and Postgres default images use them. Right now, we have to manually destroy each image when the container is stopped, but it is quite tricky and in fact if fails if the JVM is destroyed without calling shutdown hooks. Ideally, docker will not create the volume in the host if tmpfs overrides it, which means we wouldn't need to manually destroy the volumes on the host and the process will be more correct.

gortiz commented 7 years ago

If we add this feature, it is important to know that tmpfs doesn't swap, so the volume size is decided at container creation. If it takes too much memory, the test can be slower just because the running program will not have enough memory. Therefore we need to add a parameter to specify the max amount of RAM memory the tmpfs volume will use.

vladmihalcea commented 7 years ago

While on MySQL and MariaDB, using Docker and tmpfs shows a massive improvement in test speed, PostgreSQL can also benefit from mapping the DATA folder in memory. We use this technique a lot when testing Hibernate ORM against various RDBMS.

ahachete commented 7 years ago

Also setting some -otherwise unsafe- parameters like

fsync=off
synchronous_commit=off
full_page_writes=off

should speed it up even more.