search

zombiezen / postgrestest

A Go test harness that starts an ephemeral PostgreSQL server
Apache License 2.0
43 stars 3 forks source link

feature request: database preparation for instant next run #2

Open stapelberg opened 7 months ago

stapelberg commented 7 months ago

Hey @zombiezen, hope you’re doing well!

I’m currently using https://eradman.com/ephemeralpg/ and wanted to see if I could switch to your Go package instead.

One of the features I’m missing from ephemeralpg / pg_tmp is that — independently from the instance it starts — pg_tmp prepares a new postgresql instance in the background which will be immediately ready for the next run: https://github.com/eradman/ephemeralpg/blob/11c455afde85e5062d56127b78be735b14ed880b/pg_tmp.sh#L69-L79

While the timing for the first test run is unchanged, subsequent test runs are much faster because they effectively skip the lengthy PostgreSQL invocation (≈700 ms on a MacBook Air M1, so pretty noticeable).

Would you accept such a feature as a contribution?

zombiezen commented 7 months ago

Hi Michael! Good to hear from you. 🙂

Interesting! I'm aligned with improving startup speed. How would such a feature clear the database between runs? And does it eventually clean up the database, or does it rely on the OS to eventually clear /tmp?

stapelberg commented 7 months ago

Thanks for your quick reply!

Interesting! I'm aligned with improving startup speed.

Great!

How would such a feature clear the database between runs?

pg_tmp marks the ready-but-waiting database with a NEW marker file. Once the database is used, the NEW file will be deleted. There is no re-use of the same database between runs.

And does it eventually clean up the database, or does it rely on the OS to eventually clear /tmp?

When pg_tmp is used with the -w flag, it cleans up the database (if unused) after the specified number of seconds: https://github.com/eradman/ephemeralpg/blob/11c455afde85e5062d56127b78be735b14ed880b/pg_tmp.sh#L82-L84

It would probably make sense to also include this in postgrestest, but maybe separately? Or do you see cleanup as necessary before we can implement preparation?

zombiezen commented 7 months ago

When pg_tmp is used with the -w flag, it cleans up the database (if unused) after the specified number of seconds: https://github.com/eradman/ephemeralpg/blob/11c455afde85e5062d56127b78be735b14ed880b/pg_tmp.sh#L82-L84

It would probably make sense to also include this in postgrestest, but maybe separately? Or do you see cleanup as necessary before we can implement preparation?

The current API has a Cleanup() method that is used to shut down the server. I'd imagine we'd hook in there? I might be misunderstanding the difficulty or the difference in approach that pg_tmp uses.

stapelberg commented 6 months ago

The current API has a Cleanup() method that is used to shut down the server. I'd imagine we'd hook in there? I might be misunderstanding the difficulty or the difference in approach that pg_tmp uses.

Yes, but the key is that the amortization happens across processes.

I posted a draft PR to illustrate this at https://github.com/zombiezen/postgrestest/pull/4