pgcentralfoundation / pgrx

Build Postgres Extensions with Rust!
Other
3.42k stars 223 forks source link

Document CI caching strategy #1706

Open nyurik opened 1 month ago

nyurik commented 1 month ago

When running PGRX in CI, it has to download every PG version, taking considerable time and resources. Many CI platforms including Github's offer various local caching systems, e.g. Swatinem/rust-cache action. Please document how CI steps should be modified in order to speed up CI builds (e.g. which dirs can be safely cached, but will be reliably updated when newer version becomes available, etc)

Should probably be added in #1694 too

workingjubilee commented 1 month ago

uhh our current caching strategy is to not do the thing you describe.

workingjubilee commented 1 month ago

so, you would probably know better? PRs welcome?

nyurik commented 1 month ago

@workingjubilee I am mostly concerned for the users of pgrx, not pgrx repo itself (although I am sure it can also benefit from it by reducing the CI run time). In order to help, I need to understand what gets re-downloaded on every CI run (especially for the users of pgrx), and to what dir, as well as the next steps (extracting to another dir, etc). If any of those dirs have a more or less stable state, we could use dir caching.

workingjubilee commented 1 month ago

If someone does the very naive thing of running cargo pgrx init in CI, they would have to cache the $PGRX_HOME dir, as you mentioned. We do that in our CI, but we are fine with it not being meaningfully cached because we are in fact testing that we can create a new extension directory, execute the cargo pgrx init process for an arbitrarily picked Postgres version, and then build and test that extension.

workingjubilee commented 1 month ago

Otherwise we build and test against a Postgres version we install from the system. This has the disadvantage that we don't test against a Postgres built with assertions.

nyurik commented 1 month ago

@workingjubilee thx, does cargo pgrx init take into account the current state of the $PGRX_HOME? I.e. would it skip download/uncompress/compile/install steps if everything is already up to date? If not, is there a way to (semi-)reliably detect if pgrx init step should be skipped?

nyurik commented 1 month ago

I tried running cargo pgrx init twice in a row locally, and it re-downloaded and re-compiled all PG versions. So it seems the ideal case would be to add some magical flag to the init command to skip if already initialized. Eventually I would even recommend to make the default behavior unless --force is specified.