search

facebook / buck2

Build system, successor to Buck
https://buck2.build/
Apache License 2.0
3.53k stars 215 forks source link

Golden testing? #525

Closed SUPERCILEX closed 9 months ago

SUPERCILEX commented 9 months ago

Are there docs on doing golden testing? So comparing generated output with something checked into the source tree, and using buck2 to update the source tree.

ndmitchell commented 9 months ago

No docs. Internally we tend to have a variable like REGENERATE_GOLDEN https://github.com/facebook/buck2/blob/6b06646a36b0ac8b33378fd4103e461e2cec20ba/starlark-rust/starlark_syntax/src/golden_test_template.rs#L24 and if that is set, we update the source, if not, we just fail. As long as you tag the rule as must run locally it should work.

SUPERCILEX commented 9 months ago

So you can't touch the source code purly in buck? An example file says to use cargo to regenerate the file: https://github.com/facebook/buck2/blob/19678f98577799856f05ff9766d0c143d1c52e92/starlark-rust/starlark/src/typing/golden/zip.golden#L4

ndmitchell commented 9 months ago

There are two reasons we recommend Cargo:

  1. If you are using remote execution then any changing of the source code doesn't actually update anything.
  2. starlark-rust is a separate open source project as well, so we don't require Buck2.

If you are using Buck2 to update golden files, the solution is probably to have a run command that actually applies the update. So you could do buck2 run :update-golden-tests. Generally we hope a build doesn't mutate the source directory, but a run can do anything it likes.

SUPERCILEX commented 9 months ago

Got it, thanks.