asfimport
commented
2 years ago Weston Pace / @westonpace: I believe at this point we have tackled the segmentation faults as part of the R bindings PR. Given that we are probably not going to proceed with exposing this feature via R until the randomness issue is resolved we will want to port those fixes into their own PR. I'll create a new issue for this. We can retitle this issue to focus on the randomness issue (since we have a lot of good content in the description).
Regarding the randomness issue, @jonkeane @save-buffer and I had a brief chat. This shouldn't interfere with our ability to do internal profiling but we likely prevents using the generated data for benchmarking (where we want to make sure we are more consistent and deterministic). We currently generate data and random numbers in a column-major fashion. The official generators generate data and random numbers in a row-major fashion. In addition, we generate data across multiple threads which could cause issues as well.
We could write a separate row-major generator (hopefully sharing some utilities) or we could try and use the generator as is and instead generate a table of random numbers for each batch and then transpose it and use the transposed random numbers. Or we could generate a large batch of random numbers and "seek" around the batch. There may be other complications as well to ensure compatibility between the two generators. This will probably be a significant chunk of work.
When checking these against the known-good answers for scale factor 1, all of the queries are off (two are close enough that they get past arrowbench's pretty loose validation).
Looking at the TPC-H tools, the validation for dbgen is:
And the data that this generator is producing does not conform to that. Even if I sort the data by columns that they appear to be in the dbgen (or duckdb) produced data, the data we get from our TPC-H generator does not match.
We might want a mode where we produce random TPC-H like data. But for benchmarking we need a way to produce actual TPC-H compliant data out of the box (we can deal with rows in a shuffled order if we need to, but the content of the data must be the same.
Maybe taking a look at DuckDB's implementation of the random seeds might help with a way to accomplish this?
Here's an example of generating data with duckdb (first ten lines of lineitems — and it's the same each time I generate it):
And the first ten lines of lineitems I generated with the official dbgen:
And the first ten lines of the validation file form the TPC-H tools:
Note, you can generate this tpc-h data from https://github.com/apache/arrow/pull/12769 with the following. This shuffling is needed because we can only write datasets from execnodes (without materializing into memory entirely), so we move the files around as if they were single file writes:
Reporter: Jonathan Keane / @jonkeane
Note: This issue was originally created as ARROW-16100. Please see the migration documentation for further details.