Data generation for join queries fails for 1e9 rows

h2oai / db-benchmark

reproducible benchmark of database-like ops

https://h2oai.github.io/db-benchmark

Mozilla Public License 2.0

322 stars 85 forks source link

Data generation for join queries fails for 1e9 rows #153

Closed amyskov closed 4 years ago

amyskov commented 4 years ago

Command Rscript _data/join-datagen.R 1e9 1e2 0 0 fails with next message: Producing match data of 1e9 rows Error in saveRDS(DT, dataf) : error writing to connection Execution halted

jangorecki commented 4 years ago

@amyskov Thank you for reporting. If data generation works for smaller size, then it is likely an issue with space available for temp. Could you ensure you can write to your temp?

saveRDS(iris, f<-tempfile(fileext="RDS"))
readRDS(f)

amyskov commented 4 years ago

Data was successfully written to the temp file, but i have found, that my /tmp storage (that is default storage for temporary files) is space limited and adding additional argument tmpdir= to all tempfile functions in _data/join-datagen.R file solves my problem. Thanks for your response!