search

ldbc / ldbc_graphalytics_platforms_graphblas

LDBC Graphalytics implementation using SuiteSparse:GraphBLAS and LAGraph
Apache License 2.0
4 stars 7 forks source link

Workaround for duckdb parallel cvs reader OoM fault on high #threads #40

Closed yvdriess closed 1 year ago

yvdriess commented 1 year ago

This option in relabel.py:

con.execute(f"SET experimental_parallel_csv=true")

causes duckdb to throw an out of memory error, even though there is plenty of memory left. It only appears to do this on systems with high core/thread count. Running with reduced thread count no longer produces the error, hence this workaround.

szarnyasg commented 1 year ago

@yvdriess Thanks, this looks good. I added you as a contributor to repository so you will be able to merge future pull requests.