search

spark-root / laurelin

Allows reading ROOT TTrees into Apache Spark as DataFrames
BSD 3-Clause "New" or "Revised" License
8 stars 4 forks source link

Arraybuilder thread executor leaks threads #63

Closed PerilousApricot closed 4 years ago

PerilousApricot commented 4 years ago

Each TTreeDataSourceV2PartitionReader will allocate a new batch of threads, which managed to cause my to exhaust my available thread limit on my laptop by looping over spark.read().load("rootfile").write().parquet("outfile")

https://github.com/spark-root/laurelin/blob/46e5e5064c67cc2f269969700bf0edc4008183c1/src/main/java/edu/vanderbilt/accre/laurelin/Root.java#L128-L134

PerilousApricot commented 4 years ago

fixed via a89c12219e1540849e821ecef3411484bda8b052