search

d6t / d6tflow

Python library for building highly effective data science workflows
https://d6tflow.readthedocs.io/en/latest/
MIT License
951 stars 77 forks source link

Too many open files #18

Closed CardosoJr closed 4 years ago

CardosoJr commented 4 years ago

Hello,

I've been using d6tflow to run multiple ML models with different parameters. In some executions I trained over 20K models.

I'm using 10 workers in a linux environment and after some time I get an error of two many open files. I do not open any file in the custom code inside my tasks. I'm using TaskPqPandas and TaskPickle targets to this for me....

Any help on this issue would be appreciated.

Thanks!

CardosoJr commented 4 years ago

The problem was at the PickleTarget. There're calls to the open method without the with clause

d6tdev commented 4 years ago

Thanks for flagging. Fixed in v0.1.8 can you install dev version and check it works?

d6tdev commented 4 years ago

haven't heard back, assume it's fixed, reopen if not