thedatahub / Datahub-Factory

Datahub::Factory - Transport metadata between Collection Management Systems and the Datahub
Other
2 stars 4 forks source link

Datahub::Factory::Fixer::Condition breaks after 400 records #33

Closed netsensei closed 7 years ago

netsensei commented 7 years ago

The factory will break after 400 records with a cryptic "Too many open files" error thrown in Catmandu::Fix on line 91:

$fix = Path::Tiny::path($fix)->slurp_utf8;

This line opens the Fix file and reads the contents. Generally, a fix file should only be opened and read once. The text string containing the fix code is parsed and applied in fixer() Once the perl script gets terminated, Perl garbage collection will close any open file descriptors. In this case: since the Fix is only read at the start of execution, there's only one open file descriptor referring to the fix file.

Datahub::Factory::Fixer::Condition will bang of a new instance of Catmandu::Fix with each iteration and open a new file descriptor. After 400 iterations, the *nix file system limit of allowed open descriptor is reached and the OS syscall will hand off an error to Perl.

This issue requires a rewrite of the Datahub::Factory::Fixer::Condition module.