The factory will break after 400 records with a cryptic "Too many open files" error thrown in Catmandu::Fix on line 91:
$fix = Path::Tiny::path($fix)->slurp_utf8;
This line opens the Fix file and reads the contents. Generally, a fix file should only be opened and read once. The text string containing the fix code is parsed and applied in fixer() Once the perl script gets terminated, Perl garbage collection will close any open file descriptors. In this case: since the Fix is only read at the start of execution, there's only one open file descriptor referring to the fix file.
Datahub::Factory::Fixer::Condition will bang of a new instance of Catmandu::Fix with each iteration and open a new file descriptor. After 400 iterations, the *nix file system limit of allowed open descriptor is reached and the OS syscall will hand off an error to Perl.
This issue requires a rewrite of the Datahub::Factory::Fixer::Condition module.
Read out the pipelineconfig and create for an instance for each defined fixer.
Reuse these instances in the loop instead of creating a new one.
Use the condition logic to call the relevant instance for each record.
The factory will break after 400 records with a cryptic "Too many open files" error thrown in Catmandu::Fix on line 91:
This line opens the Fix file and reads the contents. Generally, a fix file should only be opened and read once. The text string containing the fix code is parsed and applied in
fixer()Once the perl script gets terminated, Perl garbage collection will close any open file descriptors. In this case: since the Fix is only read at the start of execution, there's only one open file descriptor referring to the fix file.Datahub::Factory::Fixer::Condition will bang of a new instance of Catmandu::Fix with each iteration and open a new file descriptor. After 400 iterations, the *nix file system limit of allowed open descriptor is reached and the OS syscall will hand off an error to Perl.
This issue requires a rewrite of the Datahub::Factory::Fixer::Condition module.