Open SimonGurney opened 2 years ago
To increase performance, we should thread the file enumeration efforts and run them in parallel to a multiple of the core count.
This may lead to conflicts with output to elastic and CSV, which might need queuing. We are using a Sink though, which should be thread safe already.
To increase performance, we should thread the file enumeration efforts and run them in parallel to a multiple of the core count.
This may lead to conflicts with output to elastic and CSV, which might need queuing. We are using a Sink though, which should be thread safe already.