search

swiss-art-research-net / skkg-pipeline

ETL pipeline for the Sammlung Digital project
2 stars 0 forks source link

Optimise mapping #177

Open fkraeutli opened 3 months ago

fkraeutli commented 3 months ago

Find out if speed of mapping routine can be improved

fkraeutli commented 3 months ago

Bottleneck could to be the starting up of the JVM. Since we need to map each input file individually, the java command is executed with every file which might produce overhead. A possible solution could be to implement a continually running service that accepts and outputs individual files.

fkraeutli commented 3 months ago

check https://www.martiansoftware.com/nailgun/background