Cache folder check results.
More performance tweaks for a total performance improvement of 10x in my limited testing.
Using raw SQL instead of DBIx shaved off about 60% of the initial time spent in this task.
The other major improvement is caching the results of the directory check. The file check itself wasn't that expensive after all (I'm using an all SSD based system), but pathFromFileURL() cost a lot of time.
As I said this is a very limited testing (1800 tracks, three libraries). Eg. if I had more tracks in the libraries, I would have seen less of a speedup, as the inserts would have taken a larger part. But aforementioned two major improvements should show in many configuration. Feel free to give it a try and pull.
Cache folder check results. More performance tweaks for a total performance improvement of 10x in my limited testing.
Using raw SQL instead of DBIx shaved off about 60% of the initial time spent in this task. The other major improvement is caching the results of the directory check. The file check itself wasn't that expensive after all (I'm using an all SSD based system), but pathFromFileURL() cost a lot of time.
As I said this is a very limited testing (1800 tracks, three libraries). Eg. if I had more tracks in the libraries, I would have seen less of a speedup, as the inserts would have taken a larger part. But aforementioned two major improvements should show in many configuration. Feel free to give it a try and pull.