webdetails / repositorySynchronizer

Pentaho Repository Synchronizer
5 stars 8 forks source link

Timing out #17

Open brianpipa opened 7 years ago

brianpipa commented 7 years ago

Our repository has over 11,000 kjb/ktr files that we are synching in. We got it to sync in, but now the previewDifference call is hitting our internal 55 second timeout so we can't do any re-sync. Is there anything we can do to make it faster?

I was thinking we could run one of the behind the scenes jobs that populates the cache before we hit the UI and make a change somewhere so that it ALWAYS uses the cache. By doing that, it won't matter how long it takes for the cache to be populated, it would be there ready for the UI.

Does that make sense?

carlosrusso commented 7 years ago

The simplest approach would be to dynamically modify the subtrees to be synchronized, using the fact that the file prs.xml is not cached. This means you could have a set of files prs.xml.1, prs.xml.2, and cycle which file is the current prs.xml. On each file, you would change the<location> keys.

This quick hack could be run from a behind-the-scenes job and would allow you to sync a fraction of the tree on each attempt.

Note that if you don't need to review the items to synchronize, you could eventually automate the process by iterating through all of the settings file, and on each iteration overwrite the file prs.xml and invoke the synchronization endpoint. Obviously, this is only an idea that was never tested.