search

mff-uk / odcs

ODCleanStore
1 stars 11 forks source link

Increase speed of RUIAN cache pipeline #1279

Open tomas-knap opened 10 years ago

tomas-knap commented 10 years ago

See pipeline RUIAN cache on ODCS. Speed must be increased, so that it is processed under 2 day.

Without ODCS, the pipeline runs cca 1 day, 18000 files. With ODCS, currently, it is running for 22h, and processed only 1613 files / 18000

So it should run at least 5x faster

@jakubklimek Please add any note if needed

jakubklimek commented 10 years ago

Basically you should profile the pipeline and see whats taking longer than necessary On Mar 2, 2014 5:19 PM, "Tomas Knap" notifications@github.com wrote:

See pipeline RUIAN cache on ODCS. Speed must be increased, so that it is processed under 2 day.

Without ODCS, the pipeline runs cca 1 day, 18000 files. With ODCS, currently, it is running for 22h, and processed only 1613 files / 18000

So it should run at least 5x faster

@jakubklimek https://github.com/jakubklimek Please add any note if needed

— Reply to this email directly or view it on GitHubhttps://github.com/mff-uk/ODCS/issues/1279 .