ArchiveTeam / seesaw-kit

Making a reusable toolkit for writing seesaw scripts
Other
69 stars 30 forks source link

run_pipeline: Bound concurrent option to a reasonable number #65

Closed chfoo closed 10 years ago

chfoo commented 10 years ago

Limit the --concurrent N maximum to a reasonable number such as 10 to avoid users running hundreds of threads at a time. This reduces the problem where a user claims thousands of items but does not return most of the claims. A warning message could be used if the user runs more than 6. (No need to test the C10K ability of Tornado :smile:)