search

dnbert / prm

PRM Allows you to quickly build package repositories, inspired by Jordan Sissels' FPM
MIT License
250 stars 33 forks source link

Remove parallel each as it creates a race condition #12

Closed andytinycat closed 11 years ago

andytinycat commented 11 years ago

The parallel each usage breaks the Packages file - it creates a race condition where two threads are writing to the same file at the same time, resulting in a broken Packages file.

This commit removes the 'peach' iteration, as this causes the problem.

jordansissel commented 11 years ago

Seems like it'd be better to fix the race condition than to remove parallelism?

andytinycat commented 11 years ago

Whilst I agree with you in principle, I don't think the parallelism adds more than a fractional speed increase, and introduces subtle bugs (like this one).

If you/dnbert prefer, I'll submit a PR that fixes it instead.

andytinycat commented 11 years ago

Just to be clear on the actual problem - what happens is that on some runs, two Package: stanzas get crushed together with lines from each being interweaved. This makes apt unhappy, so there's newlines where they shouldn't be and multiple Package: lines in the same stanza.

dnbert commented 11 years ago

In my experience and use case, parallel processing reduces the repository generation process from 3-4 minutes to mere seconds. That being said, I think it would be better if we aim at a temporary directory structure or queuing system to write to the Packages/Release file.

If you're up to writing that, awesome! Otherwise, I'll patch within the next few days.