How to get atomic writes with Log::Dispatch::File

[autarch]

Migrated from rt.cpan.org #108899 (status was 'open')

Requestors:

• SREZIC@cpan.org (@eserte)

Attachments:

• parallel_log_dispatch.pl

From srezic@cpan.org (@eserte) on 2015-11-13 13:42:10:

It seems that writes done from multiple processes to a shared file are only atomic if the file was opened with O_APPEND, at least on Linux and for reasonable string lengths. The attached script seems to confirm this statement --- with '>>' I get always 1000 lines of output, with '>' it's always less than 1000.

It would be nice if this could be mentioned in the Log::Dispatch::File documentation.

Some pointers about atomic writes:

• http://stackoverflow.com/questions/1154446/is-file-append-atomic-in-unix/
• http://article.gmane.org/gmane.linux.kernel/43445
• http://www.notthewizard.com/2014/06/17/are-files-appends-really-atomic/

[autarch]

From drolsky@cpan.org (@autarch) on 2015-11-13 23:16:28:

On Fri Nov 13 08:42:10 2015, SREZIC wrote:

It seems that writes done from multiple processes to a shared file are only atomic if the file was opened with O_APPEND, at least on Linux and for reasonable string lengths. The attached script seems to confirm this statement --- with '>>' I get always 1000 lines of output, with '>' it's always less than 1000.

It would be nice if this could be mentioned in the Log::Dispatch::File documentation.

There is another output, Log::Dispatch::File::Locked (https://metacpan.org/pod/Log::Dispatch::File::Locked) that is designed to share a file across processes.

I could add a pointer in the File docs to File::Locked.

[autarch]

From srezic@cpan.org (@eserte) on 2015-11-16 14:18:26:

On 2015-11-13 18:16:28, DROLSKY wrote:

On Fri Nov 13 08:42:10 2015, SREZIC wrote:

It seems that writes done from multiple processes to a shared file are only atomic if the file was opened with O_APPEND, at least on Linux and for reasonable string lengths. The attached script seems to confirm this statement --- with '>>' I get always 1000 lines of output, with '>' it's always less than 1000.

It would be nice if this could be mentioned in the Log::Dispatch::File documentation.

There is another output, Log::Dispatch::File::Locked (https://metacpan.org/pod/Log::Dispatch::File::Locked) that is designed to share a file across processes.

I could add a pointer in the File docs to File::Locked.

Locking is another possibility. But I think that file locks are less efficient than normal writes in O_APPEND mode (more syscalls involved, more switching between kernel and user space). Probably both approaches should be mentioned.

[autarch]

From drolsky@cpan.org (@autarch) on 2015-12-19 19:58:11:

On Mon Nov 16 09:18:26 2015, SREZIC wrote:

On 2015-11-13 18:16:28, DROLSKY wrote:

On Fri Nov 13 08:42:10 2015, SREZIC wrote:

It seems that writes done from multiple processes to a shared file are only atomic if the file was opened with O_APPEND, at least on Linux and for reasonable string lengths. The attached script seems to confirm this statement --- with '>>' I get always 1000 lines of output, with '>' it's always less than 1000.

It would be nice if this could be mentioned in the Log::Dispatch::File documentation.

There is another output, Log::Dispatch::File::Locked (https://metacpan.org/pod/Log::Dispatch::File::Locked) that is designed to share a file across processes.

I could add a pointer in the File docs to File::Locked.

Locking is another possibility. But I think that file locks are less efficient than normal writes in O_APPEND mode (more syscalls involved, more switching between kernel and user space). Probably both approaches should be mentioned.

Using O_APPEND and making sure that all your writes are under a certain size seems very fragile. I agree that using locks is going to be slower, but it's also not likely to just blow up entirely ;)

That said, I think any sane production app should probably be using syslog.