Audit log missing entries

[lyricnz]

The audit log from a run last night appears to be missing entries:

• revolver ran at 6pm AEST (0600Z), and stopped a bunch of EC2 in one account, which can be seen from audit-console log.

• but these are missing completely from the audit.log in S3 at the same time

Suspect race-condition in appending to the S3 audit log? Multiple accounts are running at the same time, and the S3 file writer doesn't have native/atomic file append, so fakes this by reading contents + appending + writing, which could result in lost-writes.

[lyricnz]

Technically this can happen with files too, not just S3, since fs.writeFile() is not atomic.

[lyricnz]

@abukharov thoughts?

• a mutex (on output filename)
• buffering all the output and writing at the end
• saying "just don't do that" and making users configure unique output files for each account+region? Which makes for ugly+fragmented audit logs:
  • audit.aws-whatever-account-name-nonprod.ap-southeast-2.csv
  • audit.aws-whatever-account-name-nonprod.ap-southeast-4.csv
  • audit.aws-whatever-account-name-nonprod.ap-somewhere-2.csv
  • ...

[lyricnz]

For emergency fix, I split the files to be unique by account+region, but this makes for rather messy output directory/S3 bucket.