usnistgov / CastVoteRecords

Common data format specification for cast vote records
https://pages.nist.gov/CastVoteRecords
Other
20 stars 2 forks source link

Individual CVR Files or Batch Manifests #45

Open adghayes opened 1 year ago

adghayes commented 1 year ago

Organization Name: VotingWorks

Organization Type: 2, Manufacturer

Document (e.g., CastVoteRecords): CastVoteRecords

Reference (Include section and paragraph number): -

Comment (Include rationale for comment):

The Cast Vote Records CDF is, when exported from a scanner at the end of election day, one massive file. Including linked images, it can be gigabytes of data. For us, this introduces two significant problems:

1 - VVSG Backup Requirement Compatibility

VVSG 1.2-H:

The voting system must withstand, without loss of data, the failure of any data input or storage device. The intent of this requirement is to prevent votes from being permanently lost due to the failure of a storage device that contains votes. For example, if a scanner fails, the voting system must have the ability to swap in a replacement data input device without the losing cast vote records that were previously recorded by the failed scanner.

This means that we need to be exporting CVR data to some other storage medium continuously throughout a scanner's usage, not just at the end of the day. We could use some internal format, but ideally we would use the CVR CDF format. That way, the CVRs are immediately consumable by a tabulator.

The current format does not really allow a continuous export, as it's a single JSON (or XML) blob. We don't have a way to append a CVR to an existing CVR CDF file, as it's not a newline delimited format. And due to the privacy issue of revealing the order of CVRs cast, appending would not be privacy-conscious anyway.

2 - Export Time

Because we can't export to CDF files incrementally, it all has to be exported at the end of the night. This can take a very long time when there's gigabytes of data and standard quality storage mediums. The export time is another reason to find some way to export continuously.

Suggested Change:

Allow CVR Data Distributed Across Files

Ultimately, this suggestion is similar to the suggestion in https://github.com/usnistgov/CastVoteRecords/issues/30. We need a way to place some CVRs into their own files - that way they can be exported discretely. To make that work, we need some sort of CVR Manifest or Batch Manifest file that links to the various CVRs or at least their containing directories.

There are a lot of implementation paths to this. Roughly speaking, I think there could be an "individual CVR" file format that is only the CVR.CVR data. There could be a separate or related "batch of CVRs" file format which contains the CVRs and some batch metadata. For simplicity, these could be one file format with some options at the top-level. Then, there would have to be a manifest file or manifest section in the metadata file which also contains the CVR.Election metadata and other top-level fields.

Why This Matter

We'd really like to stick to the CDF format as faithfully as possible, but the current format of one JSON blob + linked files for a day's worth of scanning will not work for our purposes. We're hoping we can create some sort of CDF extension that works for our use cases.


Organization Type: 1 = Federal, 2 = Industry, 3 = Academia, 4 = Self, 5 = Other

adghayes commented 1 year ago

Point of Clarification

To export individual CVRs or batches, we could just export each CVR or batch as its own CDF compliant file. I don't think this is ideal because:

  1. It de-normalizes and duplicates a lot of election metadata
  2. The overall group of CVR files still has no formal CDF description (how to relate the different files)
JDziurlaj commented 1 year ago

Hi @adghayes, we are looking into this!

adghayes commented 1 year ago

@JDziurlaj great! Thank you! Please do involve us or request more detail if that would be helpful, we'd be excited to help.

JDziurlaj commented 1 year ago

To restate the problem space:

The first issue, "VVSG Backward Requirement Compatibility" refers to VVSG 1.2-H

The voting system must withstand, without loss of data, the failure of any data input or storage device.\ The intent of this requirement is to prevent votes from being permanently lost due to the failure of a storage device that contains votes. For example, if a scanner fails, the voting system must have the ability to swap in a replacement data input device without the losing cast vote records that were previously recorded by the failed scanner.

It sounds like you are interpreting this as requiring the scanner to write out each cast vote record as soon as it can be constructed, i.e., from an in-memory representation to a permanent storage location.

The second issue "Export Time", is the assertion that the structure of the CDFs are not well suited to export as a single file. You say the expected raw CVR data is several gigabytes and would prefer to export each CVR as its own file.

Analysis

The CVR CDF supports export of multiple CVRs per CDF instance (e.g., a file). The intended use is for all CVRs for a vote capture device to exist in a single file, as it minimizes overhead.

We discuss our potential approaches to solving issues (1) and (2) below:

Approach 1: Use proprietary format for CVRs

This approach is for implementers to devise their own format for internal CVR storage. There is no requirement to use the CVR CDF for internal representation.

Pros:

Cons:

Approach 2: Generate separate NIST CVR 1.0 instance for each individual CVR

This approach is for implementer's to use the CVR CDF such that there is a one to one correspondence between CVRs and CVR CDF instances (i.e. files).

Pros:

Cons:

Approach 3a: Use XIncludes

Xincludes are essentially inclusions as found in programming languages. One primary file would be built containing includes to the other files. When processed, the inclusions would be dereferenced and a single logical CVR would be constructed.

e.g.

<CastVoteRecordReport xmlns=http://itl.nist.gov/ns/voting/1500-103/v1 xmlns:xi=”http://www.w3.org/2001/XInclude”>
    <xi:include href=”cvr_fragment_1.xml”/>
    <xi:include href=”cvr_fragment_2.xml”/>
    <xi:include href=”cvr_fragment_3.xml”/>
               …
</CastVoteRecordReport>

Would any of these approaches work for you?

benadida commented 1 year ago

Thanks for these suggestions @JDziurlaj .

I want to pop up a level: I think we may have a problem on our hands if, by following the VVSG2 standard, we are forced to either:

I understand, of course, that there are good reasons to resist changes to the standard, but I want to suggest that maybe the standard has been frozen too soon. We wouldn't have been able to contribute this input to the standard until we attempted this implementation – and thus it seems premature to freeze the standard before there are at least two successful compliant implementations.

The only way we can meet the VVSG2 requirements, if CVR CDF remains unchanged, is to not comply with CVR CDF. That seems like the worst possible outcome for the standard.