Creating file doesn't limit to 10,000 lines

[tipputhynhean]

NACHA files are limited to 10,000 lines. But when creating file with localhost:8080/files/create , then result that view with localhost:8080/files/{fileID}/contents can be longer than 10,000 lines.

[adamdecaf]

There are a few points I'll clarify on how ACH works as-is today. Not saying they're correct, but filling in details.

The file creation endpoint POST /files/create purposefully doesn't validate the file. This is because some workflows will include additional batches in later calls and so rejecting a partial file breaks that flow. There's also the validate endpoint for when you want to ensure a file will pass at an FI.

Our current (f *File) Validate() error function doesn't compute the line length. We only do this when merging files as that was the first scenario we ran into with files that bumped up against the limit.

We should introduce a better line counter with internal methods on the File, Batcher, EntryDetail, etc types so we can be smarter about when a file is too long. Currently we write the file out (which is pretty costly).

This is a bug in that we should be rejecting files once they exceed the limit, but what would you expect these endpoints to return once a file is too large? Reject additional batches? Should GET /files/:id/contents return an error code?

I can see how someone would want multiple files returned with the contents endpoint, but that's not really possible to support unless we changed the response body.

[adamdecaf]

@tipputhynhean I assume you were able to work around this by creating smaller files?

[tipputhynhean]

@adamdecaf i haven't been able to do that yet. So currently the only place that will compute the line length is the merging files?

I think having no limit on files/create is fine, but is there a way to split the file to fit the limit length? I see there's a /files/{fileID}/segment, I haven't test it yet, but from the docs this is only separating debit & credit.

[adamdecaf]

i haven't been able to do that yet. So currently the only place that will compute the line length is the merging files?

Yea. The code needs to handle a File being too long in Create()/Validate().

I think having no limit on files/create is fine, but is there a way to split the file to fit the limit length? I see there's a /files/{fileID}/segment, I haven't test it yet, but from the docs this is only separating debit & credit.

Segmenting files is typically done when the vendor/ODFI requires debits and credits in separate files. I've not considered that as a solution to files over the 10k line limit.

There's no endpoint for splitting files like MergeFiles in the HTTP server.

I was assuming your code has a for loop iterating on the same file to add batches/entries and suggesting you create multiple files as a temporary workaround for the library's current limitations. We do this in PayGate by calling MergeFiles prior to upload.

[tipputhynhean]

@adamdecaf Is there a way to call the merge file endpoint on the docker image? I can't find it here https://moov-io.github.io/ach/api

[adamdecaf]

That endpoint isn't currently supported. What would you expect it to look like?

Perhaps something like this?

POST /files/merge 
{
   "fileIDs": ["...", "...", "..."]
}

[tipputhynhean]

That endpoint isn't currently supported. What would you expect it to look like?

Perhaps something like this?

POST /files/merge 
{
   "fileIDs": ["...", "...", "..."]
}

that would be awesome!

[brookekline]

For historical reference, maximum number of lines was originally handled in reader. I believe the original concept was to create the file, then read it, which by doing so would validate (or manually call validate after reading).

var ( // maxLines is the maximum number of lines a file can have. It is limited by the // EntryAddendaCount field which has 8 digits, and the BatchCount field which has // 6 digits in the File Control Record. So we can have at most the 2 file records, // 2 records for each of 10^6 batches, 10^8 entry and addenda records, and 8 lines // of 9's to round up to the nearest multiple of 10. maxLines = 2 + 2000000 + 100000000 + 8 )

// Read reads each line of the ACH file and defines which parser to use based on the first character // of each line. It also enforces ACH formatting rules and returns the appropriate error if issues are found. // // A parsed file may not be valid and callers should ensure the file is valid with Validate() // and tabulate the file with Create(). Invalid files may be rejected by other Financial Institutions or ACH tools.

[adamdecaf]

Closing since this issue looks mostly resolved. Please let us know if you run into other issues.