Add support for 'process_records' in the mapping, enabling customizat…

[jaraco]

..., enabling customization of which records to process.

This change allows any mapping to define a 'process_records' field in the mapping, which accepts an iterable of raw records and can perform arbitrary transformations on those records before they're passed through later stages in the pipeline.

I've used this feature in jaraco/csv2ofx@0560beb53d5db94506142298644a226b64c8a95f to do two things:

• Filter out the footer row, which is just a copy of the header row.
• Exclude records based on the payment field when certain card numbers appear.

Let me know what you think of this as a feature and if it seems suitable for general purpose use, what additional you would like to see to have it included.

[jaraco]

Any thoughts on this change?

[reubano]

I'd rather not add additional preprocessing. You can ignore NUM initial and trailing lines in a csv with tail -n +NUM transactions.csv | csv2ofx ... and head -n -NUM transactions.csv | csv2ofx ... respectively. Filtering is a use case I hadn't considered though, so I'll need to think about the best way to handle it.

[jaraco]

You can ignore NUM initial and trailing lines in a csv with tail

You can, but then the user has to supply that information with each invocation (you can't capture that requirement/solution in the mapping).

Filtering is a use case I hadn't considered though.

I have a concrete use-case that's unmet without this feature. It saddens me that after a year, I get a simple rejection that doesn't start to address the need.

What can I do to help address my needs? Is my only option to fork the project?

[reubano]

Stay tuned. I'm working on something that should address this.

[reubano]

@jaraco let me know if the most recent updates aren't sufficient for your needs.

[jaraco]

I've finally gotten around to trying this out again and I can confirm, these new features are satisfactory for the needs of the Amazon importer. Thanks!