At some point, this stops being about moving SCC data into a better form, and starts being about actually improving the data and metadata. When is that? Do we really want to provide CSV that's just the fixed-width data as CSV? Or JSON that's just the fixed-width data as JSON? Do we move the cleanup process earlier in the pipeline (before CSV is generated at all), somewhere in the middle (creating raw CSV files and then improved ones), or at the end (after CSV, JSON, and whatever else is generated)?
At some point, this stops being about moving SCC data into a better form, and starts being about actually improving the data and metadata. When is that? Do we really want to provide CSV that's just the fixed-width data as CSV? Or JSON that's just the fixed-width data as JSON? Do we move the cleanup process earlier in the pipeline (before CSV is generated at all), somewhere in the middle (creating raw CSV files and then improved ones), or at the end (after CSV, JSON, and whatever else is generated)?