Refactor harvest_report to improve maintainability and test coverage

[aaron-collier]

https://codeclimate.com/github/sul-dlss/dlme-airflow/dlme_airflow/tasks/harvest_report.py

For context, this needs a major refactor. It was written initially piece by piece and run manually one collection at a time. It would fail from time to time. The code captures the general functionality needed but it needs tests, it probably has some bugs, and it could probably benefit from performance improvements.

[lwrubel]

Focusing on tests to start.

[aaron-collier]

@lwrubel one area of refactoring to think about is at https://github.com/sul-dlss/dlme-airflow/blob/main/dlme_airflow/tasks/harvest_report.py#L188

This is loading the whole ndjson file before moving on. These can be large files, anywhere from a couple of MBs to a couple of hundred MBs.

One bug I'm working on will strip blank lines on this load, but not focusing on the refactor there. Happy to discuss tomorrow.

[jacobthill]

Checklist:

• [x] agg_data_provider_collection_id seems to have a bug in the way average values are counted. AUB is getting 10.27 values per record. As far as I can tell this should be just 1 value per record, making the average also 1. The code might be counting every letter, instead of every word. The average is 4.0 for aims and the value in the field is always 'aims'.
• [ ] AUB is showing a long list of urls that did not resolve but they all resolve when trying them manually; we are not following re-directs in our code.
• [ ] The numbers of languages are very large (e.g. 144, 155). This field should indicate and average number of languages for the field. A number close to 2 shows that the field is mostly translated, exactly 2 would mean fully translated, large numbers like this are a good indication that there is a bug in the code.
• [ ] The crosswalk isn't catching all fields in the traject configs. This code should be easier to implement now that we are only concerned with mapping csv files. The pattern will always be to_field some_dlme_field, column('some_column_name'),... There are lines that look like 'wr_dc_rights' => [column('rights')], but I'm ok with ignoring these for now. I'm also ok with dropping the third column of this crosswalk so that it just shows in and out fields and doesn't try to explain transformations.