add flag to optionally generate structured output

[tomreitz]

To use lightbeam in an automated data pipeline it would be useful to have it generate structured output noting the number of records sent and skipped by resource and status code, the run timestamp, and possibly other information. Open questions include:

1. what should the flag be? perhaps --results-file output.json
2. what should the output format be? (YAML, JSON, something else?)
3. besides the items noted above, what else should be included in the results? perhaps the Ed-Fi API base_url? anything else?

Tagging @jayckaiser for thoughts.

[tomreitz]

Based on our conversation today, I'm proposing that

• lightbeam's structured output is enabled with a cli flag --results-file ./results.json
• output format is JSON, like this:

  {
   "timestamp": "2023-05-15 02:04:18Z00",
   "api_url": "https://edfi3.somedistrict.org/api/",
   "source_dir": "/path/to/some/dir/",
   "resources": {
     "studentSchoolAssociations": {
       "records_processed": 100,
       "records_skipped" :10,
       "records_attempted": 90,
       "records_success": 75,
       "records_failed": 15,
       "failed_statuses": {
         "409": {
           "messages": [
             {
               "message": "Invalid StudentReference",
               "count": 6,
               "line_numbers": [3, 7, 8, 31, 64, 88]
             },
             {
               "message": "Invalid SchoolReference",
               "count": 9,
               "lines": [5, 9, 12, 13, 26, 27, 29, 30, 56]
             },
           ]
         },
         ...
       }
     },
     ...
   }
   "total_records_processed": 100,
   "total_records_skipped": 10,
   "total_records_attempted": 90,
   "total_records_success": 75,
   "total_records_failed": 15
  }

  Is this too verbose? i.e., do we need the total_ across all resources?

We also discussed the exit codes lightbeam should return in different circumstances, to trigger the AirFlow BashOperator task to raise an AirflowSkipException or AirflowException (per the Airflow docs). Is it correct that

1. if total_records_skipped == total_records_processed, lightbeam should return exit code 99? (so the Airflow task skips)
2. if total_records_failed>0 lightbeam should return exit code 1? (so the Airflow task fails)

One other issue has to do with reporting line numbers: lightbeam can process either a single JSONL file (source_dir/students.jsonl) or an entire directory of JSONL files (source_dir/students/*.jsonl). In the latter case, how would line numbers be determined/interpreted?

Finally, one tiny question, should line numbers be 0-based or 1-based?

[jayckaiser]

Some thoughts on this one as well:

• As with Earthbeam logging, let's consider renaming timestamp to a more descriptive name that says whether that time is when the run started vs ended.
• As with Earthbeam logging, we can use exit code 99 to reflect a skipped run, but if there's a more transparent/generic exit-code to mark no action done, we may want to consider using that instead.
• I think we can narrow down the "totals" to remove total_records_attempted. Is there any case where a record that is not skipped is not attempted?
• Regardless of whether we 0- or 1-index the failed lines, I think we need to create an accompanying script that takes a file and a list of line numbers and outputs the failures to the console. I don't see a world where manually searching through the files by line number will be easy for the ISS team.
• As for whether to 0- or 1-index the failed lines, that's a good question. Indexing from 1 will make more sense when opening the file manually, but indexing from 0 will make more sense if we consider the header row being line 0. Regardless of what we pick, we need to clearly communicate to the ISS team which one was chosen.
• To deal with multiple files for a given resource, unfortunately we may have to nest the payload further to specify the file alongside line numbers. Something like this:

  {
  "message": "Invalid StudentReference",
  "total_count": 10,
  "files": [
  {
    "file": "{FILENAME1}",
    "count": 6,
    "line_numbers": [3, 7, 8, 31, 64, 88]
  },
  {
    "file": "{FILENAME2}",
    "count": 4,
    "line_numbers": [2, 6, 10, 35]
  },
  ]
  }

[tomreitz]

Closing this issue as it was implemented in this PR.