New command: statistics

[jpmckinney]

It can be useful to report out some order statistics and distributions that are relevant to indicators. For example:

• Distribution of procurementMethod codes: so that the user can evaluate if the distribution of open, selective, limited, direct conforms to their knowledge of the procurement market
  • From user research: "A methodology should also come with clear risk warnings for instance the use of certain fields. Are there some fields that we know are problematic when it comes to bias in the data? (e.g. Could there be a bias toward using 'selective' instead of 'limited' in procurementMethod?)"

We might also consider reporting:

• Some priority quality issues (e.g. incoherent dates).
• Outliers. If there is demand, we can also change indicators command to ignore outliers.
• Order statistics (possibly per procurement method) to assist the user in setting threshold values

[jpmckinney]

Edit: Integrated this comment into #55

---

Thinking through how users will edit the configuration file while preparing the data:

1. Edit configuration file with known fixes.
2. Run prepare command.
3. See remaining warnings in the output.
4. Repeat 1-3 until acceptable number of warnings.

So, some (or all) of this command should be part of the prepare command.

[jpmckinney]

Having worked on and thought through prepare some more, we want to keep it focused on reporting things that the user can correct via configuration.

So for this issue, we can solve part of it via documentation (e.g. the example from the user research). Otherwise, it's not yet clear whether these statistics should be done in the library, in supporting notebooks, in BI, etc. So, setting as Post-MVP.

[jpmckinney]

Partly documented in general workflow, so removing that label. Can add more later.