[Publisher] Toggle between metric reporting frequencies in "Explore Data" tab

lilidworkin commented 5 months ago

Agencies want to be able to view both CY and FY data on the same chart in the Explore Data page. Rather than allowing mixed frequency data to show on the same chart, we want to add some sort of toggle on the Explore Data page so that agencies can switch back and forth between CY, FY, and monthly charts.

[Implementation Notes from @mxosman]

Reference: Figma Design

Quick note: all of the notes on both panels are relevant, but since Explore Data page is still in the old design, please reference the right panel for how it should look.
Additional things included are:
- updating the copy for one of the time period dropdown options ("All" to "All time")
- updating the copy for text below the chart
- removing the blue banner

Relevant files: MetricsDataChart.tsx, DatapointsStore.ts

The dropdown will contain all of the unique frequencies extracted from all of the datapoints an agency produced.

Example: if an agency has recorded data for a monthly metric, calendar year metric (starting month in January), fiscal year metric (starting month in July), and an annual metric with a custom starting month (let's say March) -- the dropdown should include the following options: "Monthly", "Calendar Year", "Fiscal Year", "Annual: March".

Since we get a list of all datapoints, we can loop through them and group them in their respective unique frequency bucket and give users the option to filter the chart data by frequency.

Please connect with me (@mxosman) if you have any questions at all (or if I can help you ramp up)!

mxosman commented 5 months ago

Hi @nasaownsky - please always feel free to give me feedback on these tasks - whether they give you enough information, or not enough information, or if you think of any additional information I can include that would help make your life easier and reduce ramp up time as you switch contexts!

nasaownsky commented 5 months ago

Hi @mxosman! So I trying figure out this frequency breakdown. I see that datapoints have "frequency" parameter, that in some metrics has "ANNUAL" value, and in some "MONTHLY". Could you please explain how exactly this filtering should work in conjunction with this parameter?

mxosman commented 5 months ago

Great question! So, right now the Explore Data will only show the datapoints of the currently set frequency for a metric (e.g. if an agency sets Funding to ANNUAL (Calendar Year), then the Explore Data will only show ANNUAL Calendar Year datapoints. Before that change, it used to show a mix of frequencies (if an agency had a monthly datapoint for Funding, it would appear on the charts, and if they also had an annual datapoint for Funding, it would show up too).

Now, we want to go through all of the datapoints to determine the different types of frequencies we have and give agencies an option to select from a dropdown to view only MONTHLY datapoints, or only ANNUAL datapoints, or only ANNUAL: datapoints -- if that makes sense. If the datapoints for a metric are only Monthly and Annual (Fiscal Year) (for example), then those are the only two dropdown options. And if they select one, the chart updates to only show the datapoints that belong to that currently selected frequency.

When it comes to the datapoints and filtering, the MONTHLY frequency is the simplest - filter out all datapoints that are not monthly.

The ANNUAL frequency datapoints need the starting_month as well to group them appropriately (because you can have an ANNUAL frequency that begins in January (Calendar Year), or July (Fiscal Year) or some custom starting month like April.

Does this help clarify at all? Please let me know if it doesn't and I'll clarify further!

nasaownsky commented 5 months ago

So, does that mean that if the metric only has ANNUAL frequency datapoints they must be filtered only by annual options based on different starting months; and if it only has MONTHLY datapoints they must be only filtered with monthly option?

mxosman commented 5 months ago

So, does that mean that if the metric only has ANNUAL frequency datapoints they must be filtered only by annual options based on different starting months; and if it only has MONTHLY datapoints they must be only filtered with monthly option?

Oo - excellent question! I think you're spot on - but let me clarify incase I misunderstood!

If there is only one unique set of frequencies, then I recommend not showing a dropdown at all. This applies for when datapoints are only ALL monthly or only ALL annually (with the same starting_month).

For annual frequency, if there are datapoints with different starting_months, then we want to show a dropdown that lets users filter between the different types of annual frequencies.

nasaownsky commented 5 months ago

@mxosman So digging more through code and visual representation I see that in "Explore Data" tab there are datapoints with either only ANNUAL or only MONTHLY frequencies. Moreover, those that have an annual frequency always have the same starting month. Is this just problem with the amount and shape of the data showed in staging, or the data gets filtered out somewhere this way and I should change that? If the second is true, then maybe you know where is exactly this filtering method?

Thank you for your dedication and help in clarifying this subject to me, it is much tougher that I thought!

mxosman commented 5 months ago

@mxosman So digging more through code and visual representation I see that in "Explore Data" tab there are datapoints with either only ANNUAL or only MONTHLY frequencies. Moreover, those that have an annual frequency always have the same starting month. Is this just problem with the amount and shape of the data showed in staging, or the data gets filtered out somewhere this way and I should change that? If the second is true, then maybe you know where is exactly this filtering method?

Thank you for your dedication and help in clarifying this subject to me, it is much tougher that I thought!

Oo - good call, @nasaownsky - sorry, I didn't clarify this earlier. But, currently the metrics are being filtered to match the recently set metric frequency, and the filtering is happening here: https://github.com/Recidiviz/justice-counts/blob/4492eda02b1f1741d43398addf77467544375dc8/publisher/src/stores/DatapointsStore.ts#L84-L86

Here's an example of the data structure coming from the /datapoints endpoint for the same metric that has values in different frequencies:

        {
            "agency_name": null,
            "dimension_display_name": null,
            "disaggregation_display_name": null,
            "end_date": "Mon, 01 Jul 2024 00:00:00 GMT",
            "frequency": "ANNUAL",
            "id": 229062,
            "is_published": false,
            "metric_definition_key": "LAW_ENFORCEMENT_FUNDING",
            "metric_display_name": "Funding",
            "old_value": null,
            "report_id": 20209,
            "start_date": "Sat, 01 Jul 2023 00:00:00 GMT",
            "value": 500
        },
        {
            "agency_name": null,
            "dimension_display_name": null,
            "disaggregation_display_name": null,
            "end_date": "Mon, 01 Jan 2024 00:00:00 GMT",
            "frequency": "ANNUAL",
            "id": 229055,
            "is_published": false,
            "metric_definition_key": "LAW_ENFORCEMENT_FUNDING",
            "metric_display_name": "Funding",
            "old_value": null,
            "report_id": 20206,
            "start_date": "Sun, 01 Jan 2023 00:00:00 GMT",
            "value": 200
        },
        {
            "agency_name": null,
            "dimension_display_name": null,
            "disaggregation_display_name": null,
            "end_date": "Sat, 01 Jun 2024 00:00:00 GMT",
            "frequency": "MONTHLY",
            "id": 229048,
            "is_published": false,
            "metric_definition_key": "LAW_ENFORCEMENT_FUNDING",
            "metric_display_name": "Funding",
            "old_value": null,
            "report_id": 20195,
            "start_date": "Wed, 01 May 2024 00:00:00 GMT",
            "value": 244
        },

Recidiviz / justice-counts

[Publisher] Toggle between metric reporting frequencies in "Explore Data" tab #1368