check-in counts api

[vjeffrey]

User Story

In order to support the desktop offering view, we need to provide a "daily check in" count. This means we need an API that returns:

• total count of desktops (nodes) that have checked in to Automate
• total count of desktops (nodes) that have checked in to Automate over the last 24 hours

Considerations

Since this is useful information that can be used in views that are not desktop-only, we should make sure we design this API in a generic way. Please refer to the epic description, q&a doc, and designs to validate requirements. Chat with @apriofrost @chef/superteam about questions. API endpoint should be discussed with team and agreed upon with UX and Product. Let's try to keep most discussion/conclusions documented in this issue so we can reference later

https://chef.invisionapp.com/share/E9W3TEFXV87#/screens/406241157_Desktop_Dashboard-V2-Changes_Selected

Aha! Link: https://chef.aha.io/features/SH-554

[vjeffrey]

NOTE: in the current designs, there is a view that provides count of nodes checked in per 24 hrs compared to total nodes; we may want to combine those two APIs? let's chat about it

[vjeffrey]

request/
days ago: 1
response/
[ {start date: 2020-03-09[time], end date: 2020-03-10[time], checked in: 45, total: 50} ]
request/
days ago: 3
response/
[ 
    {start date: 2020-03-07[time], end date: 2020-03-08[time], checked in: 30, total: 40},
    {start date: 2020-03-08[time], end date: 2020-03-09[time], checked in: 40, total: 50} 
    {start date: 2020-03-09[time], end date: 2020-03-10[time], checked in: 45, total: 50} 
]

discussed in slack about doing a request/response like the above

that would satisfy the data needs of:

open questions:

• api endpoint (cnfgmgmt/check-in-counts ??? what should this be. assuming a get endpoint.)
• verbosity: should we add a field for “not checked in” or something like that on the api response?

[lancewf]

Option 1: Daily Total: From the node-state index, for each day bucket find the number of nodes created before the end of the day. Subtract the number of nodes deleted before the end of the day.

Daily Check-in: From the converge-history indexes, get the number of unique nodes that have reports.

If we can not find away to create aggregations to calculate these values with only two or three requests then this method would have a slow response. This would be flexible in the start and end dates of the day bucket.

Option 2: Create a daily summary index. The day would start and end on UTC time. The index would store the total number of of nodes for each day and the total number of check-ins. This would be a faster response, but inflexalbe for the start and end of the day bucket.

I am going to look into Option 1 first to find out if we can create aggregations to collect these values.

[lancewf]

We currently do not have a way to add project filtering on to the check-in time series because we do not have project tags on the infra reports only on the current node summary index. We could do project filtering for the last 24 hours because we can pull that information from the node summary index.

Another problem is that when a node is deleted we do not delete or mark the reports in any way. So, the time series will not remove deleted nodes from the checked-in count.

[lancewf]

On second thought about the deleted nodes problem mentioned above, it should not be a problem that they are not removed from this historical runs indexes because after they are deleted those nodes will not be adding new runs to the indexes.

[vjeffrey]

curl --silent --insecure -H "api-token: $ACCEPTANCE_TOKEN" https://a2-local-inplace-upgrade-acceptance.cd.chef.co/api/v0/cfgmgmt/stats/checkin_counts_timeseries?days_ago=5

{"counts": [
{"start":"2020-03-21T23:00:00Z","end":"2020-03-22T22:59:59Z","count":0,"total":2},
{"start":"2020-03-22T23:00:00Z","end":"2020-03-23T22:59:59Z","count":0,"total":2},
{"start":"2020-03-23T23:00:00Z","end":"2020-03-24T22:59:59Z","count":0,"total":2},
{"start":"2020-03-24T23:00:00Z","end":"2020-03-25T22:59:59Z","count":0,"total":2},
{"start":"2020-03-25T23:00:00Z","end":"2020-03-26T22:59:59Z","count":1,"total":3}
]}