Improving the inventory of missing values

[rdstern]

I suggest a major enhancement to the Climatic > Check Data > Inventory dialogue.

Here is roughly my suggestion for the new dialogue. It adds a Summary button at the top of the current inventory dialogue. It ceases to be called Inventory Graph, just Inventory.

The top of the dialogue is then the same as for the graph, including the Optional Station receiver (only half visible on the rhs. The ticks then represent check-boxes. in the dialogue and the circles are then radio buttons. There are 2 further controls to add. Starting Month in Year (default 1, i.e. January). An additional Days (more than with default 1 consecutive). And I propose that Days be the default tick here - not Years or Months. So the order should be reversed, with Days, then Complete Months then Complete Years.

The dialogue should be easy to construct. However, I can't find a function in any package that is even close to what I would like for the summary inventory. So, there is a reasonalby major exercise needed in R to write that function. I write below what I would like the function to do.

I first describe the proposed output.

It operates on a tidy climatic dataset, which will therefore possibly have multiple Stations, (in a Station Variable) and multiple Elements - each in their own variable.

So the function will have 3 initial arguments, namely Date, Elements (a list?), and Stations. Date and Elements must be provided, but Stations is optional.

There is a Summary argument and a Details argument. Default for Summary is TRUE and for Details is FALSE. If summary is TRUE then there are 2 further arguments called Start and End, with default for Start TRUE and End False. (If summary is not selected, then the settings on these arguments is ignored.) If Details is selected there is an Order argument. This could be TRUE or FALSE. Perhaps clearer if it is Element or Station - to give the option of the detailed results being by element or by station. (Perhaps this may also apply to the Summary output too?

If the Details is selected there is a further block of controls with (currently) Year, Month, Day. In the dialogue there can also be Hour, Minute, Second, but these are disabled.

In the summary output there could be the following:

One feature I have not found in any of the existing functions on the number and % missing is to be able to ignore missing values for some elements at the start of the data. This is important, because many stations started with just rainfall and added other elements later.

(The index of the first non-missing row is neatly given by start <- which.min(is.na(tmax)). Not sure how this will need to change for multiple Stations?)

When the details are asked for the default is Days selected. Then the results look like this:

If Months are selected then this just adds the number of completely missing months. And adding years is just for which complete years are missing. For example:

[lilyclements]

Have started work on this, however, I have some questions

1. Would you want this written with the instat_calculation system? Otherwise is tidyverse okay?
2. Can I just clarify that "Summary" gives summaries of missing values (number and percentage and complete years). What does "Details" count?
3. For Summary = TRUE, what would the End argument do?

[rdstern]

@dannyparsons I can't answer the first question - could you advise on

"Would you want this written with the instat_calculation system? Otherwise is tidyverseokay?"

Yes, when Summary is selected, then it gives just the first table above.

2. If Details is selected, then it gives the further summaries, shown in the tables below the Summaries table. Which ones would depend on what else is chosen.

3. If End is selected in the summary, then it ignores any sequence of missing values at the end of the data. So, if the data in the file was from 1935 to 2018, but for a particular element was only from 1958 to 1999, then it would ignore both the number of missing values up to 1958 and also the number after 1999. The settings of Start and End are ignored when giving details of the missing values.

[rdstern]

Danny's answer was first to "simply" program in R - so presumably using tidyverse. Then it can ba adapted for R-Instat.

[lilyclements]

Great, thank you for clarifying. I have got it to work for summary and details (year, month, day - not hours, minutes, seconds). It's currently in a function but only works for one element, I'm unsure how to get it to work for multiple but can look into this later.

I am a little unclear on the order option of TRUE and FALSE. What should these options alter?

Thanks

[lilyclements]

I'm still working on the code, however, it can be looked at for now (I hope!)

To do on the code:

• Currently only works for one element - need to implement it so that it works for more.
• Sort "Order" option - currently only works for one element, so this has not been set up
• Currently the Hour/Minute/Second options are not set up - @rdstern do you have a data set you could point me to so I can implement this?
• Currently the year/month/day works by radio buttons/combo box rather than a check box (hence, only one can be selected). I misinterpreted the diagram above. I'm not too sure what it means for multiple of these to be selected? Currently, if month is selected then April 1990 is different to April 1991
• Set up warnings/errors in the code

If there is anything else, then please let me know

[lilyclements]

Here is a design - this is for when summary is selected on the top radio buttons. Presumably the current set up is for when graph is selected? The bits in blue are only visible when certain check boxes are selected. Namely, the omit start and end options are visible when "Summary" is checked. The other options are visible when "Details" is checked.

[lilyclements]

As far as I'm aware, the inventory code is written and working. It is currently written using code in R - a copy of the code is given here