Products - output of tidy data?

[rdstern]

This has been a regular request of mine. This is to output data in what seems to now be called a "tidy" shape. Or it could be headed for R-Instat. In CLIMSOFT 3.2 data could be output in columns by Station and Date and Element. There would be 4 columns, namely Station, then Date (which could also be Date and time, or could be in 3 columns itself if daily, i.e. Year, Month, Day), then the element number and then the value. Perhaps 5 columns if the flag information was output also.

That's close but it isn't fully "tidy", because it is odd to have rain (mm) and tmax (degrees) in the same column.

So Albert's Version 3 had the really tidy shape where each element was in its own column. I thought Samuel showed me that this output shape was also in CLIMSOFT. That's the shape we use in R-Instat and it also reads easily into any other Statistical Package.

But I don't see it listed in the current Products > Output for other Applications menu. It may be there under another name? Otherwise it would be good to add it. Would that be possible in time for Rwanda?

[mhabimana]

@rdstern This is exactly (tidy)how Climsoft output the product file. I think you need to try this under the product,daily .

[rdstern]

@mhabimana Great. I was looking under Products > Output for Other Applications. I have now seen it under Products > Data > Daily. It works fine.

Related questions, that I know must be equally straightforward. In the tutorial I did an inventory for all 12 stations for Precip, Tmax and Tmin. I did it from 1940 to 2018, because I don't yet know how to find the extent of the data for each station. I saw that Matopos, for example, only has data in 2000 for those elements.

How do I find what elements have been entered for Matopos?
And, for an element, say Tmax, how do I find which stations have Tmax in the database?

Sorry to ask such simple questions.

[mhabimana]

@rdstern Thanks, the simplest way is to use administration menu, then update observations, select observationfinal and then select the station id, the element id and the period and then view. If you are good at using SQL queries, you can query the database from the back end as well.

[rdstern]

Thanks. I tried Administration Menu > Update Observations

a) It looks a perfect dialogue for the task I asked about. b) So I used Observation Final and then clicked View. It has no data. I had hoped list all the data, but in a convenient table view. Or at least say 17,000 records from 12 stations, etc. So tell me something about the data in the database? c) So as you say, I need to put in the Station code and the Element id d) But my question above is which elements were measured at Matopos. So do I have to try each element in turn? There are a lot of elements. e) Why are there no pull-downs here for the Station and element.? f) What would be good in this dialogue is if I put in the Station id (from the pull-down list) and then it automatically indicates the elements that were at that station in the Element id field.

You say that is the simplest way. As the simplest doesn't seem to work - can you suggest other ways?

As an administrator it is surely important to know what data are in the initial database (so I know what work is pending. Then to be able easily to have information on what is in the actual database, e.g. for Matopos to know quickly what is there in terms of the elements and the dates for each element? That's what I think of as an inventory, which CLICOM did quite well. (The inventory here is only given in Excel, and then you also seem to have to know what you want the inventory of.

The team here are asking simple questions like how many stations are there, how many records for the daily data (in each database) and then slightly more detailed as above.