Help and video on tranfer of data from old Instat

[rdstern]

In our climatic work we have a demand for support to transfer daily data from the old Instat into the new format. There are examples of the old format in the Instat library and I wonder about a text and video on the steps that are needed. The request is also to automate the procedure, as some users have many data sets in this format. So this is also an example where we could give instructions on how to produce and run a script file. That is useful in its own right. So here are the steps - given briefly.

1. Open from Library > Instat > Browse > Climatic > Climatic guide > Kabete. (Possibly include the option to say 9999 is missing.)
2. Prepare > Generate > Regular Sequence. Call the column Day. Accept the default which is (1 ... 366)
3. Prepare > Column: Reshape > Stack . All columns except the last to stack. Last column (Day) is Carry column. Call one column Year and the other one Rain.
4. In stacked data now. Get rid of Y in the Year column. Can use Prepare > Column: Text > Find/Replace then Replace All. Choose Year column and replace Y by nothing.
5. The new column is character. Make it numeric.
6. Now have to delete all rows where (rain == 9988). Can (currently do this with a filter. Then select all the filtered data (ctrl A) and remove rows. Then remove the filter. (Easier now is probably to set the filter and put the subset into a new data frame. Then remove the original data frame.)
7. Now need to change the missing value code of 9999 into Missing, if that was not done earlier, see step 1 above.) Could now delete the extra Year column and rearrange the columns, using Prepare > Data Frame > Reorder Columns.

Once that is smooth, can do this where you press script button before OK each time. You may have to copy some commands from the output. Then you have a script file to (possibly adapt) and use with other sets of data.

This might be 2 scripts and then 2 videos, i.e. first to do the task interactively. Then to do it with the commands and run the commands on a second data frame, e.g Katumani.

[trottingafrican]

Hi Roger, Thanks for sharing this. We have looked at it and tried to work through all the procedures. What now remains is to come up with a working script that can be used for video recording. We shall share it once we are done with it. In the second last paragraph, you stated that we can press the button to copy some commands before OK each time. What do you mean by this?

[rdstern]

Patrick, you asked in relation to the transfer of data from the old Instat: "we can press the button to copy some commands before OK each time. What do you mean by this?"

The bottom 5 buttons are the same in each dialogue. On the left is the OK button, and on the right is a button called Script. That's the one to press before OK.

[dannyparsons]

Alternatively you can run through the steps and then get all the commands in the Log Window

[dannyparsons]

Before you get too far with the script and video, this has made me think that it would be quite easy to have our own function which does these steps and then implement it in a dialog. In workshops this process was quite confusing for users, especially as there's a lot of steps and this is the first thing they have to do. The function we write could even be able to accept multiple files to produce either one data frame for each file or a merged file if it's multiple stations/elements. I think this is the better solution for users who will have many of these files in the same format.

It still might be useful to have video(s) on this as an exercise in data manipulation, but for climate data users I think we can make it much easier and powerful.

[mmumbo]

@dannyparsons Yes indeed the process was confusing especially when you want to copy each script before click OK. We are now suggesting that we we go ahead and produce a video, then we copy all the script from the output window, do the editing on the notepad (just changing the data frame name) then later run on the script window. At the moment the script window does not automate the find and replace option. We believe this can work for now. What do you think about that?

[dannyparsons]

The video could still be useful as its a good exercise and currently needed, although I wouldn't spend huge amounts of time on it because once we implement it through a dialog this will be far easier for people to do that way.