iecodebook: major extensions to [export] functionality

[bbdaniels]

iecodebook export is now majorly upgraded as a data handling command. This PR includes the following new features:

• [x] The ability to use data by specifying a file path as a primary argument to the command.

• [x] ~copy~ save: An option to save ~a new copy of~ the dataset in the same location as the codebook is placed. (Intended for things like Box/Git workflows and other setups where data may have to be bounced between storage and functional location.)

• [x] signature: An option to write a datasignature and, later, to ensure that the data is exactly the same as when it was last copied, otherwise throwing an error. (And its accompanying replace analogue, reset.)

• [x] verify: An option to verify the structure (not the contents) of dataset against a previously written codebook to ensure no changes have been made to the variables, their definitions, or, in the case of labelled variables, their value labels.

• [x] textonly: The ability to write a rudimentary plaintext codebook (for Git).

• [ ] And finally, trim(), which started it all, finally works in a reasonable way. Taking a list of do-files as its arguments, this option reads the contents of those do-files, and does its best to keep in the dataset only those variables which are mentioned in the do-file, before passing the data through to any of the above. This is to facilitate the rapid creation of "release" datasets accompanying specific analytical tasks. (It doesn't work with every possible case, particularly macros and loops, but it's pretty good.)

• [ ] Additional error handling has been included to detect and report invalid variable labels and value labels (containing special characters) which have been a common source of failure for writing the codebook.

This issue supersedes all other iecodebook issues and is important when it can be reviewed as these tools are majorly needed (by me, at least).

[kbjarkefur]

I have never used this command. Can you tell me what to run and what you want me to look at? Otherwise I am not sure what to do.

[bbdaniels]

Yeah hold on. I am trying to put it all into one big thing for the moment. I will write up detailed instructions after I run ad hoc checks to make sure nothing is screwed up. Thanks! On Jul 10, 2020, 3:49 PM -0400, Kristoffer Bjärkefur notifications@github.com, wrote:

I have never used this command. Can you tell me what to run and what you want me to look at? Otherwise I am not sure what to do. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.

[luizaandrade]

• [x] A When the use feature is used, the data remains open once the command finishes running. I think we should preserve whatever is open when the command is called and restore once it finishes running.
• [x] B Can the save option include an argument to specify the name of the dataset? It should probably also have a replace option. Something like save(filename, replace) would be great.
• [x] C I get the following error message when I try to use the option signature: file C:/Users/wb501238/Dropbox/WB/Analytics/DIME Analytics/Data Coordinator/iefieldkit/iecodebook/auto_export-sig.txt not found
• [x] D The trim option only works with backward slashes. I get this error when using forward slashes:
• [x] E The trim option also assumes that the do-files listed are .csv files if the extension is not included. Would be good to include a test that the right extension was specified. (I did not check that the output of trim does what it's supposed to yet)

[bbdaniels]

@luizaandrade cb9e96399dde7701dcf311cc64291a5cd9fe250d is a simple fix that addresses (D) above. I think it is an error with quotation marks, not forward slashes. Need you to confirm this one.

[bbdaniels]

(C) will be resolved by writing reset in the options since this is needed to write a new signature. 4753ba9e9717b7b83bffb92c10be816d61501bbe adds an error message to this effect.

[bbdaniels]

For (E) I now return an error in 46ce86bd17c86918f9afbdc3a30ace11e38ba816 if file extension is not specified or not .do. People can write out their file paths....

[bbdaniels]

(A) is resolved by storing data in memory before loading new data. d011be4345316ef4d07d429ed4589d98460736fb @luizaandrade can you confirm that this doesn't break any of the template functionality, especially in append. I couldn't break it at this stage but since template in particular may call this repeatedly I can imagine there might be a case where it fails.

[bbdaniels]

In c6126c3511208d029ef234b78d5f826d3441cf3b I resolve (B) by allowing an override of the default filename with saveas(). The replace option is global, is that ok?

[luizaandrade]

• I fixed a minor bug in 15539e2 and trim is working now.
• I fixed a minor bug in 9df5bc0 and saveas is working now. The only caveat is that you need to write the full file path, so it will not save in the same place as the codebook, but in the file path that you specify.