replace plyr / reshape2 functionality with data.table equivalent

[dylanbeaudette]

Keeping track of changes / testing related to data.table replacement of plyr / reshape functionality. Each change should be checked and a new test added if there isn't sufficient coverage. Handy bash script:

grep 'join(' * | awk '{print "  - [ ] `"$1"`"}' | tr -d ':.R' | sort | uniq

plyr

join

• [x] fetchHenry
• [x] fetchNASIS_pedons
• [x] fetchNASIS_report
• [x] fetchPedonPC
• [x] getHzErrorsNASIS
• [x] getHzErrorsPedonPC
• [x] get_colors_from_pedon_db
• [x] get_component_data_from_NASIS_db
• [x] get_extended_data_from_NASIS_db
• [x] get_hz_data_from_pedon_db

ddply

• [x] fetchHenry
• [x] fetchNASISLabData
• [x] fetchNASIS_components
• [x] fetchNASIS_report
• [x] fetchPedonPC
• [x] fetchSCAN
• [x] getHzErrorsNASIS
• [x] getHzErrorsPedonPC
• [x] get_colors_from_pedon_db
• [x] get_component_from_LIMS
• [x] get_component_from_SDA
• [x] get_phlabresults_data_from_NASIS_db
• [x] simplifyArtifactData

ldply

• [x] SDA-spatial
• [x] fetchSCAN

reshape2

reshape2::dcast → data.table::dcast

• [x] fetchHenry
• [x] simplfyFragmentData
• [x] simplifyArtifactData

Note that value argument becomes value.var.

melt → data.table::melt

• [x] fetchSCAN

Caveats

• It appears that data.table::melt will not preserve factor type / levels unless asked to do so explicitly via value.factor argument. A reasonable solution is to set it after the reshape operation when appropriate.
• id.vars should not contain duplicated column names
• melt.data.table expects that all measure.vars are of the same type (e.g. all numeric) otherwise warning issued

[dylanbeaudette]

Nice work on those last changes. Looks like all that is left is get_colors_from_pedon_db?

[brownag]

Thanks, yes that was all that was left, now done in https://github.com/ncss-tech/soilDB/commit/66499a7b25dca70425de4924712386c1328da2f8

The conversion from plyr https://github.com/ncss-tech/soilDB/commit/2ad40e702ed8180afe261c34c0c1075a610e638e was fine but I was unable to verify the correctness of result due to some other dangling, previously un-detected issues with (un)coded moisture state resulting in a 0-length results even with data in the DB.

Testing any MS Access stuff takes some finagling on my work machine because I am still stuck with only 32-bit MS Access drivers (boo)

I also took this opportunity to convert from the (deprecated) mix_and_clean_colors to the newer estimateColorMixture. So now the mixing should be done in LAB (woo) and I have verified that fetchPedonPC / get_colors_from_pedon_db are working as expected.