search

ncss-tech / lab-data-delivery

Re-working the LIMS + NASIS data integration process for snapshot and future web-service based delivery.
6 stars 1 forks source link

some columns in the FGDB / text files are all NA (KSSL Data) #6

Closed dylanbeaudette closed 7 years ago

dylanbeaudette commented 7 years ago

Some of the more important values that appear to be entirely NA, for example:

  1. Carbon_and_Extractions:
    • oc
    • n_tot
    • fe_dith
    • fe_ox

A complete listing can be found by looking over this file. Columns specified as TEXT when known to be numeric are likely wrong.

I have confirmed that the FGDB and resulting textfiles have this problem.

dylanbeaudette commented 7 years ago

This bug may be worse than I suspected, most of the tot_c records are NULL, but contain data in a previous Access-based snapshot.

See records associated with pedon_key values:

'5902', '5902', '5902', '5902', '5902', '5902', '5902', '5909', '5909', '5909', '5909', '5909', '5909', '5909', '5909', '5909', '5912', '5912', '5912', '5912', '5912', '6486', '6486', '6486', '6486', '6486', '6486', '6486', '6486', '6486', '6486', '6489', '6489', '6489', '6489', '6489', '6489', '6489', '6489', '6493', '6493', '6493', '6493', '6493', '6493', '6493', '6493', '8515', '8515', '8515', '8515', '8515', '8515', '9585', '9585', '9585', '9585', '9585', '9585', '9585', '9585', '9585', '9585', '9664', '9664', '9664', '9664', '9664', '9664', '9664', '9664', '9664', '11386', '11386', '11386', '11386', '11386', '11386', '11386', '11386', '11512', '11512', '11512', '11512', '11512', '11512', '11512', '11512', '11512', '11513', '11513', '11513', '11513', '11513', '11513', '11513', '11513', '13159', '13159', '13159', '13159', '13159', '13159', '13159', '24929', '24929', '24929', '24929', '24929', '24929', '24929', '24929', '24929', '24929', '24929', '24929', '24929', '24929'
jneme910 commented 7 years ago

Is this from the NASIS pedon export tool or the data straight from KSSL? I can check the data in NASIS tomorrow and see if its there if its from the pedon export tool?

dylanbeaudette commented 7 years ago

This issue is specific to the KSSL characterization data. I think that Adolfo spotted the error and is re-making the text files shortly.

dylanbeaudette commented 7 years ago

Looks like the problem has been resolved with the latest set of text files from Adolfo. This bug has inspired me to create some basic summaries with each snapshot that I produce, so that the diffs can be used to infer when something goes wrong.