Testing edge hydro merge code

[DocOtak]

Original report by see (Bitbucket: sescher, GitHub: sescher).

---

Sorry: I dont know how to make it look pretty in this box...

Go to ghdc:/Users/sharon/TRYIT.2013-08-02/ and in a hydro activation window run:

hydro merge botex_and_botex test_hy1.csv i5wi4test_hy1.csv DELC13 DELC13_FLAG_W

This will crash because there is a C13NUM in the second input file, i5wi4test_hy1.csv

So I took the column C13NUM out of the hy1.csv file, and ran

hydro merge botex_and_botex test_hy1.csv i5_no-c13num_hy1.csv DELC13 DELC13_FLAG_W

This did not fail, and a summary of the differences between the original input csv file (test_hy1.csv) and the resulting merged file are listed below. Please tell me which ones are as you intended, and which ones you will fix, and I will check only the ones you fixed on the next code update.

-sharon

#!csv

PARAM       OLD_VALUE    NEW_Value  len(OLD_VALUE) len(NEW_VALUE) - comment

ALKALI    ['   2318.9'] ['  2318.9'] 9 8     - field length changed from 9 to 8
CFC-11    ['    2.042'] ['   2.042'] 9 8     - field length changed from 9 to 8 
CFC-12    ['    1.117'] ['   1.117'] 9 8     - field length changed from 9 to 8
CTDPRS    ['    105.7'] ['   105.7'] 9 8     - field length changed from 9 to 8
DELC13    ['  -340.00'] ['   53.00'] 9 8     - field length changed from 9 to 8
DELC14    ['   -999.0'] ['  -999.0'] 9 8     - field length changed from 9 to 8
DELHE3    ['  -999.00'] [' -999.00'] 9 8     - field length changed from 9 to 8
NITRAT    ['     1.93'] ['    1.93'] 9 8     - field length changed from 9 to 8
NITRIT    ['     0.06'] ['    0.06'] 9 8     - field length changed from 9 to 8
OXYGEN    ['    190.3'] ['   190.3'] 9 8     - field length changed from 9 to 8
PHSPHT    ['     0.29'] ['    0.29'] 9 8     - field length changed from 9 to 8
SILCAT    ['     3.40'] ['    3.40'] 9 8     - field length changed from 9 to 8
TCARBN    ['   2022.7'] ['  2022.7'] 9 8     - field length changed from 9 to 8
TRITUM    [' -999.000'] ['-999.000'] 9 8     - field length changed from 9 to 8

CTDOXY    ['    190.7'] [' 190.7000'] 9 9    - field length stayed at 9
DELHER    ['-999.000'] [' -999.00'] 8 8      - field length stayed at 8
THETA     [' 21.8171'] ['  21.8171'] 8 9     - field length changed from 8 to 9

BTLNBR    ['     34'] ['         34'] 7 11   - field length changed from 7 to 11
CASTNO    ['  1'] ['          1'] 3 11       - field length changed from 3 to 11
DEPTH     [' 4555'] ['       4555'] 5 11     - field length changed from 5 to 11
LATITUDE  ['-19.9977'] ['   -19.9977'] 8 11  - field length changed from 8 to 11
LONGITUDE ['  54.5000'] ['       54.5'] 9 11 - field length changed from 9 to 11
SAMPNO    ['     34'] ['         34'] 7 11   - field length changed from 7 to 11
SECT_ID   [' TRNS1'] ['      TRNS1'] 6 11    - field length changed from 6 to 11
TIME      ['1941'] ['       1941'] 4 11      - field length changed from 4 to 11
STNNBR    ['   574'] ['     574'] 6 8        - field length changed from 6 to 8

CTDOXY    ['    190.7'] [' 190.7000'] 9 9    - places after the decimal went from 1 to 4
LONGITUDE ['  54.5000'] ['       54.5'] 9 11 - places after the decimal went from 5 to 1

[DocOtak]

Original comment by abarna (Bitbucket: abarna, GitHub: abarna).

---

Formatting

[DocOtak]

Original comment by myshen (Bitbucket: myshen, GitHub: myshen).

---

C13NUM error indicated extra derivative columns were being copied in. Thanks.

I think whitespace changes are not important. So field length changes resulting from whitespace are not of consequence.

Here is the table revised with whitespace changes removed.

PARAM       OLD_VALUE    NEW_Value  len(OLD_VALUE) len(NEW_VALUE) - comment

DELC13    ['  -340.00'] ['   53.00'] 9 8     - field length changed from 9 to 8
CTDOXY    ['    190.7'] [' 190.7000'] 9 9    - field length stayed at 9
DELHER    ['-999.000']  [' -999.00'] 8 8      - field length stayed at 8
LONGITUDE ['  54.5000'] ['       54.5'] 9 11 - field length changed from 9 to 11

[DocOtak]

Original comment by myshen (Bitbucket: myshen, GitHub: myshen).

---

resolved C13NUM error. still checking field length changes.

[DocOtak]

Original comment by see (Bitbucket: sescher, GitHub: sescher).

---

Matt,

As per your previous message, you can disregard the DELC13 field
length change below. the hydro code correctly changed the old value from -340.00 to 53.00 ( These are made
up values, but they were changed correctly). So only the CTDOXY,
DELHER, and LONGITUDE need to be checked.

[DocOtak]

Original comment by myshen (Bitbucket: myshen, GitHub: myshen).

---

Should correctly maintain precision from original data now.

The only issue is with fill values (-999). There are two cases.

Case 1: Data exists in column.

Fill values will have their precisions changed to be the same as the largest precision in the column. In most cases, this results in no change.

Case 2: No data in a column (all fill values).

The only course of action is to fall back to our parameter database's listed precision, so those fill values will all have their precision changed.

I don't see either of these being a big issue because the fill value's precision really actually doesn't matter.

ghdc hydro has been upgraded to hydro 0.8.0-31-g0e34600. Please continue testing.

[DocOtak]

Original comment by see (Bitbucket: sescher, GitHub: sescher).

---

Matt,

Great! I will keep the two cases in mind when I start merging
these. I'm sure I will run into both cases with this Indian Delc13
merging.