Closed yakra closed 6 years ago
ROADS_ACF.yTrim4.dbf
unique_id
: trim ".000000000000000"; save 16 B
ah_blm
: trim "0000000"; save 7 B
ah_elm
: trim "0000000"; save 7 B
ah_length
: trim "0000000"; save 7 B
ah_seg_num
: trim ".000000000000000"; save 16 B
TOTAL: save 53 B / record
x 439026 records = 23,268,378 B
2013-03-05: Some fields are missing. Was this list compiled from a culled file? A more relevant comment will be posted below...
7/9 test files are identical to previous DBFtrim version. Exceptions:
txdot-2015-roadways_48113.dbf (Dallas)
Shape_STLe F 12 2 0. <- 0.0000000000
Why is the extraneous decimal point not trimmed?
(It can't be left-justified; 12 chars take up entire field width. DecCount?)
• Using DBFcull, I see the DIFF is limited to this field. Good...
• Per DBFmine, 0.0000000000
is the only value for all records
ROADS_ACF.yTrim.dbf
Binary files testfiles.old/ROADS_ACF.yTrim.dbf and testfiles.new/ROADS_ACF.yTrim.dbf differ
File sizes are identical. No indication in field info display of any extraneous 0s being trimmed. (Why?) In theory then, nothing should be different... Either:
• load a 610.7 MB file into hex editor,
• fix the bug affecting TX first and hope the problem goes away, or
• something else
txdot-2015-roadways_48113.dbf (Dallas)
Shape_STLe F 12 2 0. <- 0.0000000000
Why is the extraneous decimal point not trimmed? (It can't be left-justified; 12 chars take up entire field width. DecCount?) • Using DBFcull, I see the DIFF is limited to this field. Good... • Per DBFmine,0.0000000000
is the only value for all records
Yes, the problem is in DecCount. Set to 0 in original file.
Thus tDBF.fArr[fNum].DecCount = DecCount-MinEx0;
-> 0-10; wraps around & becomes 246.
Thus if (MinEx0 == DecCount) MinEx0++;
never happens.
A couple workarounds for cases like this should be pretty easy.
First, look thru other files with type F fields, and see if having a Decimal Count of 0 is a common thing, or unique to the TXDOT files.
FWIW, the Decimal Count in the next field(Shape_Leng
) is screwy too: Length of 12, Decimal Count of 11. 12 minus the leading zero & the decimal point itself leaves only 10 decimal places.
This may just be a TXDOT thing...
First, look thru other files with type F fields, and see if having a Decimal Count of 0 is a common thing, or unique to the TXDOT files.
~/gis/data/pe/nrn_rrn_pe_12.0_shp_en/NRN_PE_12_0_ROADSEG.dbf No type F fields.
~/gis/data/md/SHA_Routes/SHA_LINE_ROUTES_MD_2015
SHAPELENGT
, DecCount 0x0B / 11, scientific notation, makes sense
SHAPE_Leng
, DecCount 0x0B / 11, scientific notation, makes sense
All good!
~/gis/data/me/medotpubrdss/2016-04-08/medotpubrdss
Shape_len
, DecCount 0x0B / 11, scientific notation, makes sense
All good!
~/gis/data/me/e911rdss/e911rds
SHAPE_len
, DecCount 0x0B / 11, scientific notation, makes sense
All good!
~/gis/data/me/medotpubrdss/2017-08-24/medotpubrds
PRIM_BMP
, PRIM_EMP
, SEG_LEN__M
, Shape_len
All DecCount 0x0B / 11, scientific notation, make sense
All good!
~/gis/data/nh/roads_dot_2016/Roads_DOT
MP_START
, MP_END
, SECT_LENGT
, SHAPE_Leng
All DecCount 0x0B / 11, scientific notation, make sense
All good!
~/gis/data/ma/RoadInv2017/Road_Inventory.yOrig.dbf No type F fields.
~/gis/data/ar/ROADS_ACF/ROADS_ACF
unique_id
, DecCount 0x0F / 15, only 0s to right of decimal. Blank value exists.
src_code
, DecCount 0x0F / 15, only 0s to right of decimal. Blank value exists.
nssda_val
, DecCount 0x0F / 15, only 0s to right of decimal. Blank value exists.
ah_blm
, DecCount 0x0F / 15, as many as 8 non-0 pl. Blank value. Trimmed DecCount == 3. HOW?
ah_elm
, DecCount 0x0F / 15, as many as 8 non-0 pl. Blank value. Trimmed DecCount == 3. HOW?
ah_length
, DecCount 0x0F / 15, as many as 8 non-0 pl. Blank value. Trimmed DecCount == 3. HOW?
ah_seg_num
, DecCount 0x0F / 15, only 0s to right of decimal. Blank value exists.
Shape_STLe
, DecCount 0x0F / 15, all sig figs used & no potential for trimming. No blank value.
Why are no extraneous zeros trimmed? How does DecCount get trimmed from 15 to 3? It's those bloody blank-space values!
0.001000000000000
, for example.
MinEx0 gets set to 12.
DecCount gets set to 3 (15-12).MinEx0 = 0;
Thus pad will never again be less than MinEx0.
Thus DecCount never gets changed again.Solution:
What happens to a blank-space value?
15 for (pad = 0; (fVal[pad] <= ' ') && pad < len; pad++);
pad gets set to strlen
16 NewVal = new char[strlen(fVal+pad)+1];
--> NewVal = new char[strlen("\0")+1];
--> NewVal = new char[0+1];
17 strcpy(NewVal, fVal+pad);
--> strcpy(NewVal, "\0");
19 fVal = NewVal;
fVal == "\0"
strlen(fVal) == 0
(...Right?)
- Reset DecCount along with MinEx0 in the else statement.
- Create an exception to this for blank values [ if (strlen(fVal)) ? ] so I can still trim stuff. How would this get handled when saving trimmed records to disk?
ah_length
gets trimmed 1 byte too many
10.10300000
datum disappears completely
all other values have 1 digit left of decimal
unsigned int RecNum = ((unsigned int)DBFf.tellg()-dbf.HeaLen)/dbf.RecLen+1;
testfiles.1/ROADS_ACF.ah_length-only.dbf RecNum = (140551-65)/19+1; RecNum = 140486/19+1; RecNum = 7394+1;
testfiles.2/ROADS_ACF.ah_length-only.dbf RecNum = 7394+1; then multiply by RecLen RecNum = 81334/11+1; then add HeaLen RecNum = (81399-65)/11+1
Datum there is 0.10300000
, I.E. the '1' @ beginning got trimmed...
All data before that only goes to 3 decimal places.
10.103000000000000
stays the same as established from Record 1:
MinEx0 = 12
DecCount = 15-12 = 3
len = strlen(fVal)-MinEx0;
len = 18-12;
len = 6
Later, we reach 0.032541810000000
@ record 9237
MinEx0 = 7
DecCount = 15-7 = 8
if (strlen(fVal) > tDBF.fArr[fNum].len+MinEx0)
if (17 > 6+7)
if (17 > 13) (Yes.)
len = strlen(fVal)-MinEx0;
len = 17-7;
len = 10
Long story short, the solution is to track how many digits are to the left of the decimal.
TX: v1 & v3 no diff. Good. PE: v0 & v3 no diff. Good. MD: v0 & v3 no diff. Good. ME e911rds: v0 & v3 no diff. Good. NH: v0 & v3 no diff. Good. ME DOT16: v0 & v3 no diff. Good. ME DOT17: v0 & v3 no diff. Good. MA: v0 & v3 no diff. Good.
And finally, AR...
unique_id
: trim ".000000000000000"; save 16 B
src_code
: trim ".000000000000000"; save 16 B
nssda_val
: trim ".000000000000000"; save 16 B
ah_blm
: trim "0000000"; save 7 B
ah_elm
: trim "0000000"; save 7 B
ah_length
: trim "0000000"; save 7 B
ah_seg_num
: trim ".000000000000000"; save 16 B
Shape_STLe
cannot be trimmed
TOTAL: save 85 B / record
x 439026 records = 37317210 B
610687600 (v1) - 37317210 = 573370390
And the verdict is... 573370390. YES!
if (MinEx0 >= DecCount) tDBF.fArr[fNum].DecCount = 0;
...inter alia.
TX: v4 filesize 27610 B greater than v1 thru v3. Good! All other files identical to v3.
If a type F field never uses scientific notation, IE, never contains 'e' (or 'E'?) look into trimming trailing 0s as done for type N fields (or sometimes if it does contain 'e'?)