search

yakra / tmtools

Tools to aid in development of the TravelMapping project
0 stars 0 forks source link

GISplunge: RecordNum calculated improperly #8

Closed yakra closed 6 years ago

yakra commented 6 years ago

gisplunge02.cpp, in ProcRte(), line 570: int RecordNum = DBF.tellg()/RecordLength+1; does not account for DBF header length. Probably gets the correct INT most of the time anyway, due to RecordLength > HeaderLength.

~~Hex editor: create a file with RecordLength < 32. • See if I can make it rain cheese (Record number mismatch)~~

• Remind myself why I started numbering at 1, and not 0. Could it be this? Line 585: if (RecordNum != SHP.get()*0x1000000 + SHP.get()*0x10000 + SHP.get()*0x100 + SHP.get()) Remind myself what exactly's stored at this point in the SHP file.

yakra commented 6 years ago

What was that thing that happened when GISplunging PEI that one time? • A possible instance of this bug having detectable effects? • Was I using a trimmed DBF? • Why did SegDump work, rather than skipping file per Record number mismatch? • Can I recreate this and test?

24fc2346be8c476383ed593ac71d0837-260.wpt

yakra commented 6 years ago

Similarly:

ShapeRoot /home/yakra/gis/data/tx/txdot-roads_48113/txdot-2015-roadways_48113
KeyField RTE_UNIQ_I
KeyValue 508

Should SegDump record 1; SegDumps record 4 This segment has RTE_UNIQ_I = 1154.

Unmolested shapefile, as downloaded from, uhh... wherever. (Hell, in this case, why does GISplunge even work right some of the time?)

yakra commented 6 years ago

PE, original DBF file:

ShapeRoot /home/yakra/gis/data/pe/nrn_rrn_pe_12.0_shp_en/NRN_PE_12_0_ROADSEG
KeyField NID
KeyValue 24fc2346be8c476383ed593ac71d0837

Dumps record 260, as expected. Correct NID value.

260-2 http://www.openstreetmap.org/?lat=46.272151&lon=-63.118827
260-1 http://www.openstreetmap.org/?lat=46.274389&lon=-63.128079

PE, trimmed DBF (10.2 MB), 20171026 thru 20171105, and PE, trimmed DBF (9.8 MB), 20171106 thru 20171110:

ShapeRoot /home/yakra/gis/data/pe/nrn_rrn_pe_12.0x_shp_en/NRN_PE_12_0_ROADSEG
KeyField NID
KeyValue 24fc2346be8c476383ed593ac71d0837

Should SegDump record 260; SegDumps record 262 This segment has NID = a1bf9edc8c324b4d8ffdbc7669583dfd.

262-2 http://www.openstreetmap.org/?lat=46.708760&lon=-64.215424
262-1 http://www.openstreetmap.org/?lat=46.708702&lon=-64.215755
yakra commented 6 years ago

TX

ProcRte 566 DBF.seekg(HeaderLength + KeyOffset + 1); --> DBF.seekg(577 + 79 + 1); --> DBF.seekg(657); 569 if (KeyMatch(KeyValue, DBFname, DBF.tellg())) --> if (KeyMatch("508", DBFname, 657))

KeyMatch 515 for (int index = 0; index < strlen(KeyValue); index++) --> for (int index = 0; index < strlen("508"); index++) --> for (int index = 0; index < 3; index++) // "508" matches, followed by 0x20 523 return 1;

ProcRte 569 KeyMatch("508", DBFname, 657) == 1 570 int RecordNum = DBF.tellg()/RecordLength+1; --> int RecordNum = 657/199+1; --> int RecordNum = 3+1; --> int RecordNum = 4;

BOOM! Note: HeaderLength > RecordLength

yakra commented 6 years ago

PE, original

targeting record 1 this time

566 DBF.seekg(HeaderLength + KeyOffset + 1); --> DBF.seekg(1569 + 0 + 1); --> DBF.seekg(1570); 570 int RecordNum = DBF.tellg()/RecordLength+1; --> int RecordNum = 1570/2029+1; --> int RecordNum = 0+1; --> int RecordNum = 1;

Ayuh. Note: HeaderLength < RecordLength And THAT is how I failed to notice this bug for 7 years.

yakra commented 6 years ago

PE, trimmed

targeting record 1 this time

566 DBF.seekg(HeaderLength + KeyOffset + 1); --> DBF.seekg(1569 + 0 + 1); --> DBF.seekg(1570); 570 int RecordNum = DBF.tellg()/RecordLength+1; --> int RecordNum = 1570/593+1; --> int RecordNum = 2+1; --> int RecordNum = 3;

Boom. Note: HeaderLength > RecordLength

yakra commented 6 years ago

Revisiting...

TX 570 int RecordNum = (DBF.tellg()-HeaderLength)/RecordLength+1; --> int RecordNum = (657-577)/199+1; --> int RecordNum = 80/199+1; --> int RecordNum = 0+1; --> int RecordNum = 1;

PE, original 570 int RecordNum = (DBF.tellg()-HeaderLength)/RecordLength+1; --> int RecordNum = (1570-1569)/2029+1; --> int RecordNum = 1/2029+1; --> int RecordNum = 0+1; --> int RecordNum = 1;

PE, trimmed 570 int RecordNum = (DBF.tellg()-HeaderLength)/RecordLength+1; --> int RecordNum = (1570-1569)/593+1; --> int RecordNum = 1/593+1; --> int RecordNum = 0+1; --> int RecordNum = 1;