waypoints, segments, clinched

[yakra]

Ideas for speeding up the highest-impact DB tables

C++:

[157.8] ...waypoints    4.0
[161.8] ...segments 6.8
[168.6] ...clinched 1.7
[170.3] ...

Python:

[444.9] ...waypoints    6.8
[451.7] ...segments 3.4
[465.1] ...clinched 5.7
[470.8] ...

• C++
  • [x] waypoints: The the same approach as for the routes & connectedRoutes tables, and nix std::string Waypoint::csv_line in favor of inserting the info directly into the ofstream.
  • segments has 2 components to improve:
  • [ ] Similarly replacing std::string HighwaySegment::csv_line
  • [ ] clinched_list setup: just construct std::pair<TravelerList*,unsigned int>s
  • [ ] clinched has nothing much to do other than adapt to the new clinched_list set up while processing segments. Will probably run a wee bit slower than just inserting already-constructed strings as we do now, but that should be outweighed by the improvement to segments.
• Python:
  • Odd that in C++, segments took 1.7x as long as waypoints but here it takes half as long. Whatever, I'll blame the mysteries that go on behind the curtain, like string interning or whatever.
  • waypoints
  • [ ] Including the parens & newline in Waypoint.csv_line may help, avoiding constructing one string and then another string based off of it.
  • [ ] Would eliminating Waypoint.csv_line altogether help? AIUI, a string object is still constructed for the file object to write. As a temporary value, is it interned?
  • segments
  • [ ] Follow the lead of waypoints with parens, newlines, and the csv_line function.
  • Not much to do with clinched_list, as that has to be stringified before writing anyway. Except:
  • [ ] clinched can lose the parens & newlines, with those constructed directly in clinched_list earlier. Mirror image of the C++ diffs -- slower in segments, made up for by being faster here.

[yakra]

related: https://github.com/TravelMapping/DataProcessing/issues/324

[yakra]

• [x] waypoints DB table needs Python-style floats for latitude & longitude.
• [ ] more broadly, search & destroy csv_line functions project-wide (Only HighwaySegment::csv_line left)
• [x] Waypoint::out_of_bounds needs pyfloats
• [x] DUPLICATE_COORDS too