search

google-code-export / pysal

Automatically exported from code.google.com/p/pysal
Other
1 stars 1 forks source link

Overhaul shapefile reader #89

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
See issue number 47
In line as much as possible.
Read directly into arrays instead of unpacking to lists.

Original issue reported on code.google.com by schmi...@gmail.com on 5 May 2010 at 7:18

GoogleCodeExporter commented 9 years ago

Original comment by schmi...@gmail.com on 28 Jan 2011 at 1:55

GoogleCodeExporter commented 9 years ago

Original comment by schmi...@gmail.com on 19 May 2011 at 5:01

GoogleCodeExporter commented 9 years ago

Original comment by sjsrey on 6 Aug 2011 at 12:15

GoogleCodeExporter commented 9 years ago 
shapefile profiling/refactoring...

read all polygons from us census tracts shapefile.
Baseline: ~9.559 seconds

refactored to use arrays for homogeneous data (point arrays):
~9.069 seconds, ~5% improvement over baseline

refactored to reduce number of function calls by grouping heterogeneous data 
into single struct.unpack calls:
~8.063 seconds, ~11% over arrays, ~15.7% improvement over baseline

committed in r1021

~37% of cumulative time is spent on is_clockwise function, which accounts for 
only 5.5% of function calls...
ncalls      tottime  percall  cumtime  percall filename:lineno(function)
128472    2.980    0.000    3.004    0.000 standalone.py:604(is_clockwise)

is_clockwise is currently invoked twice per ring, once in the shapefile reader 
and once in the Polygon constructor.  removing this duplication will provide 
another ~%19 speedup over the refactored reader for a total improvement of ~%31 
over baseline.

The only other obvious route to a more efficient reader is to drastically 
reduce the number of function calls. At over 2.3 million functions for ~60k 
polygons, we're using ~38 function calls per polygon. function call overhead 
could account for a large percentage of running time.

Original comment by schmi...@gmail.com on 20 Oct 2011 at 8:33

GoogleCodeExporter commented 9 years ago

Original comment by schmi...@gmail.com on 9 Jan 2012 at 9:10

GoogleCodeExporter commented 9 years ago

Original comment by schmi...@gmail.com on 13 Jul 2012 at 6:09