tarsh / pyshp

Automatically exported from code.google.com/p/pyshp
MIT License
0 stars 0 forks source link

optimization for appending features #50

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
What steps will reproduce the problem?
1.keep appending features to the shape file

What is the expected output? What do you see instead?
The writing time should not increase that much. 

What version of the product are you using? On what operating system?
1.14. Windows

Please provide any additional information below.

A simple testing example is as following:

----------------------------------------------------------------
def append():
    start = datetime.now()
    w = shapefile.Editor(shapefile=r'E:\data\shape.shp')
    w.point(90.3, 30)
    w.record('First', 'Point', datetime.now(), '123213212.22')
    w.save(r'E:/data/shape')
    print(str(i) + ': ' + str((datetime.now() - start).total_seconds()))

for i in range(10000):
    append()
----------------------------------------------------------------

The writing speed keep declining as more features are written, is there any way 
to improve this? I noted in the code, when opening a Editor:

-----------------------------------------
 elif is_string(shapefile):
            base = os.path.splitext(shapefile)[0]
            if os.path.isfile("%s.shp" % base):
                r = Reader(base)
                Writer.__init__(self, r.shapeType)
                self._shapes = r.shapes()
                self.records = r.records()
                self.fields = r.fields
---------------------------------------------

I assume that the writer basically reads all the data into the cache, and then 
rewrite them together with the appended data when "save" is called. Am I right? 
This makes it a real problem after several thousand of records are stored. 

I expect a way that can use the "append" mode to write new features. Is this 
possible?

Original issue reported on code.google.com by LiangYU....@gmail.com on 6 Jun 2013 at 8:08

GoogleCodeExporter commented 9 years ago

Original comment by jlawh...@gmail.com on 6 Jun 2013 at 10:44