WRI-Cities / static-GTFS-manager

GUI interface for creating, editing, exporting of static GTFS data for a public transit authority
GNU General Public License v3.0
148 stars 46 forks source link

Chunked tables: create new chunk when all existing ones are too big #118

Closed answerquest closed 6 years ago

answerquest commented 6 years ago

When provisioning new data like new shapes or trips, typically a smallestChunk function finds the smallest .h5 file to put the new data in.

Set an absolute max file size, and if all the chunks have become bigger than this limit, then provision a new chunk.

answerquest commented 6 years ago

Added some steps in smallestChunk function:

    sizeList = [ os.path.getsize(os.path.join(dbFolder, filename))/(2**20) for filename in filenames ]
    # get sizes in MB
    if min(sizeList) < maxSizeMB:
        chunkFile = filenames[ sizeList.index(min(sizeList)) ]

    else:
        nextChunkNum = len(filenames) + 1
        chunkFile = '{}_{}.h5'.format(prefix, nextChunkNum)
        logmessage('smallestChunk: All chunks for {} too big, lets create a new one, {}'.format(prefix,chunkFile))
    return chunkFile
answerquest commented 6 years ago

already fixed with v3.1.0