Closed answerquest closed 6 years ago
Added some steps in smallestChunk
function:
sizeList = [ os.path.getsize(os.path.join(dbFolder, filename))/(2**20) for filename in filenames ]
# get sizes in MB
if min(sizeList) < maxSizeMB:
chunkFile = filenames[ sizeList.index(min(sizeList)) ]
else:
nextChunkNum = len(filenames) + 1
chunkFile = '{}_{}.h5'.format(prefix, nextChunkNum)
logmessage('smallestChunk: All chunks for {} too big, lets create a new one, {}'.format(prefix,chunkFile))
return chunkFile
already fixed with v3.1.0
When provisioning new data like new shapes or trips, typically a
smallestChunk
function finds the smallest .h5 file to put the new data in.Set an absolute max file size, and if all the chunks have become bigger than this limit, then provision a new chunk.