Closed jamcinnes2 closed 9 months ago
I found that the h5py chunking I setup is the problem. The chunking I used was great for speeding up the creation and updating of the wxdb file. But when reading the file it results in a seek and read I/O operation for EACH year of data... 80.
Im adding a final data processing step that will h5repack the file so that it is optimized for reading.
Awesome that you found that. And makes sense. Then I will wait for that before I redeploy -- I haven't put the new stuff up yet but will when I see this ticket get closed.
Ok so after creation of the hwitw.wxdb file and all initial data processing is complete, we can repack the file for better read performance. This only needs to be done once so I wont make it part of the weekly update process. We should probably document this as part of the app installation process.
h5repack --verbose --layout wxdb:CHUNK=10x10x80x52x29 hwitw.wxdb hwitw_repack.wxdb
Then copy hwitw_repack.wxdb over hwitw.wxdb (or symlink). In my testing accessing location data went from taking 20+ seconds to < 1 second.
We will see how it performs on the cluster.
Reading the wxdb dataset is very slow for what should be a seek and a contiguous read of a few kB.