jblindsay
commented
3 months ago The problem of I/O is with the design of WhiteboxTools unfortunately. Each tool is effectively an independent program and it needs to read its inputs and write its outputs. When you have a long workflow involving many intermediate steps, this compounds the I/O requirements substantially. However, the I/O issue is largely resolved with the use of Whitebox Workflows for Python (WbW) rather than WhiteboxTools (WbT), since there is no need for I/O in the intermediate steps. Each WbW tool takes an input in-memory geospatial objects and outputs in-memory objects. With WbW, you only need to read the input and write the output but not any of the steps in between.
InsolublePancake
I have been experimenting with using cloud computing to run whitebox tools on DEMs that are too large to process locally. Memory is the biggest issue of course, but I've noticed that I/O is also a bottleneck. I was wondering whether there was anything you could do to improve this within the tools? For example, I know that whitebox tools use parallel processing in places but does it take full advantage of this at the I/O stage? What about adjusting the read-ahead value when reading in large datasets? I'm afraid that my knowledge of low-level programming is limited so I won't presume to advise you, but any improvements you could make here would be appreciated.
Similarly, is there anything that I could do at my end to improve I/O? I notice that the raster output of your tools is arranged 1 row per block. I infer from this that this is more efficient for the tools. Presuming that is true, would there be any advantage to expanding blocksize further - such as two rows per block?
Any help or advice very welcome. Thank you