search

jblindsay / whitebox-tools

An advanced geospatial data analysis platform
https://www.whiteboxgeo.com/
MIT License
957 stars 160 forks source link

Allow piping between tools #68

Closed jfbourdon closed 3 years ago

jfbourdon commented 4 years ago

Being about to pipe the output of one tool to the input of another would save some I/O time on large files by avoiding the step to write intermediate data on disk.

A short example could be with the FillSingleCellPits and BreachSingleCellPits tools that you want to run successively. The actual way of doing it would be:

> whitebox_tools -r=FillSingleCellPits -i dem.tif -o dem_filled.tif
> whitebox_tools -r=BreachSingleCellPits -i dem_filled.tif -o dem_filled_breached.tif

But with piping it could be something like: > whitebox_tools -r=FillSingleCellPits -i dem.tif -stdout | -r=BreachSingleCellPits -stdin -o dem_filled_breached.tif

When I asked about Numpy Array support (#52), saving I/O time was one of my goal. I now use an imperfect workaround with a ramdisk as I explain in my last comment.

jblindsay commented 4 years ago

This would require a fundamental change in the way that WhiteboxTools has been engineered. It is a command-line program, called by higher-level scripting languages by process calls. This allows for the maximum flexibility in terms of the environments in which it can be applied. To do what you are asking would require the software to be designed as a pluggable library and would require a much more intimate connection with the various supported front-ends. Also, each of the tools do not take in raster objects as inputs. Instead they take string-based file names as input parameters. This feature would require a complete redesign of the software. Early on in the project's history, I grappled with the decision of whether WBT should be a command-line tool or a library, and ultimately landed on the former for the flexibility it provided.

jfbourdon commented 4 years ago

I thought it could be done even if it was command-line tool because LAStools is also one and it allows piping. I must say however that I know nothing about the ins and outs and the engineering behind a software like WBT.

jblindsay commented 4 years ago

I'm not really that familiar with LASTools but from what I've seen, each of it's tools (e.g. las2shp, las2las, etc.) are separate executable files that can each be called by command line. I would be very surprised if the output of one tool can be put directly into another tool without an I/O even between operations. I believe it operates in a similar way to WhiteboxTools.