runOTB produces no information when quiet = F

[tertiarymatt]

Hi there. I'm running R version 4.0.3, with link2GI version 0.4-5, in RStudio 1.4.1056.

Using runOTB works mostly, but the only message it ever returns is if it can't make a raster from an output (either due to a failure or due to retRaster = F not being set when it should be). Otherwise it either runs and succeeds silently or runs and fails silently. This makes troubleshooting the configuration of the algorithms rather difficult, and it doesn't seem like the intended result.

Take for instance the following code:

# Define connection to OTB
# note, building connection fails if using anything other than searchLocation
otblink <- link2GI::linkOTB(searchLocation = "C:/OTB_7.2.0/")
algos <- parseOTBAlgorithms(gili = otblink) # get list of algorithms

# Define folders
pow <- "E:/GIS_Database/04_PRUSVI/Powerlines"

# get image statistics
stats_call <- parseOTBFunction(algo = "ComputeImagesStatistics", gili = otblink)
listviewer::jsonedit(stats_call$help)

# set params
stats_call$il <- file.path(pow, "Stacks/AOI_01_stack01_forRF.tif")
stats_call$ram <- 4096
stats_call$out <- file.path(pow, "Stacks/aoi_01_stack01_stats.xml")

runOTB(stats_call, gili=otblink, retRaster = F, quiet = F)

(Note also that the $out parameter is missing from the help on this algorithm, as is the background value parameter.)

This worked multiple times earlier in the day, but for some reason it appears not to like the new version of the file I am pointing it at (though this file can be processed via OTB fine in QGIS) and running it takes a second, and it returns nothing, so I am unable to troubleshoot what is causing the failure.

Cheers, s

[gisma]

@tertiarymatt thanks for reporting - your example put me on the right track to isolate this error. In fact the error is not in the missing grid but that the input file argument must be called not with -in but with -il which is either new or has escaped me until now. I will systematically check to what extent calls have changed or I have ignored them so far and fix them as soon as possible.

[tertiarymatt]

Thanks for the quick response! I'm happy to have pointed up that issue...

However my main question/concern is that runOTB() simply never produces any messages regardless of the quiet setting, even when successfully running other tools such as TrainImagesClassifier.

I had assumed that runOTB() would pipe the messages produced from the command line call to the R console or the like, but this doesn't appear to be the case.

[gisma]

Thanks for clarifying. The correct piping of the command messages is a bit tricky. I focused first the error and will asap fix the piping of the status and messages dialogues.

[gisma]

@tertiarymatt please check if the preliminary piping of the messages works out for you.

[tertiarymatt]

Good news: The messages from the OTB command line run now appear to be routed to R! Thanks for your responsiveness and hard work on this project. It's quite useful!

Bad news: The runOTB() appears to be sending some slightly off-kilter parameters to the command line.

When running SFSFeatureExtraction (which uses $input_in now, it seems) there seems to be something odd with the $ram parameter setting getting swapped into the $out slot on the actual command line call

# filter algorithms to get SFS Features
sfs_call <- parseOTBFunction(algo = "SFSTextureExtraction", gili = otblink)

# View help for function in viewer! Super nice. 
listviewer::jsonedit(sfs_call$help)

# set SFS params
sfs_call$input_in <- file.path(pow, "CanopyRasters/AOI_1_Data/AOI_FilteredCanopy_R.tif")
sfs_call$channel <-  1   # Raster band to use: 1
sfs_call$parameters.spethre <- 10  #   Spectral threshold: 10
sfs_call$parameters.spathre <- 10   #   Spatial threshold: 10
sfs_call$parameters.nbdir <- 20    #   Number of directions: 20
sfs_call$parameters.alpha <- "None"
sfs_call$parameters.maxcons <- 5     #   Ratio Max: 5
sfs_call$ram <- 4096            #    amount of RAM to use
sfs_call$progress <- 1
sfs_call$out <- file.path(pow, "FeaturesExtraction/test.tif")

SFS <- runOTB(sfs_call, gili = otblink, quiet = FALSE)`

[1] "2020-12-17 13:35:24 (INFO) SFSTextureExtraction: Default RAM limit for OTB is 256 MB"
[1] "2020-12-17 13:35:24 (INFO) SFSTextureExtraction: GDAL maximum cache size is 810 MB"
[1] "2020-12-17 13:35:24 (INFO) SFSTextureExtraction: OTB will use at most 12 threads"
[1] "2020-12-17 13:35:25 (WARNING) SFSTextureExtraction: Check filename: no extension detected, using TIF as default."
[1] "2020-12-17 13:35:25 (INFO): Estimated memory for full processing: 65636.4MB (avail.: 1 MB), optimal image partitioning: 65637 blocks"
[1] "2020-12-17 13:35:25 (INFO): File 4096.tif will be written in 74740 blocks of 10503x1 pixels"
[1] "Writing 4096.tif...: 100% [**************************************************] (5m 18s)"
Error in .local(.Object, ...) : 

Error in .rasterObjectFromFile(x, objecttype = "RasterBrick", ...) : 
  Cannot create a RasterLayer object from this file. (file does not exist)

Commenting out the $ram parameter causes the following to be returned:

[1] "2020-12-17 13:58:59 (INFO) SFSTextureExtraction: Default RAM limit for OTB is 256 MB"
[1] "2020-12-17 13:58:59 (INFO) SFSTextureExtraction: GDAL maximum cache size is 810 MB"
[1] "2020-12-17 13:58:59 (INFO) SFSTextureExtraction: OTB will use at most 12 threads"
[1] "2020-12-17 13:59:00 (WARNING) SFSTextureExtraction: Check filename: no extension detected, using TIF as default."
[1] "2020-12-17 13:59:00 (INFO): Estimated memory for full processing: 65636.4MB (avail.: 1 MB), optimal image partitioning: 65637 blocks"
[1] "2020-12-17 13:59:00 (INFO): File 256.tif will be written in 74740 blocks of 10503x1 pixels"
[1] "Writing 256.tif...: 100% [**************************************************] (5m 21s)"

The parameters for TrainImagesClassifier also seem a bit wonky (and a few are missing from, such as imstat and confmatout (for images statistics xml file and a results confusion matrix, respectively):

# Random Forest
otb_rf <- parseOTBFunction(algo="TrainImagesClassifier", gili=otblink)
listviewer::jsonedit(otb_rf$help)

otb_rf$io.il <- file.path(pow, "FeaturesExtraction/AOI_01_Null_Filtered_Canopy_SFS.tif")
otb_rf$io.imstat <- file.path(pow,"FeaturesExtraction/AOI_01_Null_Filtered_Canopy_SFS_stats.xml")
otb_rf$io.vd <- file.path(pow, "training_data_32161.gpkg")
otb_rf$classifier <- "rf"
otb_rf$classifier.rf.max <- 7
otb_rf$classifier.rf.nbtrees <- 100
otb_rf$sample.mt <- -1
otb_rf$sample.mv <-  -1
otb_rf$sample.bm <- 1
otb_rf$sample.vtr <- 0.70
otb_rf$sample.vfn <- "class_num"
otb_rf$ram <- 4096
otb_rf$progress <- 1
otb_rf$io.out <- file.path(pow,"Modeling/model_fromR_12_17_01.txt")
otb_rf$io.confmatout <- file.path(pow,"Modeling/model_fromR_12_17_01_CM.txt") 

runOTB(otb_rf, gili=otblink, retRaster = F, quiet = F)

[1] "ERROR: Parameter -out does not exist in the application."

[gisma]

Thank you again for testing and tracking down. I was kind of hoping that it could work with a simple workaround. But obviously the naming of the arguments is some kind of _evolving_and unfortunately the command line help output is not generated strictly systematic. I suppose Ihave to think about how it can be avoided to check this manually (and probably for each OTB release) and to improve the quality of this simple wrapper tool. It seems that deeper nested / more complex commands are using a different help output scheme. Additionally the call of procedures that return no images but XML files seems to be different. I will check it asap (fortunately it is xmas time...).

[tertiarymatt]

There is definitely inconsistency in how the parameters themselves seem to be labelled, unfortunately. I don't know how to build packages, but I will try to find some time over the weekend to work on the parseOTBFunction and runOTB functions to see if I can help find a solution.

[gisma]

I have added the return of the used command line call if quiet=FALSE Maybe that helps in tracking down whats going wrong. It is also useful for a manual call of the command either...

[gisma]

I think I was able to fix this weird behaviour of mixing up the params. I getting this output now:

otblink<-link2GI::linkOTB(searchLocation = "/usr")

  projRootDir<-tempdir()
  data('rgb', package = 'link2GI')  
  raster::plotRGB(rgb)
  r<-raster::writeRaster(rgb, 
                         filename=file.path(projRootDir,"test.tif"),
                         format="GTiff", 
                         overwrite=TRUE)

sfs_call <- parseOTBFunction(algo = "SFSTextureExtraction", gili = otblink)

# set SFS params
sfs_call$input_in <- file.path(projRootDir,"test.tif")
sfs_call$channel <-  1   # Raster band to use: 1
sfs_call$parameters.spethre <- 3  #   Spectral threshold: 10
sfs_call$parameters.spathre <- 10   #   Spatial threshold: 10
sfs_call$parameters.nbdir <- 20    #   Number of directions: 20
sfs_call$parameters.alpha <- "None"
sfs_call$parameters.maxcons <- 5     #   Ratio Max: 5
sfs_call$ram <- 8112            #    amount of RAM to use
sfs_call$progress <- 1
sfs_call$out <- file.path(projRootDir,"outtest.tif")

SFS <- runOTB(sfs_call, gili = otblink, quiet = FALSE)

1] "2020-12-18 21:06:32 (INFO) SFSTextureExtraction: Default RAM limit for OTB is 256 MB"
[1] "2020-12-18 21:06:32 (INFO) SFSTextureExtraction: GDAL maximum cache size is 3214 MB"
[1] "2020-12-18 21:06:32 (INFO) SFSTextureExtraction: OTB will use at most 8 threads"
[1] "2020-12-18 21:06:32 (INFO): Estimated memory for full processing: 5.82586MB (avail.: 8112 MB), optimal image partitioning: 1 blocks"
[1] "2020-12-18 21:06:32 (INFO): File /tmp/RtmpR0SXyb/outtest.tif will be written in 1 blocks of 263x254 pixels"
[1] "Writing /tmp/RtmpR0SXyb/outtest.tif...: 100% [**************************************************] (0s)"
/usr/bin/otbcli_SFSTextureExtraction  -in /tmp/RtmpR0SXyb/test.tif -channel 1 -parameters.spethre 3 -parameters.spathre 10 -parameters.nbdir 20 -parameters.alpha None -parameters.maxcons 5 -out /tmp/RtmpR0SXyb/outtest.tif -ram 8112 -progress 1

Actually I am not sure if there are more side effects.

[tertiarymatt]

Very nice! I'll do some more testing over the weekend with additional algorithms. Thanks for your quick work on this.

[tertiarymatt]

Sorry for the delay in getting back to you!

I was able to test and for non-machine learning functions got good outputs as you did above. Including the call at the end is also nice. However, for some of the Learning algorithms, the input and output files are grouped into "io" and take the form "io.il", "io.vd", "io.out" etc.

To address this, I restructured the runOTB() function as shown below:

runOTB <- function(otbCmdList=NULL,
                   gili=NULL,
                   retRaster=TRUE,
                   quiet = TRUE){

    if (is.null(gili)) {
        otb<-link2GI::linkOTB()
        path_OTB<- otb$pathOTB
    } else path_OTB<- gili$pathOTB

    otb_algorithm<-unlist(otbCmdList[1])  
    otbCmdList[1]<-NULL
    otbCmdList$help<-NULL

    if(Sys.info()["sysname"]== "Windows") otb_algorithm <- paste0(otb_algorithm,".bat")

    if (names(otbCmdList)[1] == "input_in")  {
        otbCmdList$input_in <- gsub(" ", "\\/ ", R.utils::getAbsolutePath(otbCmdList$input_in))  
        names(otbCmdList)[1]<-"in"
    }
    else if (names(otbCmdList)[1] == "input_il")  {
        otbCmdList$input_il <- gsub(" ", "\\/ ", R.utils::getAbsolutePath(otbCmdList$input_il))  
        names(otbCmdList)[1]<-"il"
    }
    else if (names(otbCmdList)[1] == "io.il")  {
        otbCmdList$io.il <- gsub(" ", "\\/ ", R.utils::getAbsolutePath(otbCmdList$io.il))
    }

    if(!is.null(otbCmdList$out)){
        otbCmdList$out <-gsub(" ", "\\/ ", R.utils::getAbsolutePath(otbCmdList$out))  
        outn = otbCmdList$out
    } else {
        otbCmdList$io.out <-gsub(" ", "\\/ ", R.utils::getAbsolutePath(otbCmdList$io.out))
        outn = otbCmdList$io.out
    }

    command<-paste(paste0(path_OTB,"otbcli_",otb_algorithm," "),
                   paste0("-",names(otbCmdList)," ",otbCmdList,collapse = " "))

    command = gsub("\\\\", "/", command)

    if (quiet){
        system(command,ignore.stdout = TRUE,ignore.stderr = TRUE,intern = FALSE)
        if (retRaster ){
            #outn=gsub("\\/", "", path.expand(otbCmdList$out))
            if (length(grep("xml", outn)) == 0) {
                rStack <- assign(tools::file_path_sans_ext(basename(outn)),raster::stack(outn))
                return(rStack)}
            else {

                #warning("NOTE: ", outn," is not a raster\n")
                return(readLines(outn)) 
            }
        }
    }
    else {
        ret=system(command,ignore.stdout = FALSE,ignore.stderr = FALSE,intern = TRUE)
        lapply(ret, print)
        message(command)
        if (retRaster){
            #outn=gsub("\\/", "", path.expand(otbCmdList$out))
            if (length(grep("xml", outn)) == 0) {
                rStack <- assign(tools::file_path_sans_ext(basename(outn)),raster::stack(outn))
                return(rStack)}
            else {
                #warning("NOTE: ", outn," is not a raster\n")
                return(data=readLines(outn))
            }

        }
    }
}

Note that the names for the added variables incorporate a period/dot in them rather than an underscore, but this works for OTB algorithms containing an io grouping for parameters. The logic on the test of otbCmdList$out could probably be flipped, but I liked having the rare case (out being null) on the else.

I've looked a bit through the OTB documentation, and I think this covers all the naming conventions for the inputs and outputs that are currently in use. Adding the "io" prefix seems a bit superfluous, but so it goes.

Thanks again for your hard work on this project, and hopefully these additions are useful.

[gisma]

Very nice, big thanks for finding out the rest of the specifics. If you prefer you can make a pull request. If not I will integrate the improvements quickly.

[tertiarymatt]

Given my inexperience with pull requests, it's probably easier to integrate them!

I may contribute more in the future, and can address that then.