processing catalog does not write files

[wiesehahn]

Hey, When I try to process a LAScatalog it happens from time to time that the catalog seems to be processed (CPU and Memory are used and progress bar counts upwards for processed files), however no files are written to the specified output path. This seems to appear quite randomly, sometimes after restarting the R-session the exactly same script resulted in files written to disk.

renv::restore()
library(here)
library(lidR)
library(dplyr)

##___________________________________________________

## load functions into memory
source(here("code/functions/las_to_imean.R"))

##___________________________________________________

#### apply function

load(here::here("data/interim/lidr-catalog-new.RData"))

storage_path <- here("data/processed/imean/upper")
opt_output_files(ctg) <- paste0(storage_path, "/{XLEFT}_{YBOTTOM}_iupper")
 ctg@output_options$drivers$SpatRaster$param$gdal = c("COMPRESS=DEFLATE", "PREDICTOR=2")
 ctg@output_options$drivers$SpatRaster$param$datatype = "INT2U"

opt_chunk_buffer(ctg) <- 10
opt_chunk_size(ctg) <- 0
opt_merge(ctg) <- FALSE
opt_stop_early(ctg) <- FALSE

# dont run for tiles already processed
processed_images <- list.files(storage_path, pattern = ".tif")
processed_images <- substr(processed_images, 1, 14)

ctg@data <- ctg@data %>%
  mutate(processed = case_when(paste0(as.integer(Min.X), "_", as.integer(Min.Y)) %in% processed_images ~ FALSE,
                               TRUE ~ TRUE))

myMetric<-function(i, z){
  q75=quantile(z,probs=c(0.75))
  aboveq75= z>q75

  zq75 = i[aboveq75]
  imeanq75=mean(zq75, na.rm=TRUE)

  return(imeanq75)
  }

create_imean <- function(las, resolution = 1)
{
  if (is(las, "LAScatalog"))  {
    options <- list(automerge = FALSE, need_buffer = TRUE)
    opt_select(las) <- "xyzci"
    opt_filter(las) <- "-keep_class 2 -keep_class 13 -keep_class 20"
    return(catalog_apply(las, create_imean, resolution=resolution, .options = options))
  }
  else if (is(las, "LAScluster")) {
    las <- readLAS(las)
    if (is.empty(las)) return(NULL) 
    imean <- create_imean(las, resolution=resolution)
    imean <- terra::crop(imean, ext(chunk))
    return(imean)
  }
  else if (is(las, "LAS")) {
    # normalize
    nlas <- normalize_height(las, tin())
    # select above DBH
    noground <- filter_poi(nlas, Classification != 2, Z >1.3)

    imean <- pixel_metrics(noground, ~myMetric(Intensity, Z), res = resolution)

    return(imean)
  }
  else {
    stop("This type is not supported.")
  }
}

create_imean(ctg, 0.5)

At least this time it might be related to the files themselfe as I define which files should be processed and currently no files are written when I try to continue processing the catalog (I tried it several times). But when I skip this part of the script and instead start processing the entire catalog it begins with writing files to disk. However, I am pretty sure (although not 100%) that the previous times it was not related to certain files, as no files where written in the first attempt and the process succeeded for the same files in the second or third attempt.

So at this time I get Warning messages: 1: In min(x) : no non-missing arguments to min; returning Inf which somehow results from my function

myMetric<-function(i, z){

  # select upper 25% and 
  q75=quantile(z,probs=c(0.75))
  aboveq75= z>q75

  # calculate mean intensity
  zq75 = i[aboveq75]
  imeanq75=mean(zq75, na.rm=TRUE)

  ifelse(imeanq75<0, imeanq75, NA)
  return(imeanq75)
}

If I understand it right opt_stop_early(ctg) <- FALSE makes that the process is ongoing for files without errors. So if this warning would be the reason no files are written, the engine should write a file once it processes a file without error again, right? I did not wait that long until now, but since several hundred files were processed without a file being written I assumed this was not the reason.

Sorry, its hard to explain and make it reproducible but maybe there are ideas why no files are written?

[wiesehahn]

Ok I am sorry for the issue and not being patiently enough.

After running the script again and waiting a little bit longer this time, files are being written again after a while. Hence, the problem was that all previous tiles could not be written (several hundred tiles in the west of the entire area cover shoreline areas where my metric could not be calculated).

(Although I am still unsure if this was the case in previous attempts too.)

[Jean-Romain]

No idea and your explanation is hard to grasp. It is unclear when the script writes nothing and when it misses to write some random files. If I understand well (which is not the case) when you do not assign an attribute processed it works well but if you assign an attribute processed it starts missing files randomly. I am right ?

Your script looks perfect at first glance.

If I understand it right opt_stop_early(ctg) <- FALSE makes that the process is ongoing for files without errors. So if this warning would be the reason no files are written,

I don't think the warning is critical for lidR if it comes from myMetrics but is critical in your code because it means there is an unexpected NA somewhere. If it comes from lidR however it is a big issue and I need a reproducible example.

[wiesehahn]

I will close this issue as it is working currently and reopen it if I experience this issue again and it is not related to the files.

[Jean-Romain]

Not related at all but using opt_* inside a function is often a bad practice. Actually you do it the way you want because your are the user and the developer. But when the users are not the developer opt_*() functions are expected to be called by users on the user side and not by the developer. In summary, do not package your code as is for other users :wink: