search

e-sensing / sits

Satellite image time series in R
https://e-sensing.github.io/sitsbook/
GNU General Public License v2.0
470 stars 76 forks source link

sits_train crash - sits 1.3 #966

Closed Yehrol closed 1 year ago

Yehrol commented 1 year ago

Describe the bug

Since I updated sits to 1.3, sits_train crash at the end of the 2nd epoch.

sits 1.3

Epoch 1/100
Train metrics: Loss: NaN - Acc: 0.1106                                        
Valid metrics: Loss: NaN - Acc: 0.1106
Epoch 2/100
Train metrics: Loss: NaN - Acc: 0.1106                                        
Valid metrics: Loss: NaN - Acc: 0.1106
Error in if (self$compare(qty, self$current_best)) { : 
  missing value where TRUE/FALSE needed

sits 1.2

Epoch 1/100
Train metrics: Loss: 1.2822 - Acc: 0.7608                                     
Valid metrics: Loss: 1.2653 - Acc: 0.7767
Epoch 2/100
Train metrics: Loss: 1.2722 - Acc: 0.7705                                     
Valid metrics: Loss: 1.2715 - Acc: 0.7716
Epoch 3/100
256/595 [==========>---------------] - ETA: 38s - Loss: 1.2713  - Acc: 0.7703

To Reproduce

The following code is the same for 1.3 or 1.2

I've attached the file samplesOFS_gva_train_test which is a sample of the file I use. samplesOFS_gva_train_test.csv

library(sits)

s2_cube <- sits_cube(
  source = "AWS",
  collection = "SENTINEL-S2-L2A-COGS",
  tiles = c("31TGM"),
  bands = c("B01", "B02", "B03", "B04", "B05", "B06", "B07", "B08", "B8A", "B09", "B11", "B12", "CLOUD"),
  start_date = as.Date("2018-01-01"),
  end_date = as.Date("2018-03-31")
)

s2_regular_cube <- sits_regularize(
  cube = s2_cube,
  output_dir = "./regular_cube/",
  res = 10,
  period = "P5D",
  progress = TRUE
)

samples_csv_file <- read.csv(file = "./samplesOFS_gva_train_test.csv")
points <- sits_get_data(
  cube = s2_regular_cube,
  samples = samples_csv_file,
  bands = c("B02", "B03", "B04", "B05", "B06", "B07", "B08", "B11"),
  output_dir = "./get_data/",
  progress = TRUE
)

tempcnn_model <- sits_train(
  samples = points,
  ml_method = sits_tempcnn(
    optimizer         = torchopt::optim_adamw,
    cnn_layers        = c(128, 128, 128),
    cnn_kernels       = c(7, 7, 7),
    cnn_dropout_rates = c(0.2, 0.2, 0.2),
    epochs            = 100,
    batch_size        = 64,
    validation_split  = 0.2,
    verbose           = TRUE
  )
)

Additional context

sessionInfo() result

1.3

R version 4.2.1 (2022-06-23) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Rocky Linux 8.7 (Green Obsidian) Matrix products: default BLAS/LAPACK: /opt/ebsofts/FlexiBLAS/3.2.0-foss-2022a/lib64/libflexiblas.so.3.2 locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] sits_1.3.0 loaded via a namespace (and not attached): [1] Rcpp_1.0.10 pillar_1.7.0 compiler_4.2.1 prettyunits_1.1.1 [5] progress_1.2.2 class_7.3-20 tools_4.2.1 torchopt_0.1.3 [9] zeallot_0.1.0 bit_4.0.5 digest_0.6.29 ncdf4_1.21 [13] lubridate_1.8.0 jsonlite_1.8.0 lifecycle_1.0.3 tibble_3.1.7 [17] pkgconfig_2.0.3 rlang_1.1.0 cli_3.6.0 DBI_1.1.3 [21] curl_4.3.2 yaml_2.3.5 parallel_4.2.1 coro_1.0.3 [25] warp_0.2.0 e1071_1.7-11 withr_2.5.0 dplyr_1.0.9 [29] httr_1.4.5 hms_1.1.1 fs_1.5.2 generics_0.1.2 [33] vctrs_0.6.0 rstac_0.9.2-2 bit64_4.0.5 grid_4.2.1 [37] classInt_0.4-9 tidyselect_1.1.2 glue_1.6.2 sf_1.0-12 [41] R6_2.5.1 processx_3.8.0 fansi_1.0.3 luz_0.4.0 [45] callr_3.7.3 tidyr_1.2.0 purrr_0.3.4 slider_0.3.0 [49] magrittr_2.0.3 ps_1.7.3 units_0.8-1 gdalcubes_0.6.3 [53] ellipsis_0.3.2 torch_0.10.0 KernSmooth_2.23-20 utf8_1.2.2 [57] proxy_0.4-27 crayon_1.5.1

1.2

R version 4.2.1 (2022-06-23) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Rocky Linux 8.7 (Green Obsidian) Matrix products: default BLAS/LAPACK: /opt/ebsofts/FlexiBLAS/3.2.0-foss-2022a/lib64/libflexiblas.so.3.2 locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] sits_1.2.0 loaded via a namespace (and not attached): [1] progress_1.2.2 gdalcubes_0.6.3 luz_0.4.0 tidyselect_1.1.2 [5] terra_1.7-29 purrr_0.3.4 sf_1.0-12 vctrs_0.6.0 [9] generics_0.1.2 yaml_2.3.5 utf8_1.2.2 rlang_1.1.0 [13] e1071_1.7-11 pillar_1.7.0 withr_2.5.0 glue_1.6.2 [17] DBI_1.1.3 bit64_4.0.5 slider_0.3.0 lifecycle_1.0.3 [21] codetools_0.2-18 callr_3.7.3 ps_1.7.3 parallel_4.2.1 [25] curl_4.3.2 class_7.3-20 fansi_1.0.3 Rcpp_1.0.10 [29] KernSmooth_2.23-20 torch_0.10.0 classInt_0.4-9 coro_1.0.3 [33] jsonlite_1.8.0 fs_1.5.2 bit_4.0.5 hms_1.1.1 [37] digest_0.6.29 processx_3.8.0 dplyr_1.0.9 ncdf4_1.21 [41] grid_4.2.1 cli_3.6.0 tools_4.2.1 magrittr_2.0.3 [45] rstac_0.9.2-2 proxy_0.4-27 tibble_3.1.7 crayon_1.5.1 [49] tidyr_1.2.0 pkgconfig_2.0.3 zeallot_0.1.0 ellipsis_0.3.2 [53] data.table_1.14.2 prettyunits_1.1.1 torchopt_0.1.3 warp_0.2.0 [57] lubridate_1.8.0 httr_1.4.5 R6_2.5.1 units_0.8-1 [61] compiler_4.2.1
OldLipe commented 1 year ago

Hi @LorisDeBiasi,

Thank you for providing a detailed description of the issue. I was able to reproduce the error.

It appears that the Sentinel-2 collection in the AWS V0 catalog has some missing images. For instance, only images from March were retrieved. However, when the same cube is created in the Microsoft Planetary Computer (MPC) catalog, all images for the period are retrieved.

# AWS cube
s2_cube_aws <- sits_cube(
    source = "AWS",
    collection = "SENTINEL-S2-L2A-COGS",
    tiles = c("31TGM"),
    bands = c("B01", "B02", "B03", "B04", "B05", "B06", "B07", "B08", "B8A", "B09", "B11", "B12", "CLOUD"),
    start_date = as.Date("2018-01-01"),
    end_date = as.Date("2018-03-31")
)
# Get cube timeline
sits_timeline(s2_cube_aws)
#> [1] "2018-03-21" "2018-03-26" "2018-03-29" "2018-03-31"
# MPC cube
s2_cube_mpc <- sits_cube(
    source = "MPC",
    collection = "SENTINEL-2-L2A",
    tiles = c("31TGM"),
    bands = c("B01", "B02", "B03", "B04", "B05", "B06", "B07", "B08", "B8A", "B09", "B11", "B12", "CLOUD"),
    start_date = as.Date("2018-01-01"),
    end_date = as.Date("2018-03-31")
)
# Get cube timeline
sits_timeline(s2_cube_mpc)
#> [1] "2018-01-03" "2018-01-05" "2018-01-08" "2018-01-10" "2018-01-13" "2018-01-15"
#> [7] "2018-01-18" "2018-01-20" "2018-01-23" "2018-01-25" "2018-01-28" "2018-01-30"
#> [13] "2018-02-02" "2018-02-04" "2018-02-07" "2018-02-09" "2018-02-12" "2018-02-14"
#> [19] "2018-02-17" "2018-02-19" "2018-02-22" "2018-02-24" "2018-02-27" "2018-03-01"
#> [25] "2018-03-04" "2018-03-06" "2018-03-14" "2018-03-16" "2018-03-19" "2018-03-21"
#> [31] "2018-03-24" "2018-03-26" "2018-03-29"

After regularizing the AWS cube:

s2_regular_cube_aws <- sits_regularize(
    cube = s2_cube_aws,
    output_dir = "./regular_cube/aws/",
    res = 10,
    period = "P5D",
    progress = TRUE
)
# Get cube timeline
sits_timeline(s2_regular_cube_aws)
#> [1] "2018-03-21" "2018-03-26" "2018-03-31"

MPC cube regularization:

s2_regular_cube_mpc <- sits_regularize(
    cube = s2_cube_mpc,
    output_dir = "./regular_cube/mpc/",
    res = 10,
    period = "P5D",
    progress = TRUE
)
# Get cube timeline
sits_timeline(s2_regular_cube_mpc)
#>  [1] "2018-01-03" "2018-01-08" "2018-01-13" "2018-01-18" "2018-01-23" "2018-01-28"
#>  [7] "2018-02-02" "2018-02-07" "2018-02-12" "2018-02-17" "2018-02-22" "2018-02-27"
#>  [13] "2018-03-04" "2018-03-09" "2018-03-14" "2018-03-19" "2018-03-24" "2018-03-29"

The images retrieved by the AWS cube have a high cloud coverage and a small temporal interval. Sits package interpolates the cloud points to avoid null values in the time series. However, due to the limited number of cloud-free observations (as shown in the table below), the interpolation will result in repeated values within the time series.

# Get cloud cover for each date
unique(s2_cube$file_info[[1]][, c("date", "cloud_cover")])

#>  # A tibble: 4 × 2
#>  date       cloud_cover
#>  <date>           <dbl>
#>  1 2018-03-21        47.7
#>  2 2018-03-26        81.9
#>  3 2018-03-29        98.3
#>  4 2018-03-31        89.7
points$time_series[[1]]
#>  # A tibble: 3 × 9
#>  Index         B02    B03   B04   B05   B06   B07   B08   B11
#>  <date>      <dbl>  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#>  1 2018-03-21 0.0529 0.0719 0.072 0.129 0.215 0.244 0.277 0.221
#>  2 2018-03-26 0.0529 0.0719 0.072 0.129 0.215 0.244 0.277 0.221
#>  3 2018-03-31 0.0529 0.0719 0.072 0.129 0.215 0.244 0.277 0.221

For some reason, Deep Learning models cannot process this sequence of repeated values and return the reported error.

I recommend using the Microsoft Planetary Computer (MPC) catalog to obtain more images for regularization. I tested this approach in my environment using the same sits version (1.3), and everything worked as expected. With the MPC cube regularization, the timeline is more extensive and contains additional temporal information. 

samples_csv_file <- read.csv(file = "./samplesOFS_gva_train_test.csv")

points_mpc <- sits_get_data(
    cube = s2_regular_cube_mpc,
    samples = samples_csv_file,
    bands = c("B02", "B03", "B04", "B05", "B06", "B07", "B08", "B11"),
    output_dir = "./samples/",
    progress = TRUE
)

sits_timeline(points_mpc)
#>  [1] "2018-01-03" "2018-01-08" "2018-01-13" "2018-01-18"
#>  [5] "2018-01-23" "2018-01-28" "2018-02-02" "2018-02-07"
#>  [9] "2018-02-12" "2018-02-17" "2018-02-22" "2018-02-27"
#>  [13] "2018-03-04" "2018-03-09" "2018-03-14" "2018-03-19"
#>  [17] "2018-03-24" "2018-03-29"

I hope this information is helpful in resolving the issue you reported.

Yehrol commented 1 year ago

Hi @OldLipe

Thank you for the quick reply.

I tried with MPC, but unfortunately I'm getting an error when calling sits_regularize.

From what I've read, MPC should by default provide a short-lived token but it looks like there's an issue with it.

> s2_cube <- sits_cube(
  source = "MPC",
  collection = "SENTINEL-2-L2A",
  tiles = c("31TGM"),
  bands = c("B01", "B02", "B03", "B04", "B05", "B06", "B07", "B08", "B8A", "B09", "B11", "B12", "CLOUD"),
  start_date = as.Date("2018-01-01"),
  end_date = as.Date("2018-03-31")
)
  |======================================================================| 100%
> s2_regular_cube <- sits_regularize(
  cube = s2_cube,
  output_dir = here("regular_cube"),
  res = 10,
  period = "P5D",
  progress = TRUE
)
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
  |======================================================================| 100%
Tiles 31TGM (B01, B02, B03, B04, B05, B06, B07, B08, B09, B11, B12, B8A) are missing or malformed and will be reprocessed.
Error: .gc_regularize: invalid mpc token. (!is.null(res_content) is not TRUE)
OldLipe commented 1 year ago

Hi @LorisDeBiasi,

Sorry for the delay in responding.

As you mentioned, MPC cubes have a limited duration token, so I think it is more manageable to download the images first (sits_cube_copy() function) and then regularize them later.