Support for Harmonized Sentinel Landsat data

[daniel-althoff]

I am sorry if this is something you are already working on. I looked up in issues and could not find anything related.

The Harmonized Sentinel Landsat data can be very useful, especially in regions where cloud cover is high. I thought it could be a great addition to the list of collections made available by sits.

Here is an example made by USGS to access the CMR-STAC and download HSL.S and HSL.L https://lpdaac.usgs.gov/resources/e-learning/getting-started-with-cloud-native-harmonized-landsat-sentinel-2-hls-data-in-r/

Do you guys think this could be added to the list of collections in a later version?

Thanks

[gilbertocamara]

Dear @daniel-althoff many thanks for your suggestion. We will include access to the HSL collection in the next version of sits. We plan to include this access first in the "dev" version and will inform you as soon as possible.

[rolfsimoes]

Available in sits dev version. To test it, please run this command:

devtools::install_github("e-sensing/sits@dev")

[rolfsimoes]

Hi @daniel-althoff

I've prepared an example to you on how to use the HLS collection. We will include this example soon in the sits documentation. To run the example, you must create a ~/.netrc plain text file in Unix (or %HOME%\_netrc in Windows) containing the following fields:

machine urs.earthdata.nasa.gov
login <YOUR-LOGIN>
password <YOUR-PASSWORD>

You must create an account here. Once you have create ~/.netrc file, you can run the example using sits 1.4 available in github (development version). You can install it by typing devtools::install_github("e-sensing/sits@dev").

Example:

bbox <- c(lon_min = -63.153456, 
          lat_min = -7.681265, 
          lon_max = -62.862239, 
          lat_max = -7.385608)

hls_s2 <- sits_cube(
    source = "HLS",
    collection = "HLSS30",
    roi = bbox,
    start_date = "2020-08-01",
    end_date = "2021-07-31"
)

sits_timeline(hls_s2)
sits_bands(hls_s2)

hls_l8 <- sits_cube(
    source = "HLS",
    collection = "HLSL30",
    roi = bbox,
    start_date = "2020-08-01",
    end_date = "2021-07-31"
)

sits_timeline(hls_l8)
sits_bands(hls_l8)

hls_merged <- sits_merge(hls_s2, hls_l8)

sits_bands(hls_merged)
sits_timeline(hls_merged)

[daniel-althoff]

Hi @rolfsimoes Thank you so much for this. I'll try the example sometime soon and hopefully get a good outcome :)

Do you guys have any personal opinion or preferences when comparing HLS to S2 or other collections? Any drawback? Thanks again!

[gilbertocamara]

Dear @daniel-althoff We have not yet run major applications using HLS data, but we'll keep you posted.

[daniel-althoff]

Hi @daniel-althoff

I've prepared an example to you on how to use the HLS collection. We will include this example soon in the sits documentation. To run the example, you must create a ~/.netrc plain text file in Unix (or %HOME%\_netrc in Windows) containing the following fields:

machine urs.earthdata.nasa.gov
login <YOUR-LOGIN>
password <YOUR-PASSWORD>

You must create an account here. Once you have create ~/.netrc file, you can run the example using sits 1.4 available in github (development version). You can install it by typing devtools::install_github("e-sensing/sits@dev").

so the .netrc, in my case, should be created here 'C:\Users\danie.netrc', right?

However, I have not been able to install sits@dev I get the following:

The downloaded source packages are in ‘C:\Users\danie\AppData\Local\Temp\Rtmp8OT3p1\downloaded_packages’ ✔ checking for file 'C:\Users\danie\AppData\Local\Temp\Rtmp8OT3p1\remotes24f85637760d\e-sensing-sits-6a988b9/DESCRIPTION' ... ─ preparing 'sits': (1.9s) ✔ checking DESCRIPTION meta-information ... ─ cleaning src ─ checking for LF line-endings in source and make files and shell scripts ─ checking for empty or unneeded directories ─ building 'sits_1.4.0.tar.gz'

Installing package into ‘C:/Users/danie/AppData/Local/R/win-library/4.2’ (as ‘lib’ is unspecified)

• installing source package 'sits' ... using staged installation libs g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c RcppExports.cpp -o RcppExports.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c combine_data.cpp -o combine_data.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c kernel.cpp -o kernel.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c label_class.cpp -o label_class.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c linear_interp.cpp -o linear_interp.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c nnls_solver.cpp -o nnls_solver.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c normalize_data.cpp -o normalize_data.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c normalize_data_0.cpp -o normalize_data_0.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c sampling_window.cpp -o sampling_window.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c smooth.cpp -o smooth.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c smooth_bayes.cpp -o smooth_bayes.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c smooth_sgp.cpp -o smooth_sgp.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c smooth_whit.cpp -o smooth_whit.o g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-42~1.1/include" -DNDEBUG -I'C:/Users/danie/AppData/Local/R/win-library/4.2/Rcpp/include' -I'C:/Users/danie/AppData/Local/R/win-library/4.2/RcppArmadillo/include' -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c uncertainty.cpp -o uncertainty.o g++ -std=gnu++11 -shared -s -static-libgcc -o sits.dll tmp.def RcppExports.o combine_data.o kernel.o label_class.o linear_interp.o nnls_solver.o normalize_data.o normalize_data_0.o sampling_window.o smooth.o smooth_bayes.o smooth_sgp.o smooth_whit.o uncertainty.o -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib -LC:/PROGRA~1/R/R-42~1.1/bin/x64 -lR C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnlssolver.cp:(.text.hot+0x982): undefined reference to `dgemm' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnlssolver.cp:(.text.hot+0xa6d): undefined reference to `dgemv' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnlssolver.cp:(.text.hot+0xb5e): undefined reference to `dgemv' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnlssolver.cp:(.text.hot+0xfaa): undefined reference to `dgemm' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnlssolver.cp:(.text.hot+0x108c): undefined reference to `dgemv' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnlssolver.cp:(.text.hot+0x1549): undefined reference to `dsyrk' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0_EEi[_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0EEi]+0x27b): undefined reference to `dgemm' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0_EEi[_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0EEi]+0x466): undefined reference to `dgemm' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0_EEi[_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0EEi]+0x5a3): undefined reference to `dgemv' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0_EEi[_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0EEi]+0x70d): undefined reference to `dgemv' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0_EEi[_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0EEi]+0x80a): undefined reference to `dgemv' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0_EEi[_ZN4arma10glue_times18apply_inplace_plusINS_11subview_colIdEENS_3RowIdEEEEvRNS_3MatINT_9elem_typeEEERKNS_4GlueIS7_T0_S0EEi]+0x8f8): undefined reference to `dgemv' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times5applyIdLb0ELb1ELb1ENS_3MatIdEES3_EEvRNS2_IT_EERKT3_RKT4S4[_ZN4arma10glue_times5applyIdLb0ELb1ELb1ENS_3MatIdEES3_EEvRNS2_IT_EERKT3_RKT4S4]+0x1c9): undefined reference to dgemm_' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times5applyIdLb0ELb1ELb1ENS_3MatIdEES3_EEvRNS2_IT_EERKT3_RKT4_S4_[_ZN4arma10glue_times5applyIdLb0ELb1ELb1ENS_3MatIdEES3_EEvRNS2_IT_EERKT3_RKT4_S4_]+0x2ad): undefined reference todgemv_' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: nnls_solver.o:nnls_solver.cp:(.text$_ZN4arma10glue_times5applyIdLb0ELb1ELb1ENS_3MatIdEES3_EEvRNS2_IT_EERKT3_RKT4S4[_ZN4arma10glue_times5applyIdLb0ELb1ELb1ENS_3MatIdEES3_EEvRNS2_IT_EERKT3_RKT4S4]+0x772): undefined reference to dsyrk_' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text+0x1771): undefined reference todtrtri_' C:\rtools42\x8664-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text.hot+0x1d4): undefined reference to `dgemm' C:\rtools42\x8664-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text.hot+0x2b7): undefined reference to `dgemv' C:\rtools42\x8664-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text.hot+0x64e): undefined reference to `dsyrk' C:\rtools42\x8664-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text.hot+0x99f): undefined reference to `dgemm' C:\rtools42\x8664-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text.hot+0xa7f): undefined reference to `dgemv' C:\rtools42\x8664-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text.hot+0xb76): undefined reference to `dgemv' C:\rtools42\x8664-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text.hot+0xda3): undefined reference to `dgemv' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text$_ZN4arma6auxlib9inv_sympdIdEEbRNS_3MatIT_EERb[_ZN4arma6auxlib9inv_sympdIdEEbRNS_3MatITEERb]+0x94): undefined reference to `dpotrf' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text$_ZN4arma6auxlib9inv_sympdIdEEbRNS_3MatIT_EERb[_ZN4arma6auxlib9inv_sympdIdEEbRNS_3MatITEERb]+0xbf): undefined reference to `dpotri' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text$_ZN4arma6auxlib3invIdEEbRNS_3MatIT_EE[_ZN4arma6auxlib3invIdEEbRNS_3MatITEE]+0xad): undefined reference to `dgetrf' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text$_ZN4arma6auxlib3invIdEEbRNS_3MatIT_EE[_ZN4arma6auxlib3invIdEEbRNS_3MatITEE]+0x10c): undefined reference to `dgetri' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: smooth.o:smooth.cpp:(.text$_ZN4arma6auxlib3invIdEEbRNS_3MatIT_EE[_ZN4arma6auxlib3invIdEEbRNS_3MatITEE]+0x1cc): undefined reference to `dgetri' C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: smooth_sgp.o:smoothsgp.cpp:(.text.hot+0x10d): undefined reference to `dgemv' collect2.exe: error: ld returned 1 exit status no DLL was created ERROR: compilation failed for package 'sits'
• removing 'C:/Users/danie/AppData/Local/R/win-library/4.2/sits'
• restoring previous 'C:/Users/danie/AppData/Local/R/win-library/4.2/sits'

Is there something I am missing?

version _
platform x86_64-w64-mingw32
arch x86_64
os mingw32
crt ucrt
system x86_64, mingw32
status
major 4
minor 2.1
year 2022
month 06
day 23
svn rev 82513
language R
version.string R version 4.2.1 (2022-06-23 ucrt) nickname Funny-Looking Kid

[gilbertocamara]

Dear @daniel-althoff, this seems to be a problem with the Windows version of sits 1.4. We are investigating the problem, and will come back to you ASAP.

[OldLipe]

Hi @daniel-althoff,

I just updated the dev branch in my repo. Could you please test it?

devtools::install_github("oldlipe/sits@dev2")

[daniel-althoff]

Hi @OldLipe. It worked, thanks a lot!