Bug in process / Landsat2L2 chunk

[PabRod]

Problem

The last chunk in the vignette:

https://github.com/RETURN-project/makeDataCube/blob/7b85e17a50f910b160ec87f78b3a9ae2a1c8a678/vignettes/make_Landsat_cube.Rmd#L179-L181

fails with:

Error in list_files(file, ignore_missing, TRUE, verbosity) : 
  File '' does not exist.

Details

The error can be traced to line 113 in checkLSlog:

https://github.com/RETURN-project/makeDataCube/blob/7b85e17a50f910b160ec87f78b3a9ae2a1c8a678/R/General.R#L112-L130

When trying:

file.exists(file.path(logfolder, scenes))

a vector of all FALSE is returned. So it seems that the files:

file.path(logfolder, scenes) 
# [local_path]/makeDataCube/data/log/LE07_L1TP_220063_20001115_20170209_01_T1.tar.gz.log
# ...

are required, but have not been created on a previous stage.

@wandadk Do you know which chunk is supposed to create these files? We can also talk about this in our meeting.

[wandadk]

Hi Pablo,

These log files are generated by FORCE when processing the level 1 files to level 2. So using this line of code: https://github.com/RETURN-project/makeDataCube/blob/7b85e17a50f910b160ec87f78b3a9ae2a1c8a678/R/Landsat.R#L60.

Did FORCE generate any level 2 files? It looks like FORCE failed to run at all.

[wandadk]

Would you like to take a closer look at this together now? I'm available until 3PM.

[PabRod]

I'll try to take a look before. I'll let you know. Thanks!

[PabRod]

Hi Pablo,

These log files are generated by FORCE when processing the level 1 files to level 2. So using this line of code: https://github.com/RETURN-project/makeDataCube/blob/7b85e17a50f910b160ec87f78b3a9ae2a1c8a678/R/Landsat.R#L60

. Did FORCE generate any level 2 files? It looks like FORCE failed to run at all.

This command returns: No images queued in data/level1/queue.txt

[PabRod]

tar.gz compression is failing. The line:

https://github.com/RETURN-project/makeDataCube/blob/7b85e17a50f910b160ec87f78b3a9ae2a1c8a678/R/Landsat.R#L31

translates to a wrong system command:

tar -zcvf [path]/makeDataCube/data/temp/c(\"LE07_L1TP_220063_20010526_20170205_01_T1\", \"LT05_L1TP_220063_20010331_20161211_01_T1\", \"LE07_L1TP_220063_20010424_20170205_01_T1\", \"LT05_L1TP_220063_20010518_20161211_01_T1\", \"LT05_L1TP_220063_20001123_20161213_01_T1\", \"LE07_L1TP_220063_20010307_20170206_01_T1\", \"LT05_L1TP_220063_20001107_20161213_01_T1\", \"LT05_L1TP_220063_20010315_20161212_01_T1\", \"LE07_L1TP_220063_20001115_20170209_01_T1\").tar.gz -C[path]/makeDataCube/data/temp/c(\"LE07_L1TP_220063_20010526_20170205_01_T1\", \"LT05_L1TP_220063_20010331_20161211_01_T1\", \"LE07_L1TP_220063_20010424_20170205_01_T1\", \"LT05_L1TP_220063_20010518_20161211_01_T1\", \"LT05_L1TP_220063_20001123_20161213_01_T1\", \"LE07_L1TP_220063_20010307_20170206_01_T1\", \"LT05_L1TP_220063_20001107_20161213_01_T1\", \"LT05_L1TP_220063_20010315_20161212_01_T1\", \"LE07_L1TP_220063_20001115_20170209_01_T1\") .

Possible fix

The line:

https://github.com/RETURN-project/makeDataCube/blob/7b85e17a50f910b160ec87f78b3a9ae2a1c8a678/R/Landsat.R#L27

Is assigning dllList AND scenesid to the variable scenes. We should use only one.

dllInfo <- dllLandsat(queuefolder, queuefile, tmpfolder, logfile, ext, starttime, endtime, sensors, tiers, cld)
scenes <- dllInfo[[2]] # Or is it 1?

[wandadk]

Hi Pablo,

Yes, you're right. I believe that we need dllInfo[[1]]. That should represent the images that match the query and are not already available (so are actually downloaded). dllInfo[[2]] should equal all images that match the query (including the ones that were already available and are thus not downloaded again).