search

ropensci / software-review

rOpenSci Software Peer Review.
286 stars 104 forks source link

Implementation of OPTRAM algorithm to derive soil moisture from remote sensing imagery #612

Closed micha-silver closed 3 weeks ago

micha-silver commented 8 months ago

Date accepted: 2024-06-05

Submitting Author Name: Micha Silver Submitting Author Github Handle: !--author1-->@micha-silver<!--end-author1-- Other Package Authors Github handles: (comma separated, delete if none) @github_handle1, @github_handle2 Repository: https://gitlab.com/rsl-bidr/roptram Version submitted: 0.0.1.000 Submission type: Standard Editor: !--editor-->@adamhsparks<!--end-editor-- Reviewers: @harryeslick, @obrl-soil

Archive: TBD Version accepted: TBD Language: en

Package: rOPTRAM
Title: Derive soil moisture using the OPTRAM algorithm
Version: 0.0.1.000
Authors@R: c(
  person("Micha", "Silver", , "silverm@post.bgu.ac.il", role = c("aut", "cre"),
           comment = c(ORCID = "0000-0002-1128-1325")),
  person("Arnon", "Karnieli", , "karnieli@bgu.ac.il", role = c("ctb", "fnd"),
          comment = c(ORCID = "0000-0001-8065-9793"))
  )
Description: The OPtical TRapezoid Model (OPTRAM) derives soil moisture
  based on the linear relation between a vegetation index
  and Land Surface Temperature (LST). The Short Wave Infra-red (SWIR) band
  is used as a proxy for LST. See:
  Sadeghi, M., Babaeian, E., Tuller, M., Jones, S.B., 2017.
  The optical trapezoid model:
  A novel approach to remote sensing of soil moisture 
  applied to Sentinel-2 and Landsat-8 observations.
  Remote Sensing of Environment 198, 52–68,
  https://doi.org/10.1016/j.rse.2017.05.041 .
License: GPL (>= 3) + file LICENSE
URL: https://gitlab.com/rsl-bidr/roptram.git
BugReports: https://gitlab.com/rsl-bidr/
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.2.3
Depends:
  R (>= 4.1.0)
Imports:
    dplyr,
    ggplot2,
    sf,
    terra,
    tools,
    utils
Suggests:
    geojsonio,
    geojsonlint,
    stats,
    sen2r (> 1.5.0),
    testthat (>= 3.0.0),
    xml2
Config/testthat/edition: 3

Scope

Technical checks

Confirm each of the following by checking the box.

This package:

Publication options

MEE Options - [ ] The package is novel and will be of interest to the broad readership of the journal. - [ ] The manuscript describing the package is no longer than 3000 words. - [ ] You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see [MEE's Policy on Publishing Code](http://besjournals.onlinelibrary.wiley.com/hub/journal/10.1111/(ISSN)2041-210X/journal-resources/policy-on-publishing-code.html)) - (*Scope: Do consider MEE's [Aims and Scope](http://besjournals.onlinelibrary.wiley.com/hub/journal/10.1111/(ISSN)2041-210X/aims-and-scope/read-full-aims-and-scope.html) for your manuscript. We make no guarantee that your manuscript will be within MEE scope.*) - (*Although not required, we strongly recommend having a full manuscript prepared when you submit here.*) - (*Please do not submit your package separately to Methods in Ecology and Evolution*)

Code of conduct

ropensci-review-bot commented 8 months ago

Thanks for submitting to rOpenSci, our editors and @ropensci-review-bot will reply soon. Type @ropensci-review-bot help for help.

ropensci-review-bot commented 8 months ago

Error (500). The editorcheck service is currently unavailable

maelle commented 8 months ago

@ropensci-review-bot check package

ropensci-review-bot commented 8 months ago

Thanks, about to send the query.

ropensci-review-bot commented 8 months ago

Error (500). The editorcheck service is currently unavailable

maelle commented 8 months ago

@micha-silver thanks for your submission! We'll fix this glitch in our system next week.

micha-silver commented 8 months ago

Hi @maelle : It's good to hear from you. I'll look forward to getting the review underway next week.

noamross commented 8 months ago

Thanks @micha-silver for your submission. As Maëlle has noted our automatic package checker has a glitch, one that we think is a GitLab-related edge case. (We have fewer - totally fine! - submissions from GitLab so the system has had less stress-testing against it).

In the meantime, I've determined that your package is in-scope. From an initial manual check of the repository I note a few important things that that will be required before proceeding to assign an editor and reviewers. It may be worth moving ahead on these while we get the automatic checks going:

maelle commented 8 months ago

@ropensci-review-bot check package

ropensci-review-bot commented 8 months ago

Thanks, about to send the query.

ropensci-review-bot commented 8 months ago

:rocket:

Editor check started

:wave:

ropensci-review-bot commented 8 months ago

Checks for rOPTRAM (v0.0.1.000)

git hash: dd810f2f

Important: All failing checks above must be addressed prior to proceeding

Package License: GPL (>= 3) + file LICENSE

1. Package Dependencies

Details of Package Dependency Usage (click to open)

The table below tallies all function calls to all packages ('ncalls'), both internal (r-base + recommended, along with the package itself), and external (imported and suggested packages). 'NA' values indicate packages to which no identified calls to R functions could be found. Note that these results are generated by an automated code-tagging system which may not be entirely accurate. |type |package | ncalls| |:----------|:---------|------:| |internal |base | 125| |internal |rOPTRAM | 20| |internal |datasets | 1| |imports |terra | 25| |imports |utils | 7| |imports |dplyr | 3| |imports |sf | 1| |imports |ggplot2 | NA| |imports |tools | NA| |suggests |xml2 | 17| |suggests |stats | 8| |suggests |sen2r | 1| |suggests |geojsonio | NA| |suggests |testthat | NA| |suggests |knitr | NA| |suggests |rmarkdown | NA| |linking_to |NA | NA| Click below for tallies of functions used in each package. Locations of each call within this package may be generated locally by running 's <- pkgstats::pkgstats()', and examining the 'external_calls' table.

base

file.path (16), lapply (15), c (13), basename (7), paste (7), strsplit (6), unlist (6), data.frame (5), list.files (5), as.data.frame (4), dir (4), nrow (4), grepl (3), sample (3), as.Date (2), do.call (2), length (2), rbind (2), split (2), t (2), as.character (1), dirname (1), gsub (1), log2 (1), max (1), min (1), package_version (1), paste0 (1), path.expand (1), row.names (1), suppressWarnings (1), Sys.which (1), system.file (1), T (1), try (1)

terra

rast (10), as.data.frame (6), writeRaster (4), ext (3), crs (1), project (1)

rOPTRAM

calculate_str (6), calculate_vi (5), aoi_to_name (3), check_scihub_access (2), optram_wetdry_coefficients (2), check_aoi (1), optram_calculate_str (1)

xml2

read_xml (5), xml_find_first (5), xml_text (5), xml_contents (1), xml_find_all (1)

stats

lm (3), quantile (3), offset (1), step (1)

utils

data (3), vi (3), packageVersion (1)

dplyr

full_join (3)

datasets

iris (1)

sen2r

check_gcloud (1)

sf

st_read (1)

**NOTE:** Some imported packages appear to have no associated function calls; please ensure with author that these 'Imports' are listed appropriately.

2. Statistical Properties

This package features some noteworthy statistical properties which may need to be clarified by a handling editor prior to progressing.

Details of statistical properties (click to open)

The package has: - code in R (100% in 13 files) and - 1 authors - 1 vignette - no internal data file - 6 imported packages - 15 exported functions (median 42 lines of code) - 21 non-exported functions in R (median 20 lines of code) --- Statistical properties of package structure as distributional percentiles in relation to all current CRAN packages The following terminology is used: - `loc` = "Lines of Code" - `fn` = "function" - `exp`/`not_exp` = exported / not exported All parameters are explained as tooltips in the locally-rendered HTML version of this report generated by [the `checks_to_markdown()` function](https://docs.ropensci.org/pkgcheck/reference/checks_to_markdown.html) The final measure (`fn_call_network_size`) is the total number of calls between functions (in R), or more abstract relationships between code objects in other languages. Values are flagged as "noteworthy" when they lie in the upper or lower 5th percentile. |measure | value| percentile|noteworthy | |:------------------------|-----:|----------:|:----------| |files_R | 13| 68.2| | |files_vignettes | 2| 85.7| | |files_tests | 11| 91.7| | |loc_R | 759| 60.2| | |loc_vignettes | 142| 37.0| | |loc_tests | 206| 55.5| | |num_vignettes | 1| 64.8| | |n_fns_r | 36| 45.9| | |n_fns_r_exported | 15| 58.5| | |n_fns_r_not_exported | 21| 42.1| | |n_fns_per_file_r | 2| 32.7| | |num_params_per_fn | 4| 54.6| | |loc_per_fn_r | 30| 76.5| | |loc_per_fn_r_exp | 42| 74.5| | |loc_per_fn_r_not_exp | 20| 63.0| | |rel_whitespace_R | 15| 56.6| | |rel_whitespace_vignettes | 23| 25.0| | |rel_whitespace_tests | 16| 47.0| | |doclines_per_fn_exp | 26| 25.3| | |doclines_per_fn_not_exp | 0| 0.0|TRUE | |fn_call_network_size | 20| 46.4| | ---

2a. Network visualisation

Click to see the interactive network visualisation of calls between objects in package

3. goodpractice and other checks

Details of goodpractice checks (click to open)

--- #### 3b. `goodpractice` results #### `R CMD check` with [rcmdcheck](https://r-lib.github.io/rcmdcheck/) R CMD check generated the following error: 1. Error in proc$get_built_file() : Build process failed R CMD check generated the following check_fail: 1. no_import_package_as_a_whole #### Test coverage with [covr](https://covr.r-lib.org/) ERROR: Test Coverage Failed #### Cyclocomplexity with [cyclocomp](https://github.com/MangoTheCat/cyclocomp) Error : Build failed, unknown error, standard output: * checking for file ‘roptram/DESCRIPTION’ ... OK * preparing ‘rOPTRAM’: * checking DESCRIPTION meta-information ... OK * installing the package to build vignettes ----------------------------------- * installing *source* package ‘rOPTRAM’ ... ** using staged installation ** R ** inst ** byte-compile and prepare package for lazy loading Error in library(hexSticker) : there is no package called ‘hexSticker’ Error: unable to load R code in package ‘rOPTRAM’ Execution halted ERROR: lazy loading failed for package ‘rOPTRAM’ * removing ‘/tmp/RtmpcVV09j/Rinst1adf150625cf/rOPTRAM’ ----------------------------------- ERROR: package installation failed #### Static code analyses with [lintr](https://github.com/jimhester/lintr) [lintr](https://github.com/jimhester/lintr) found the following 67 potential issues: message | number of times --- | --- Avoid 1:length(...) expressions, use seq_len. | 2 Avoid library() and require() calls in packages | 4 Lines should not be more than 80 characters. | 59 Use <-, not =, for assignment. | 2

Package Versions

|package |version | |:--------|:--------| |pkgstats |0.1.3.9 | |pkgcheck |0.1.2.10 |

Editor-in-Chief Instructions:

Processing may not proceed until the items marked with :heavy_multiplication_x: have been resolved.

micha-silver commented 8 months ago

Thanks: 

* We require that all packages have at least on vignette (https://devguide.ropensci.org/building.html#documentation)

I've written and added an appropriate vignette.

* We also require a CONTRIBUTING.md or similar file to provide guidance to potential package reviewers (https://devguide.ropensci.org/collaboration.html?q=contributing.md#contributing-guide)

Added CONTRIBUTING.md (in docs subdir)

* We require testing against the current (release), last (oldrel), and development (devel) versions of R, as well as across linux, macos, and windows platforms (usually just on release).  We also require coverage reporting of testing.  Based on your [.gitlab-ci.yml file](https://gitlab.com/rsl-bidr/roptram/-/blob/main/.gitlab-ci.yml), I think you are only testing on `release`. (https://devguide.ropensci.org/ci.html)

Yes, I have only one CI so far. I'll start preparing tests for additional releases and OS.

noamross commented 8 months ago

Thanks, @micha-silver. Now that we have automated checks, I also see that the repo doesn't show the test coverage (though you may be measuring this locally as I see you have a .covrignore file). Since reviewers may be less familiar with navigation in GitLab, please put badges in your readme to point us to the CI build logs and coverage outputs once those are set up.

micha-silver commented 8 months ago

Correct, I'm running covr locally.

My coverage is not good, (about 48%) and I'd like some advice here: I have three main functions that pull down multiple Sentinel 2 or Landsat imagery tiles, over a time span.

noamross commented 8 months ago

This is a common challenge we see. There are several complementary strategies one can take:

micha-silver commented 7 months ago

@noamross : I'm considering migrating the rOPTRAM repo from gitlab to github. I think I'll be able to configure the CI tests more easily on github (lots more examples...). And since most ROpenSci projects are on github, the review process might also be smoother. Shall I go ahead with this? Anything I need to be aware of??

Regarding your explanation above about testing functions with large downloads, I'd like to go with the 4th option, setting a condition, and skipping those tests (with explanation in CONTRIBUTING).

Thanks, Micha

noamross commented 7 months ago

@micha-silver No problem for us if you move to GitHub, though we are happy to do the process either way. We might have to make a tweak so the bot knows that the repository is in a new location, so let us know when you switch.

micha-silver commented 7 months ago

Hi: I've made progress with the rOPTRAM package:

  1. Added examples to all functions

  2. CI tests are now integrated.

  3. covr::package_coverage() now shows: 

    r$> covr::package_coverage()
rOPTRAM Coverage: 75.49%
R/optram_acquire_s2.R: 17.86%
R/utilities.R: 85.29%
R/optram_wetdry_coefficients.R: 95.10%
R/optram_ndvi_str.R: 95.12%
R/optram_calculate_str.R: 95.45%
R/optram_soilmoisture.R: 100.00%

    (TODO: Still needs to be improved. I've added one addition testthat function to test downloading in the optram_acquire_s2.R function by limiting the timespan so that only one sentinel tile is downloaded. But that function still has low coverage. )

  4. I run tests on 5 platforms: Windows release, Windows oldrel, Windows devel, MacOS, and Ubuntu. The gitlab-ci.yml performs these tests by sending to rhub. Here's the summary currently:

                  platform errors warnings notes
       ubuntu-gcc-devel      0        0     0
 windows-x86_64-release      0        0     0
  windows-x86_64-oldrel      0        0     0
   windows-x86_64-devel      0        0     0
          macos_release      0        0    11

[1] "2023-11-26 11:21:16.729992  - Check completed"

Here are links to the results: for Ubuntu: for Win release: release for Win devel: for Win oldrel: for Mac macbuilds:

Not sure how long these artifacts are kept on rhub. But the full artifact on gitlab is available: artifacts on gitlab

What's next? Best regards, Micha

micha-silver commented 7 months ago

Following up: On my local machine I get good coverage:

r$> cvr <- covr::package_coverage()

r$> cvr
rOPTRAM Coverage: 92.70%
R/utilities.R: 85.29%
R/optram_acquire_s2.R: 90.59%
R/optram_wetdry_coefficients.R: 95.10%
R/optram_ndvi_str.R: 95.12%
R/optram_calculate_str.R: 95.45%
R/optram_soilmoisture.R: 100.00%

The explanation: The optram_acquire_s2() function relies on Google Cloud CLI to download Sentinel images. I have that SDK installed locally, but I have not seen any way to install non-interactively, on a rocker image. There seems to be no work around to get access authentication to Google services, and the gsutil utility setup in a CI machine. I tried to copy my Google access token into a CI variable, and use that, but there is no way to get gsutil setup automatically to recognize a username and token. So when I run covr in gitlab-ci it skips the whole download procedure, and shows lower coverage.

noamross commented 7 months ago

Hi @micha-silver, thanks for moving a ahead on this. It's good you got to 75% coverage without the gsutil dependency! I note that https://cloud.google.com/sdk/docs/downloads-interactive#silent to have some guidance for installing on CI services. I'd make good-faith effort with that, but if it is not possible a full description in the install section of the README as well as CONTRIBUTING for those running tests will be sufficient. I note this is one thing that we might ask reviewers to comment on in the next stage - would another dependency be a simpler way to fetch the images?

micha-silver commented 7 months ago

@noamross I appreciate your encouraging words.

Two comments

  1. The link you suggested to an automated install of the Gcloud CLI works fine. The problem is that after installing you still need to be authenticated. And that, AFAIK, requires an interactive step: Google forces you to go to a certain website, put in credentials and get the OAuth token on your machine. I tried to put a valid token into a gitlab CI variable, then somehow use that to manually recreate the local auth directory (in a docker image) that gcloud requires - no success here :-(. Even if we could find some work around, it would surely be very fragile.
  2. As to a simpler way to fetch the imagery: Indeed Copernicus now (since Oct 2023) offers several new options for downloading - many more than when I began this package. We have recruited another contributor who will help me to weed thru these various options, and implement the best, smoothest ones. If you agree, I'd like to present the package in its current state for review, then add the additional image acquisition capabilities in a future version.
noamross commented 7 months ago

It should be fine to present the package in its current state. Let me re-run checks.

noamross commented 7 months ago

@ropensci-review-bot check package

ropensci-review-bot commented 7 months ago

Thanks, about to send the query.

ropensci-review-bot commented 7 months ago

:rocket:

Editor check started

:wave:

ropensci-review-bot commented 7 months ago

Checks for rOPTRAM (v0.0.1.000)

git hash: b54b2c5f

Important: All failing checks above must be addressed prior to proceeding

Package License: GPL (>= 3) + file LICENSE

1. Package Dependencies

Details of Package Dependency Usage (click to open)

The table below tallies all function calls to all packages ('ncalls'), both internal (r-base + recommended, along with the package itself), and external (imported and suggested packages). 'NA' values indicate packages to which no identified calls to R functions could be found. Note that these results are generated by an automated code-tagging system which may not be entirely accurate. |type |package | ncalls| |:----------|:---------|------:| |internal |base | 120| |internal |rOPTRAM | 19| |imports |terra | 38| |imports |sf | 5| |imports |utils | 4| |imports |dplyr | 3| |imports |ggplot2 | NA| |imports |tools | NA| |suggests |xml2 | 13| |suggests |stats | 9| |suggests |sen2r | 1| |suggests |geojsonio | NA| |suggests |lwgeom | NA| |suggests |knitr | NA| |suggests |rmarkdown | NA| |suggests |testthat | NA| |linking_to |NA | NA| Click below for tallies of functions used in each package. Locations of each call within this package may be generated locally by running 's <- pkgstats::pkgstats()', and examining the 'external_calls' table.

base

lapply (15), c (14), file.path (14), basename (7), paste (7), strsplit (6), unlist (6), data.frame (5), list.files (5), nrow (4), dir (3), sample (3), as.Date (2), do.call (2), drop (2), grepl (2), length (2), rbind (2), split (2), t (2), as.character (1), dirname (1), gsub (1), log (1), log2 (1), max (1), min (1), package_version (1), paste0 (1), row.names (1), suppressWarnings (1), Sys.getenv (1), T (1), try (1), tryCatch (1)

terra

as.data.frame (16), rast (12), writeRaster (4), ext (3), crs (1), mean (1), project (1)

rOPTRAM

calculate_str (6), calculate_vi (5), aoi_to_name (3), optram_wetdry_coefficients (2), check_aoi (1), check_scihub_access (1), optram_calculate_str (1)

xml2

read_xml (5), xml_text (5), xml_contents (1), xml_find_all (1), xml_find_first (1)

stats

lm (3), quantile (3), offset (1), proj (1), step (1)

sf

st_read (3), st_zm (2)

utils

data (3), packageVersion (1)

dplyr

full_join (3)

sen2r

check_gcloud (1)

**NOTE:** Some imported packages appear to have no associated function calls; please ensure with author that these 'Imports' are listed appropriately.

2. Statistical Properties

This package features some noteworthy statistical properties which may need to be clarified by a handling editor prior to progressing.

Details of statistical properties (click to open)

The package has: - code in R (100% in 11 files) and - 1 authors - 1 vignette - no internal data file - 6 imported packages - 15 exported functions (median 32 lines of code) - 15 non-exported functions in R (median 40 lines of code) --- Statistical properties of package structure as distributional percentiles in relation to all current CRAN packages The following terminology is used: - `loc` = "Lines of Code" - `fn` = "function" - `exp`/`not_exp` = exported / not exported All parameters are explained as tooltips in the locally-rendered HTML version of this report generated by [the `checks_to_markdown()` function](https://docs.ropensci.org/pkgcheck/reference/checks_to_markdown.html) The final measure (`fn_call_network_size`) is the total number of calls between functions (in R), or more abstract relationships between code objects in other languages. Values are flagged as "noteworthy" when they lie in the upper or lower 5th percentile. |measure | value| percentile|noteworthy | |:------------------------|-----:|----------:|:----------| |files_R | 11| 62.6| | |files_vignettes | 2| 85.7| | |files_tests | 10| 90.7| | |loc_R | 761| 60.3| | |loc_vignettes | 157| 40.5| | |loc_tests | 253| 60.4| | |num_vignettes | 1| 64.8| | |n_fns_r | 30| 40.1| | |n_fns_r_exported | 15| 58.5| | |n_fns_r_not_exported | 15| 32.7| | |n_fns_per_file_r | 2| 25.6| | |num_params_per_fn | 5| 69.6| | |loc_per_fn_r | 37| 83.0| | |loc_per_fn_r_exp | 32| 65.2| | |loc_per_fn_r_not_exp | 40| 86.1| | |rel_whitespace_R | 14| 55.1| | |rel_whitespace_vignettes | 30| 36.6| | |rel_whitespace_tests | 16| 52.0| | |doclines_per_fn_exp | 33| 38.5| | |doclines_per_fn_not_exp | 0| 0.0|TRUE | |fn_call_network_size | 18| 44.2| | ---

2a. Network visualisation

Click to see the interactive network visualisation of calls between objects in package

3. goodpractice and other checks

Details of goodpractice checks (click to open)

--- #### 3b. `goodpractice` results #### `R CMD check` with [rcmdcheck](https://r-lib.github.io/rcmdcheck/) R CMD check generated the following check_fail: 1. no_import_package_as_a_whole #### Test coverage with [covr](https://covr.r-lib.org/) Package coverage: 74.44 #### Cyclocomplexity with [cyclocomp](https://github.com/MangoTheCat/cyclocomp) The following functions have cyclocomplexity >= 15: function | cyclocomplexity --- | --- optram_safe | 29 optram_landsat | 20 #### Static code analyses with [lintr](https://github.com/jimhester/lintr) [lintr](https://github.com/jimhester/lintr) found the following 17 potential issues: message | number of times --- | --- Avoid changing the working directory, or restore it in on.exit | 1 Lines should not be more than 80 characters. | 15 Use <-, not =, for assignment. | 1

Package Versions

|package |version | |:--------|:--------| |pkgstats |0.1.3.9 | |pkgcheck |0.1.2.11 |

Editor-in-Chief Instructions:

Processing may not proceed until the items marked with :heavy_multiplication_x: have been resolved.

micha-silver commented 7 months ago

I have improved the coverage of optram_acquire_s2() by moving the actual download code to a separate file, and entering that new file into .covrignore.

CI tests are, if I understand correctly, implemented in the .gitlab-ci.yml file. In that file I put a rule to run the pipeline manually (to avoid firing off a check at every minor commit). Is that sufficient to fill the requirement for "✖️ Package has no continuous integration checks."?

I also fixed the lintr comments.

noamross commented 7 months ago

Thanks, for the CI the reason for failure is that we don't yet have a standard way to detect it on GitLab. We'll work on that in the future, both detection and some guidance on how to signal it for non-GitHub users. In the meantime, I'm assigning an editor to get this going!

noamross commented 7 months ago

@ropensci-review-bot assign @adamhsparks as editor

ropensci-review-bot commented 7 months ago

Assigned! @adamhsparks is now the editor

adamhsparks commented 7 months ago

Hi @micha-silver, I'll start looking for reviewers, but in the meantime I've found a few issues that should be corrected before they start their reviews.

Documentation

Package Overall

micha-silver commented 7 months ago

Hello @adamhsparks Thanks for your helpful comments. I've addressed most:

  1. Corrected formatting for imported packages in help files;
  2. Formatted all function titles in Title Case;
  3. Spell check done;
  4. Corrected 'T' to 'TRUE';
  5. Removed extra bibliography details from DESCRIPTION;
  6. As for paper.md and bibliography.bib, I intend to submit a short paper with the package. Is this not the proper place to have the paper in the same repo as the code?
  7. Regarding the vignette, I guess I don't yet understand how this should work :-( . When I build a source tarball, then install from that, as suggested in r-bloggers then browseVignettes(package = "rOPTRAM") works as expected. Am I missing something?
adamhsparks commented 7 months ago

Hi @micha-silver,

I'll try to answer your queries, if I'm still not providing enough information, please do not hesitate to ask for further clarification.

paper.md and bibliography.bib

Traditionally these would be in "inst", e.g., https://github.com/ropensci/nasapower/tree/main/inst/paper. Not in the top level directory of the package.

Vignette

You have a file, _rOPTRAM.Rmd.orig_, therefore I assumed you were pre-compiling, which given that this package uses Internet resources, is a good idea. However, I do not see the script that you are using to actually compile the .Rmd file of the vignette itself, e.g., https://github.com/ropensci/nasapower/blob/main/vignettes/precompile.R

micha-silver commented 6 months ago

@adamhsparks :

OK, that's clear now. I've moved the paper into /inst, and added the precompile script as you suggested.

One glitch, maybe you have some idea: I want a tmap image in the vignette (the final code chunk). However when I knit the rOPTRAM.Rmd.orig, I get this error at the end, inside the final compiled rOPTRAM.Rmd:

#> Error in path.expand(path): invalid 'path' argument

and there's no image in the output HTML, just the same error message.

Any ideas?

Thanks, Micha

adamhsparks commented 6 months ago

Hi @micha-silver, I'm trying to troubleshoot this for you but I'm having difficulties getting everything set up. I think one thing that will be necessary for users of this package will be to have very clear instructions on how to set up the computing environment to use this package as the normal install.packages("rOPTRAM") won't work here. I've tried to set up the Google tools that are required and have created a new roptram project but I still get this error in R:

Searching for a valid Google Cloud SDK installation...
No access to Google cloud
Exiting
Warning message:
In normalizePath(path, ...) : path[1]="": No such file or directory

I can get the vignette to properly knit using this code, which avoids using setwd() so it is portable. This also fixes the output directory, since I don't have a "/home/micha/EO_Data/Israel/Migda_full" directory to knit into, again, portability.

This assumes that you're using an RStudio Project. If not, I'd suggest using {here} for your directory structure handling rather than using setwd().

# vignette that depends on Internet access need to be pre-compiled and takes a
# while to run
library("knitr")
knit("vignettes/rOPTRAM.Rmd.orig", "vignettes/rOPTRAM.Rmd")

# remove file path such that vignettes will build with figures
replace <- readLines("vignettes/rOPTRAM.Rmd")
replace <- gsub("<img src=\"vignettes/", "<img src=\"", replace)
fileConn <- file("vignettes/rOPTRAM.Rmd")
writeLines(replace, fileConn)
close(fileConn)

# build vignette
library("devtools")
build_vignettes()

# move resource files to /docs
resources <-
  list.files("vignettes/", pattern = ".png$", full.names = TRUE)
file.copy(from = resources,
          to = "docs",
          overwrite =  TRUE)

It knits, but when I tried to step through the examples, the examples in the vignette don't work for me due to the use of hardcoded directories, once again. Here I used, tempdir() to fix that and I'm unable to configure my "Google Cloud" environment properly for some reason, so I cannot generate the whole vignette as it should appear.

Maybe this helps you get it to knit properly and will be useful for the reviewers to suggest ways that the documentation can be improved on how to set up the computing environment.

Also, I noted that,

adamhsparks commented 6 months ago

Editor checks:

Editor comments

Everything checks out with the normal editor checks. I'll start looking for reviewers now.

micha-silver commented 6 months ago

My last commit:

Best regards,

adamhsparks commented 6 months ago

@micha-silver, I'll have a look tomorrow if I have some time. Thanks for all the effort thus far!

adamhsparks commented 6 months ago

Hi @micha-silver, I would advise against your examples writing to "./inst" in the vignette as you do when saving the data. There is also no need to use {here} in your package (aside from the pre-compile function, which isn't really part of the R package so {here} won't need to be in the DESCRIPTION file anywhere) for these examples, just use tempdir() for your vignette examples as I previously suggested. You're vignette also indicates in the text that the downloads are saved to tempdir() whereas you're putting them in "./inst".

Also, there is no need to install {rOPTRAM} in the vignette example. One could assume that the user browsing the documents locally has installed it locally so that they can view the docs locally.

Unfortunately I can't really troubleshoot the map for you since I can't run the functions in the vignette due to difficulties with the Google tools that you require, sorry. Perhaps the reviewers will be able to assist with that.

micha-silver commented 6 months ago

@adamhsparks Thanks for your patience with this question.

I'm hesitating to use tempdir() only because the download for each compile of the vignette is around 50GB, and takes around 1.5 hrs. (necessary to get a reasonable/realistic trapezoid from the model). If I define a local, permanent directory for my precompile script, then the package "sees" the files already downloaded from a previous compile, and completes the vignette in a few minutes. If I use tempdir() then obviously each minor change to the vignette will require a long wait to see the results.

In any case, let's wait for reviewers, and then I'll revisit this issue.

adamhsparks commented 6 months ago

Ah, well, since you’re precompiling and it’s an Rmd, you could use a hidden code chunk to download and knit but display another for the user rather than using tempdir() for everything. That would be my suggestion. But putting it in “./inst” for the example is poor form, IMO. Use something like “~/Downloads” or the current working directory for the example code shown to the user while you put the real file somewhere else in your local system.

On 6 Dec 2023, at 17:44, Micha Silver @.***> wrote:

@adamhsparks https://github.com/adamhsparks Thanks for your patience with this question.

I'm hesitating to use tempdir() only because the download for each compile of the vignette is around 50GB, and takes around 1.5 hrs. (necessary to get a reasonable/realistic trapezoid from the model). If I define a local, permanent directory for my precompile script, then the package "sees" the files already downloaded from a previous compile, and completes the vignette in a few minutes. If I use tempdir() then obviously each minor change to the vignette will require a long wait to see the results.

In any case, let's wait for reviewers, and then I'll revisit this issue.

— Reply to this email directly, view it on GitHub https://github.com/ropensci/software-review/issues/612#issuecomment-1842529588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYMIAV5FZRQNX2AZUW2GPTYIA46LAVCNFSM6AAAAAA56DYGH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBSGUZDSNJYHA. You are receiving this because you were mentioned.

adamhsparks commented 6 months ago

@ropensci-review-bot assign @harryeslick as reviewer

ropensci-review-bot commented 6 months ago

@harryeslick added to the reviewers list. Review due date is 2023-12-29. Thanks @harryeslick for accepting to review! Please refer to our reviewer guide.

rOpenSci’s community is our best asset. We aim for reviews to be open, non-adversarial, and focused on improving software quality. Be respectful and kind! See our reviewers guide and code of conduct for more.

ropensci-review-bot commented 6 months ago

@harryeslick: If you haven't done so, please fill this form for us to update our reviewers records.

adamhsparks commented 6 months ago

@harryeslick, as discussed, take the time that you need given the holidays coming up

micha-silver commented 6 months ago

@adamhsparks Thanks for your help in enlisting reviewers. Looking forward to working with @harryeslick

adamhsparks commented 6 months ago

@ropensci-review-bot add @obrl-soil as reviewer

ropensci-review-bot commented 6 months ago

@obrl-soil added to the reviewers list. Review due date is 2023-12-29. Thanks @obrl-soil for accepting to review! Please refer to our reviewer guide.

rOpenSci’s community is our best asset. We aim for reviews to be open, non-adversarial, and focused on improving software quality. Be respectful and kind! See our reviewers guide and code of conduct for more.

ropensci-review-bot commented 6 months ago

@obrl-soil: If you haven't done so, please fill this form for us to update our reviewers records.