search

ropensci / ruODK

ruODK: An R Client for the ODK Central API
https://docs.ropensci.org/ruODK/
GNU General Public License v3.0
42 stars 13 forks source link

odata_submission_get not downloading attachments from top level form fields #93

Closed SamNelson081 closed 4 years ago

SamNelson081 commented 4 years ago

Problem

When calling odata_submission_get(fid="", download=TRUE, local_dir="media"), a media directory appears but with no downloaded attachments.

Downloading a copy of the current master branch and adding some debugging statements, i was able to find this piece of code, when odata_submission_get() calls handle_ru_attachments(), the following can be found inside handle_ru_attachments(): 

att_cols <- form_schema %>%
    dplyr::filter(type == "binary") %>%
    magrittr::extract2("ruodk_name") %>%
    intersect(names(data))
if (verbose == TRUE) {
    x <- paste(att_cols, collapse = ", ") # nolint
    ru_msg_info(glue::glue("Found attachments: {x}."))
    ru_msg_info("Downloading attachments...")
}

The output of running odata_submission_get() with verbose enabled, produces the following

ℹ Found attachments: . 
ℹ Downloading attachments...

This seemingly insinuates that attr_cols is returning no data, and therefore not getting collapsed into the verbose message.

Following up with print() statements led me to the following: 

 [1] "Form Schema" 
 # A tibble: 5 x 5   
    path             name            type      binary ruodk_name
    <chr>            <chr>           <chr>     <lgl>  <chr>           
  1 /meta            meta            structure NA     meta           
  2 /meta/instanceID instanceID      string    NA     meta_instanceID 
  3 /User_QR_Code    User_QR_Code    barcode   NA     User_QR_Code   
  4 /User_Name       User_Name       string    NA     User_Name       
  5 /User_Photograph User_Photograph binary    TRUE   User_Photograph 
[1] "att_cols" 
    character(0)

Where form schema is a print on form schema before att_cols is assigned, and att_cols is afterwards.

I'm not a tidyverse user, so i'm not 100% sure how the att_cols assignment should be working, but given this example it may be malformed?

Reproducible example

library(ruODK)

ruODK::ru_setup(
  svc = "https://ksa-central.3dsupportplatform.com/v1/projects/2/forms/Create_New_User.svc",
  un = "<omitted>",
  pw = "<omitted>",
  tz = "Australia/Sydney",
  verbose = TRUE, # great for demo or debugging,
  url="https://ksa-central.3dsupportplatform.com"
)

odata_submission_get(fid="Create_New_User")
Session Info ```{r} R version 4.0.2 (2020-06-22) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Catalina 10.15.6 Matrix products: default BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib locale: [1] en_AU.UTF-8/en_AU.UTF-8/en_AU.UTF-8/C/en_AU.UTF-8/en_AU.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached): [1] compiler_4.0.2 shiny.i18n_0.1.0 assertthat_0.2.1 cli_2.0.2 tools_4.0.2 glue_1.4.1 rstudioapi_0.11 yaml_2.2.1 crayon_1.3.4 fansi_0.4.1 jsonlite_1.7.0 ```
florianm commented 4 years ago

Thanks for the bug report! This is definitely not the intended behaviour of ruODK::odata_submission_get().

att_cols should be c("User_Photograph"), and odata_submission_get should print and download the attachment "User_Photograph". E.g. the test https://github.com/ropensci/ruODK/blob/main/tests/testthat/test-odata_submission_get.R#L28 finds its form's attachments.

If you call odata_submission_get(fid="") with an empty string as fid, this empty string will override the default (get_default_fid()) which was set by the OData Service URL through ru_setup(svc="...").

If you are set up correctly, a simple call to form_schema() (without any args) should return the correct form schema with "User_Photograph".

SamNelson081 commented 4 years ago

Hi florianm,

Thanks for your reply!

What version of ODK Central are you running?

I'm not certain of what version the server is currently running, it was set up by a co-worker less than a week ago, so my assumption is the latest. I've asked them if they know, so i'll get back to you if they are aware of the version (I can't seem to find the version on the User Interface).

Could I test your form to reproduce this on the ODK Central Sandbox?

Of course! I assume best way would be by me providing XML? I'll put it below for you, but feel free to reply with other instructions if needed!

<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:jr="http://openrosa.org/javarosa">
  <h:head>
    <h:title>Create New User</h:title>
    <model>
      <instance>
        <data id="Create_New_User">
          <meta>
            <instanceID/>
          </meta>
          <User_QR_Code/>
          <User_Name/>
          <User_Photograph/>
        </data>
      </instance>
      <itext>
        <translation lang="English">
          <text id="/data/User_QR_Code:label">
            <value>Scan the barcode to be used to identify this user</value>
          </text>
          <text id="/data/User_QR_Code:hint">
            <value>You need to create a QR code that uniquely identifies this staff member and scan it.</value>
          </text>
          <text id="/data/User_Name:label">
            <value>Enter the full name of this staff member</value>
          </text>
          <text id="/data/User_Photograph:label">
            <value>Photograph of the staff member (optional)</value>
          </text>
        </translation>
      </itext>
      <bind nodeset="/data/meta/instanceID" type="string" readonly="true()" calculate="concat('uuid:', uuid())"/>
      <bind nodeset="/data/User_QR_Code" type="barcode" required="true()"/>
      <bind nodeset="/data/User_Name" type="string" required="true()"/>
      <bind nodeset="/data/User_Photograph" type="binary"/>
    </model>
  </h:head>
  <h:body>
    <input ref="/data/User_QR_Code">
      <label ref="jr:itext('/data/User_QR_Code:label')"/>
      <hint ref="jr:itext('/data/User_QR_Code:hint')"/>
    </input>
    <input ref="/data/User_Name">
      <label ref="jr:itext('/data/User_Name:label')"/>
    </input>
    <upload ref="/data/User_Photograph" mediatype="image/*">
      <label ref="jr:itext('/data/User_Photograph:label')"/>
    </upload>
  </h:body>
</h:html>

Did you call odata_submission_get(fid="Create_New_User") or odata_submission_get(fid="", download=TRUE, local_dir="media")?

My bad, i can see how this could be misinterpreted. The code called is definitely odata_submission_get(fid="Create_New_User"), i have tried both with default values for download and local_dir, as well as with defined values.

I will note that i am able to retrieve the file correctly using get_one_attachment()

svc <- odata_service_get()
submission <- odata_submission_get()
fn <- submission$user_photograph[1]
id <- submission$id[1]
att_url <- ruODK:::attachment_url(id, fn)
userPhoto <- get_one_attachment(pth = "~/Desktop/test.jpg", fn = fn, src = att_url)
florianm commented 4 years ago

Found the bug. ruODK's form_schema_parse fails to parse top level fields outside a form "group". I've saved your form as an XML file and uploaded that to the ODK Central sandbox project "ruODK playground".

Your original form:

ruODK::ru_setup(
  svc = "https://sandbox.central.getodk.org/v1/projects/44/forms/Create_New_User.svc",
  un = ruODK::get_test_un(),
  pw = ruODK::get_test_pw(),
  odkc_version = 1.0
)
fs <- ruODK::form_schema()
x <- ruODK::odata_submission_get(download=T, local_dir = ".")
ℹ Downloading submissions...
✔ Downloaded submissions.
ℹ Reading form schema...
ℹ Form schema v1
ℹ Parsing submissions...
ℹ Not unnesting geo fields: value_   # hand on, that's my "don't unnest GeoJSON" mistaking the three top level fields (QR code, username, photograph) for a geo field.. my bad!
ℹ Unnesting: value
ℹ Unnesting column "value"
ℹ Unnesting more list cols: value___system, value_meta
ℹ Not unnesting geo fields: value_ # yeah nah that's not geojson
ℹ Unnesting: value___system, value_meta
ℹ Unnesting column "value___system"
ℹ Unnesting column "value_meta"
ℹ Found date/times: .
ℹ Found attachments: . # strange
ℹ Downloading attachments...
ℹ Found geopoints: .
ℹ Found geotraces: .
ℹ Found geoshapes: .
✔ Returning parsed submissions.

No attachments found = no attachments downloaded.

Your form re-implemented in ODK Build (zip contains .odkbuild, .xml, .xls) CreateNewUser01.zip

R> fs <- ruODK::form_schema()
ℹ Form schema v1
R> fs
# A tibble: 8 x 5
  path             name          type      binary ruodk_name     
  <chr>            <chr>         <chr>     <lgl>  <chr>          
1 /meta            meta          structure NA     meta           
2 /meta/instanceID instanceID    string    NA     meta_instanceID
3 /subscriber_id   subscriber_id string    NA     subscriber_id  
4 /start_time      start_time    dateTime  NA     start_time     
5 /user            user          structure NA     user           
6 /user/qrcode     qrcode        barcode   NA     user_qrcode    
7 /user/name       name          string    NA     user_name      
8 /user/photo      photo         binary    TRUE   user_photo     
R> x <- ruODK::odata_submission_get(download=T, local_dir = ".")
ℹ Downloading submissions...
✔ Downloaded submissions.
ℹ Reading form schema...
ℹ Form schema v1
ℹ Parsing submissions...
ℹ Not unnesting geo fields: value_
ℹ Unnesting: value
ℹ Unnesting column "value"
ℹ Unnesting more list cols: value___system, value_meta, value_user
ℹ Not unnesting geo fields: value_
ℹ Unnesting: value___system, value_meta, value_user
ℹ Unnesting column "value___system"
ℹ Unnesting column "value_meta"
ℹ Unnesting column "value_user"
ℹ Found date/times: start_time.
ℹ Found attachments: user_photo.
ℹ Downloading attachments...
ℹ Using local directory ".".
✔ File saved to "./turtle-14_4_17.png".
ℹ Found geopoints: .
ℹ Found geotraces: .
ℹ Found geoshapes: .
✔ Returning parsed submissions.

While I'll have to upgrade form_schema_parse's "skip unnesting GeoJSON" logic, you can place your form fields inside a "Group" (and cluster widgets through "keep on one screen", or not), which then will work in ruODK. Is that workaround a suitable patch for you?

SamNelson081 commented 4 years ago

This works fine for me! Thank you very much for your assistance in this bug! It is Very much appreciated!

florianm commented 4 years ago

@SamNelson081 ruODK now supports your form by cleaning form_schema ruodk_name the same way it cleans submission columns. Thanks for pointing out the bug and supplying all the detail info!