SpikeLewis commented 2 years ago

Problem

I'm trying to use RuODK but having trouble accessing the data that has been encrypted at the project level. sorry if im missing something obvious.

Reproducible example

# insert reprex here

Session Info

```{r} # utils::sessionInfo() ```

florianm commented 2 years ago

What functions did you use? Encryption is not supported in the OData and RESTful API endpoints, so https://docs.ropensci.org/ruODK/reference/odata_submission_get.html or https://docs.ropensci.org/ruODK/reference/submission_get.html won't work, but you can use https://docs.ropensci.org/ruODK/reference/submission_export.html (the CSV.ZIP export). Does the latter work for you?

FYI we're working on a vignette explaining encryption a bit better: https://github.com/ropensci/ruODK/issues/112

SpikeLewis commented 2 years ago

Hi Florian, Thank you for this message. I'm getting error 400 "Error in response %>% httr::stop_for_status(task = glue::glue("get desired response from server {url} as user \"{un}\".\n\n", : Bad Request (HTTP 400). Failed to get desired response from server https://myserver as user "my_email".

Using this (I've checked all the credentials, and they are correct). Sorry this has stumped me!

Spike

From: Florian Mayer @.> Sent: Friday, April 8, 2022 2:56 AM To: ropensci/ruODK @.> Cc: SpikeLewis @.>; Author @.> Subject: Re: [ropensci/ruODK] issue accessing data decrypted at project level (Issue #134)

What functions did you use? Encryption is not supported in the OData API endpoints, but you can use https://docs.ropensci.org/ruODK/reference/submission_export.html (the CSV.ZIP export). Does the latter work for you?

— Reply to this email directly, view it on GitHubhttps://github.com/ropensci/ruODK/issues/134#issuecomment-1092365610, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYTKCFHYQT2OEGJ6AJMKSX3VD6G5XANCNFSM5SZKNHMQ. You are receiving this because you authored the thread.Message ID: @.***>

florianm commented 2 years ago

Could you provide a reprex please? At least answer my question above re which function?

Status 400 is an authentication error, so it points to a wrong credential or maybe a lack of permissions of that account in ODK Central.

SpikeLewis commented 2 years ago

Hi Florian, Sorry I'm not sure what function you're asking about, but hopefully the following is reproducible.

if (!requireNamespace("remotes")) install.packages("remotes")

Full install

remotes::install_github( @.***", dependencies = TRUE, upgrade = "always", build_vignettes = TRUE)

usethis::edit_r_environ(scope = "user")

library(ruODK)

ru_setup( url = Sys.getenv("ODKC_URL"), un = Sys.getenv("ODKC_UN"), pw = Sys.getenv("ODKC_PW"), pp = Sys.getenv("ODKC_PP"), tz = "Australia/Perth", pid = Sys.getenv("ODKC_PID"), odkc_version = "1.1", fid = Sys.getenv("ODKC_FID"), test_pw = Sys.getenv("ODKC_TEST_PW"), test_un = Sys.getenv("ODKC_TEST_UN"), test_pp = Sys.getenv("ODKC_TEST_PP"), test_url = Sys.getenv("ODKC_TEST_URL"), test_pid = "1", test_fid = Sys.getenv("ODKC_TEST_FID"), test_fid_zip = Sys.getenv("ODKC_TEST_FID_ZIP", ""), test_fid_gap = Sys.getenv("ODKC_TEST_FID_ATT", ""), test_fid_att = Sys.getenv("ODKC_TEST_FID_ATT", ""), test_odkc_version = "1.1", test_fid_wkt = Sys.getenv("ODKC_TEST_FID_WKT", ""), verbose = TRUE)

submission_export( local_dir = here::here(), overwrite = TRUE, media = TRUE, repeats = TRUE, pid = get_default_pid(), fid = get_default_fid(), url = get_default_url(), un = get_default_un(), pw = get_default_pw(), pp = get_default_pp(), retries = get_retries(), odkc_version = get_default_odkc_version(), verbose = get_ru_verbose() )

From: Florian Mayer @.> Sent: Wednesday, April 13, 2022 2:46 AM To: ropensci/ruODK @.> Cc: SpikeLewis @.>; Author @.> Subject: Re: [ropensci/ruODK] issue accessing data decrypted at project level (Issue #134)

Could you provide a reprex please? At least answer my question above re which function?

Status 400 is an authentication error, so it points to a wrong credential or maybe a lack of permissions of that account in ODK Central.

— Reply to this email directly, view it on GitHubhttps://github.com/ropensci/ruODK/issues/134#issuecomment-1097465414, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYTKCFGISKYYBN7O5OBRKOLVEYROXANCNFSM5SZKNHMQ. You are receiving this because you authored the thread.Message ID: @.***>

yanokwa commented 2 years ago

I've confirmed that there is an issue with @SpikeLewis's passphrase. It works when exporting directly from Central, but not with ruODK. I will continue to investigate to see if we can get a repro.

SpikeLewis commented 2 years ago

Hi all, Just to update it seems to not be an issue with passphrase (it works on a dummy project). My only other thought is that it is the number of forms within the project (I'm on 3000+ forms). As I said, I can get in, and see that there are submissions, it's just throwing the error up on decryption. Going in via central itself, there is no issues downloading and decrypting from there. Best, Spike

From: Yaw Anokwa @.> Sent: Thursday, April 14, 2022 6:58 PM To: ropensci/ruODK @.> Cc: SpikeLewis @.>; Mention @.> Subject: Re: [ropensci/ruODK] issue accessing data decrypted at project level (Issue #134)

FYI, I've confirmed that there is an issue with @SpikeLewishttps://github.com/SpikeLewis's passphrase. It works when exporting directly from Central, but not with ruODK. I will continue to investigate to see if we can get a repro.

— Reply to this email directly, view it on GitHubhttps://github.com/ropensci/ruODK/issues/134#issuecomment-1099472107, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYTKCFH6JR74L7POHYMVB3LVFBMDDANCNFSM5SZKNHMQ. You are receiving this because you were mentioned.Message ID: @.***>

yanokwa commented 2 years ago

I've now tried a project with 3000 forms with no success. I'm now thinking it might be a network issue with OP's connection.

florianm commented 2 years ago

Thanks both for the troubleshooting. Looking at the code for submission_export, there are some limitations in ruODK:

Only the first encryption key is used
If the server throws an internal server error (HTTP status 500), ruODK terminates the export as not to exhaust server memory: https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L183-L188 - seeing you've got a different error caught in the next command (stop_for_status) tells me we're not getting a HTTP status 500, so the encryption key seems to have worked.

@SpikeLewis some more diagnostic questions:

Can you reproduce this error with the same form, same encryption method, and fewer submissions?
Would you know how many different encryption keys your form uses?
How long does submission_export run until you get the HTTP status 500?
Did you see this info message (with verbose=TRUE): https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L167-L168

florianm commented 2 years ago

@SpikeLewis did you get to investigate this further?

SpikeLewis commented 2 years ago

Hi all, Apologies for the delay (appendix). The same exact code seems to be now working, with no errors (Yaw suggested it might be an internet stability thing). I'm now unable to replicate the errors (apologies for consuming any of your time on this). just to note:

Can you reproduce this error with the same form, same encryption method, and fewer submissions?
Yes on my dummy central project (with only 1 form and 2 submissions), the same form with fewer submissions worked (my real server has few submissions but 1000s of forms).
Would you know how many different encryption keys your form uses?
I have been using project level encryption via central.
How long does submission_exportrun until you get the HTTP status 500?
This took seconds (but now working)
Did you see this info message (with verbose=TRUE): https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L167-L168
No I did not.

Again thank you for your time, and fingers crossed for a continued stable network.

Best, Spike

*

From: Florian Mayer @.> Sent: Wednesday, April 20, 2022 3:58 AM To: ropensci/ruODK @.> Cc: SpikeLewis @.>; Mention @.> Subject: Re: [ropensci/ruODK] issue accessing data decrypted at project level (Issue #134)

Thanks both for the troubleshooting. Looking at the code for submission_exporthttps://github.com/ropensci/ruODK/blob/main/R/submission_export.R, there are some limitations in ruODK:

Only the first encryption key is used
If the server throws an internal server error (HTTP status 500), ruODK terminates the export as not to exhaust server memory: https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L183-L188 - seeing you've got a different error caught in the next command (stop_for_status) tells me we're not getting a HTTP status 500, so the encryption key seems to have worked.

@SpikeLewishttps://github.com/SpikeLewis some more diagnostic questions:

Can you reproduce this error with the same form, same encryption method, and fewer submissions?
Would you know how many different encryption keys your form uses?
How long does submission_export run until you get the HTTP status 500?
Did you see this info message (with verbose=TRUE): https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L167-L168

—

Reply to this email directly, view it on GitHubhttps://github.com/ropensci/ruODK/issues/134#issuecomment-1103420037, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYTKCFHJAQ4AJBZEVOESCETVF5XE7ANCNFSM5SZKNHMQ.

You are receiving this because you were mentioned.Message ID: @.***>

florianm commented 2 years ago

Brilliant, thanks for the detailed answers! Seeing the same ruODK code worked on different forms and occasions I doubt there's a bug in there. However, we are always at the mercy of the internet pigeons flying straight and true. I'm closing this issue with the most likely cause being network timeouts. If the pigeons start missing home again feel free to re-open this issue or create a new one!

ropensci / ruODK

issue accessing data decrypted at project level #134

Problem

Reproducible example

Full install