search

IUSCA / sca-issues

1 stars 0 forks source link

SPH-RSC: How to link dynamic data with Shiny app #35

Closed StephanieD123 closed 4 years ago

StephanieD123 commented 4 years ago

I was able to upload our app for our Whistle project: https://rstudio.sca.iu.edu/connect/#/apps/17/access

Now where/how can i upload the data as it updates every day or so to be used by the Shiny app.

agopu commented 4 years ago

Hi Stephanie,

Once again, congrats on publishing the Whistle project app!

We have several choices re: data updates:

1) You can setup the data downloads on our server with our assistance so the updated data get downloaded locally.

2) You can setup an rsync of data, once a day, from your server to ours with our assistance.

3) We can work with you to use a shared disk: a) Mounting your NFS server on our Shiny server. b) We can let you mount a local disk share on the Shiny server via SMB. c) Geode2 group storage mount that is mounted on our server for use by the Shiny app; you/others will push data to the same location as just you on your end.

The initial discussion regarding data storage needs had started with a need to access (read/write) it from your computers, our Shiny server AND from RED. Following our Zoom discussion we inferred you don't quite need to access it from RED which opens up the options above.

StephanieD123 commented 4 years ago

Thanks Arvind.

I’m CC’ing Xiwei if she has thoughts on preference, and she may correspond with you about options/directions.

You’re right that it looks like we probably don’t need RED. That was Scott Michael’s recommendation to make sure the correct R packages are available for the app, but as long as we can get the app to work I don’t think we need the data to connect to RED for now.

Thanks, Stephanie

From: Arvind Gopu [mailto:notifications@github.com] Sent: Monday, March 16, 2020 2:26 PM To: IUSCA/sca-issues sca-issues@noreply.github.com Cc: Dickinson, Stephanie sd3@indiana.edu; Author author@noreply.github.com Subject: Re: [IUSCA/sca-issues] SPH-RSC: How to link dynamic data with Shiny app (#35)

Hi Stephanie,

Once again, congrats on publishing the Whistle project app!

We have several choices re: data updates:

  1. You can setup the data downloads on our server with our assistance so the updated data get downloaded locally.

  2. You can setup an rsync of data, once a day, from your server to ours with our assistance.

  3. We can work with you to use a shared disk: a) Mounting your NFS server on our Shiny server. b) We can let you mount a local disk share on the Shiny server via SMB. c) Geode2 group storage mount that is mounted on our server for use by the Shiny app; you/others will push data to the same location as just you on your end.

The initial discussion regarding data storage needs had started with a need to access (read/write) it from your computers, our Shiny server AND from RED. Following our Zoom discussion we inferred you don't quite need to access it from RED which opens up the options above.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IUSCA/sca-issues/issues/35#issuecomment-599693228 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AGE3HIB43M2WOXXEMKCCO4TRHZVNFANCNFSM4LMMUZKA . https://github.com/notifications/beacon/AGE3HIEFETH44GN5DBHUZZ3RHZVNFA5CNFSM4LMMUZKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEO7JPLA.gif

agopu commented 4 years ago

Hi Stephanie/Xiwei,

We know a lot is going on right now so no pressure but wanted to check if you have further thoughts on how you want to proceed.

-a

xinoooo commented 4 years ago

Hi Arvind,

Thanks for listing us the options, but I don't understand the options quite well.

Maybe it would help by further explaining the situation we are facing: This data is currently in a Dropbox folder and that folder would be updated manually by tech-people from the Whistle company daily. What we want to accomplish is that the Shiny app would always grab the latest version of data from the Dropbox folder(or from the server if we can copy/update the data from Dropbox to the server.)

From my understanding, we need to publish the Shiny app daily (since the data updates every day) to show the latest information. But I don't know how to automatically publish the Shiny app every day. Or do you have any better ideas on how to show the latest information on the Shiny app?

Thank you, Xiwei @StephanieD123 @agopu

StephanieD123 commented 4 years ago

@agopu https://github.com/agopu @xinoooo

Xiwei – I think that we shouldn’t need to re PUBLISH the app.

I think that either:

  1. If the app can pull in the data from somewhere, then we just need to send the clean data to that spot every day to be read in by the app.

  2. Or, maybe if the app (or script on their server) can pull the data directly from its original source Dropbox and Fitbit (but I’m hesitant that this will work easily in a shared space with private API keys.

A few more details:

  1. There are Fitbit data (from human activity) which we download using the Fitbit API with Python code and then clean/rearrange and save the clean data

  2. There are Whistle data (tracking dogs activity) which the Whistle company puts in Dropbox and our Python code cleans and rearranges that and saves clean data .

  3. Right now, we are running scripts every day on Xiwei’s Mac to do these two pieces (which I think Xiwei has automated)

  4. Then both the clean Fitbit data and Whistle data are stored on Box, but we can put it anywhere you tell us to.

The options you gave are:

  1. You can setup the data downloads on our server with our assistance so the updated data get downloaded locally.

  2. You can setup an rsync of data, once a day, from your server to ours with our assistance.

  3. We can work with you to use a shared disk: a) Mounting your NFS server on our Shiny server. b) We can let you mount a local disk share on the Shiny server via SMB. c) Geode2 group storage mount that is mounted on our server for use by the Shiny app; you/others will push data to the same location as just you on your end.

Can you say more about how your options 1 to 3c will work with our needs. And what our “workflow” will look like as far as what we need to log-into and where our scripts and data would be stored and run.

Thanks!

Stephanie

From: Xiwei Chen [mailto:notifications@github.com] Sent: Wednesday, March 25, 2020 10:49 PM To: IUSCA/sca-issues sca-issues@noreply.github.com Cc: Dickinson, Stephanie sd3@indiana.edu; Mention mention@noreply.github.com Subject: Re: [IUSCA/sca-issues] SPH-RSC: How to link dynamic data with Shiny app (#35)

Hi Arvind,

Thanks for listing us the options, but I don't understand the options quite well.

Maybe it would help by further explaining the situation we are facing: This data is currently in a Dropbox folder and that folder would be updated manually by tech-people from the Whistle company daily. What we want to accomplish is that the Shiny app would always grab the latest version of data from the Dropbox folder(or from the server if we can copy/update the data from Dropbox to the server.)

From my understanding, we need to publish the Shiny app daily (since the data updates every day) to show the latest information. But I don't know how to automatically publish the Shiny app every day. Or do you have any better ideas on how to show the latest information on the Shiny app?

Thank you, Xiwei @StephanieD123 https://github.com/StephanieD123 @agopu https://github.com/agopu

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IUSCA/sca-issues/issues/35#issuecomment-604197460 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AGE3HIGCZYNCAZGDJQQZDLTRJK7CPANCNFSM4LMMUZKA . https://github.com/notifications/beacon/AGE3HIEQFGAKTLX3K3HUAL3RJK7CPA5CNFSM4LMMUZKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEQBVEVA.gif

agopu commented 4 years ago

Hi Stephanie and Xiwei,

Correct, you should not need to republish the app to pick up new data; the app should able to pick up fresh data from a designated data location. We just need to identify what location is optimally for this project.

clean/rearrange and save the clean data

It is unclear how you clean/rearrange the Fitbit or Whistle data picked up by your automated scripts. Expanding on:

I get the impression option 2 would be the simplest so perhaps we should start with that and see if it suffices? This way you can keep your existing data retrieval processes, and no administrative/operational changes need to made on any disk servers.

Please let us know. If a quick zoom call would help, we are happy to do that as well. Hope you all are safe at this time!

Best regards, -a

StephanieD123 commented 4 years ago

Hi Arvind,

I agree that option 2 using rsync should be easiest to start. Please send directions and we’ll get started.

Thanks!

Stephanie

From: Arvind Gopu [mailto:notifications@github.com] Sent: Thursday, March 26, 2020 11:58 AM To: IUSCA/sca-issues sca-issues@noreply.github.com Cc: Dickinson, Stephanie sd3@indiana.edu; Assign assign@noreply.github.com Subject: Re: [IUSCA/sca-issues] SPH-RSC: How to link dynamic data with Shiny app (#35)

Hi Stephanie and Xiwei,

Correct, you should not need to republish the app to pick up new data; the app should able to pick up fresh data from a designated data location. We just need to identify what location is optimally for this project.

clean/rearrange and save the clean data

It is unclear how you clean/rearrange the Fitbit or Whistle data picked up by your automated scripts. Expanding on:

I get the impression option 2 would be the simplest so perhaps we should start with that and see if it suffices? This way you can keep your existing data retrieval processes, and no administrative/operational changes need to made on any disk servers.

Please let us know. If a quick zoom call would help, we are happy to do that as well. Hope you all are safe at this time!

Best regards, -a

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/IUSCA/sca-issues/issues/35#issuecomment-604513480 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AGE3HIGJV2VTHBAFXMPOF23RJN3RPANCNFSM4LMMUZKA . https://github.com/notifications/beacon/AGE3HIHYMI42ROWPA7GQPOLRJN3RPA5CNFSM4LMMUZKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEQECJSA.gif

agopu commented 4 years ago

We are working with Stephanie, et al. off line to get them setup with the rsync'ing - will report back here when we are done.

agopu commented 4 years ago

As noted previously, we worked out details offline. Xiwei has access to our server to rysnc in the data from their end.

rperigo commented 4 years ago

Hi all - Bringing everyone back into the loop now that updated data has been placed on the server.

The data on the RSC server lives in a sub-dir: sphdata within the SCA standard appdata location.

It is owned by the user account running the Shiny applications within RStudio Connect - we will have to verify this, but once the Shiny apps are updated to use this directory when executed on the server, it should have access to that data directory. If not, we can help debug. It will probably be necessary to add some logic to the Shiny applications such that they are able to tell if they are running on the production machine or not (perhaps a hostname check or some such would suffice).

agopu commented 4 years ago

@StephanieD123 @xinoooo -- With the rsync in place, could we confirm that your app has the dynamic data you requested on this ticket? i.e could we close out this ticket?

StephanieD123 commented 4 years ago

Hi – It doesn’t seem to be working.

https://rstudio.sca.iu.edu/connect/#/apps/19/access

Am I missing something? Does it work for you?

From: Arvind Gopu [mailto:notifications@github.com] Sent: Friday, April 10, 2020 11:42 AM To: IUSCA/sca-issues sca-issues@noreply.github.com Cc: Dickinson, Stephanie sd3@indiana.edu; Mention mention@noreply.github.com Subject: Re: [IUSCA/sca-issues] SPH-RSC: How to link dynamic data with Shiny app (#35)

@StephanieD123 https://github.com/StephanieD123 @xinoooo https://github.com/xinoooo -- With the rsync in place, could we confirm that your app has the dynamic data you requested on this ticket? i.e could we close out this ticket?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IUSCA/sca-issues/issues/35#issuecomment-612086016 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AGE3HIDOWYUO2YBXKYT5HFTRL4443ANCNFSM4LMMUZKA . https://github.com/notifications/beacon/AGE3HIAVGER5FZMBH3HPJELRL4443A5CNFSM4LMMUZKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOER53CAA.gif

agopu commented 4 years ago

Hi – It doesn’t seem to be working.

@StephanieD123 @xinoooo I was talking with @rperigo a while back and he mentioned there is character encoding issues in the data that is causing errors. Could Xiwei please open a new ticket to follow up on that? It's not a problem with the the data setup as much as it is a problem with the data itself. Thanks!

rperigo commented 4 years ago

To elucidate on that - I added some print statements to log the contents of the fitbit_var and study_id variables taken from the data files. This resulted in the following log additions:

2020/04/09 20:30:42.688645690 [1] "steps"                "calories"             "caloriesBMR"        
2020/04/09 20:30:42.688669697 [4] "distance"             "minutesSedentary"     "minutesLightlyActive"
2020/04/09 20:30:42.688771910 [7] "minutesFairlyActive"  "minutesVeryActive"    "activityCalories"   
2020/04/09 20:30:42.916269387 # A tibble: 11 x 10
2020/04/09 20:30:42.916285369    `Subject ID` `Accelerometer 
 `Randomization 
 Group `Fitbit email`
2020/04/09 20:30:42.916353422           <dbl> <lgl>                       <dbl> <chr> <lgl>        
2020/04/09 20:30:42.916364871  1            1 NA                              1 Fitb
 NA           
2020/04/09 20:30:42.916382868  2            2 NA                             NA <NA>  NA           
2020/04/09 20:30:42.916384426  3            3 NA                              2 Cont
 NA           
2020/04/09 20:30:42.916401518  4            4 NA                              3 Fitb
 NA           
2020/04/09 20:30:42.916403098  5            5 NA                              4 Fitb
 NA           
2020/04/09 20:30:42.916437535  6            6 NA                              5 Fitb
 NA           
2020/04/09 20:30:42.916439039  7            7 NA                              6 Fitb
 NA           
2020/04/09 20:30:42.916456071  8            8 NA                              7 F+W   NA           
2020/04/09 20:30:42.916457576  9            9 NA                              8 Fitb
 NA           
2020/04/09 20:30:42.916475081 10           10 NA                              9 F+W   NA           
2020/04/09 20:30:42.916476526 11           11 NA                             10 Whis
 NA           
2020/04/09 20:30:42.916495271 # 
 with 5 more variables: `Fitbit code` <chr>, `Whistle serial #` <chr>, `Dog
2020/04/09 20:30:42.916496822 #   ID` <dbl>, `Accelerometer Post` <lgl>, notes <chr>

It does appear the files are at least accessible to the application. However, the garbage characters above and the lines:

2020/04/09 19:56:13.884623871 Warning in force(expr) : input string 'é' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?
2020/04/09 19:56:13.884711497 Warning in force(expr) : input string 'é' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?
2020/04/09 19:56:13.885044413 Warning in force(expr) : input string 'é' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?

lead me to believe there may be an encoding issue. It could be something else, but it definitely appears to be in the way R is ingesting the data. The cause for the error message onscreen appears to be that the plot function cannot get proper values for X axis.

2020/04/10 16:14:29.889871963 Warning: Length of logical index must be 1 or 11, not 0
2020/04/10 16:14:29.896927937 Warning: Error in if: argument is of length zero
2020/04/10 16:14:29.907056554   180: PlotSubject
2020/04/10 16:14:29.907065427   179: renderPlot [/opt/rstudio-connect/mnt/app/app.R#102]
2020/04/10 16:14:29.907107400   177: func
2020/04/10 16:14:29.907109454   137: drawPlot
2020/04/10 16:14:29.907126694   123: <reactive:plotObj>
2020/04/10 16:14:29.907128292   107: drawReactive
2020/04/10 16:14:29.907144866    94: origRenderFunc
2020/04/10 16:14:29.907146463    93: output$PlotSubject
2020/04/10 16:14:29.907162763    13: runApp
2020/04/10 16:14:29.907164372    12: fn
2020/04/10 16:14:29.907189844     7: connect$retry
2020/04/10 16:14:29.907191996     6: eval
2020/04/10 16:14:29.907229365     5: eval
2020/04/10 16:14:30.000214436 Warning in min(x) : no non-missing arguments to min; returning Inf
2020/04/10 16:14:30.000632994 Warning in max(x) : no non-missing arguments to max; returning -Inf
2020/04/10 16:14:30.022418103 Warning: Error in plot.window: need finite 'xlim' values
StephanieD123 commented 4 years ago

Xiwei says she’s working on it. The data format changed from the Whistle people, so we need to adjust. So we can’t test the “dynamic data” part until we get that settled. Thanks for understanding – lots of moving parts. It’s also not dynamic yet until we set a script to run rsync every day. But I think we can do that.

From: rperigo [mailto:notifications@github.com] Sent: Friday, April 10, 2020 12:22 PM To: IUSCA/sca-issues sca-issues@noreply.github.com Cc: Dickinson, Stephanie sd3@indiana.edu; Mention mention@noreply.github.com Subject: Re: [IUSCA/sca-issues] SPH-RSC: How to link dynamic data with Shiny app (#35)

To elucidate on that - I added some print statements to log the contents of the fitbit_var and study_id variables taken from the data files. This resulted in the following log additions:

2020/04/09 20:30:42.688645690 [1] "steps" "calories" "caloriesBMR"

2020/04/09 20:30:42.688669697 [4] "distance" "minutesSedentary" "minutesLightlyActive"

2020/04/09 20:30:42.688771910 [7] "minutesFairlyActive" "minutesVeryActive" "activityCalories"

2020/04/09 20:30:42.916269387 # A tibble: 11 x 10

2020/04/09 20:30:42.916285369 Subject ID Accelerometer �Randomization � Group Fitbit email

2020/04/09 20:30:42.916353422

2020/04/09 20:30:42.916364871 1 1 NA 1 Fitb� NA

2020/04/09 20:30:42.916382868 2 2 NA NA NA

2020/04/09 20:30:42.916384426 3 3 NA 2 Cont� NA

2020/04/09 20:30:42.916401518 4 4 NA 3 Fitb� NA

2020/04/09 20:30:42.916403098 5 5 NA 4 Fitb� NA

2020/04/09 20:30:42.916437535 6 6 NA 5 Fitb� NA

2020/04/09 20:30:42.916439039 7 7 NA 6 Fitb� NA

2020/04/09 20:30:42.916456071 8 8 NA 7 F+W NA

2020/04/09 20:30:42.916457576 9 9 NA 8 Fitb� NA

2020/04/09 20:30:42.916475081 10 10 NA 9 F+W NA

2020/04/09 20:30:42.916476526 11 11 NA 10 Whis� NA

2020/04/09 20:30:42.916495271 # � with 5 more variables: Fitbit code , Whistle serial # , `Dog

2020/04/09 20:30:42.916496822 # ID<dbl>,Accelerometer Post` , notes

It does appear the files are at least accessible to the application. However, the garbage characters above and the lines:

2020/04/09 19:56:13.884623871 Warning in force(expr) : input string '�é' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?

2020/04/09 19:56:13.884711497 Warning in force(expr) : input string '�é' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?

2020/04/09 19:56:13.885044413 Warning in force(expr) : input string '�é' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?

lead me to believe there may be an encoding issue. It could be something else, but it definitely appears to be in the way R is ingesting the data. The cause for the error message onscreen appears to be that the plot function cannot get proper values for X axis.

2020/04/10 16:14:29.889871963 Warning: Length of logical index must be 1 or 11, not 0

2020/04/10 16:14:29.896927937 Warning: Error in if: argument is of length zero

2020/04/10 16:14:29.907056554 180: PlotSubject

2020/04/10 16:14:29.907065427 179: renderPlot [/opt/rstudio-connect/mnt/app/app.R#102]

2020/04/10 16:14:29.907107400 177: func

2020/04/10 16:14:29.907109454 137: drawPlot

2020/04/10 16:14:29.907126694 123:

2020/04/10 16:14:29.907128292 107: drawReactive

2020/04/10 16:14:29.907144866 94: origRenderFunc

2020/04/10 16:14:29.907146463 93: output$PlotSubject

2020/04/10 16:14:29.907162763 13: runApp

2020/04/10 16:14:29.907164372 12: fn

2020/04/10 16:14:29.907189844 7: connect$retry

2020/04/10 16:14:29.907191996 6: eval

2020/04/10 16:14:29.907229365 5: eval

2020/04/10 16:14:30.000214436 Warning in min(x) : no non-missing arguments to min; returning Inf

2020/04/10 16:14:30.000632994 Warning in max(x) : no non-missing arguments to max; returning -Inf

2020/04/10 16:14:30.022418103 Warning: Error in plot.window: need finite 'xlim' values

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IUSCA/sca-issues/issues/35#issuecomment-612105489 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AGE3HIGOB4W7VEOSBAYXY6TRL5BT5ANCNFSM4LMMUZKA . https://github.com/notifications/beacon/AGE3HIC73FBU2IHHCWCMQE3RL5BT5A5CNFSM4LMMUZKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOER572EI.gif

agopu commented 4 years ago

Sounds good, we will stay tuned!

agopu commented 4 years ago

Following up on the new ticket linked above: https://github.com/IUSCA/sca-issues/issues/38