Code in ctmm vs. shinny app, pack app into package

[jmcalabrese]

One thing I've been thinking about lately is that the shiny app is gaining functionality that the command line version of ctmm does not have. Some users may want to stick with the command line, particularly if they are automating analyses, and may want access to some of the shiny app features. Should we think about migrating some of the code that enables the additional functionality into ctmm somewhere down then road?

[xhdong-umd]

Here is a brief list on top of my mind, just to open the discussion.

data representation

• ctmm put multiple individuals into a list of objects, which have its advantages. I merged all individuals into a data frame with a column identity because ggplot require this structure. I'm not sure if the data frame format may also have some advantages in interactive command line analysis.

plots

• Users may have interests in some of the plots. Put each plot into a function is doable. Though these plots will lost the interactivity of zooming, sliders etc in command line.

  An ideal solution may be a function that take user's data input then create a Shiny page with the plots.

  This is not easy to implement. I plan to look at the code later to see if I can convert some common used pattern into modules, then it'll be easier to do things like this.

Speaking of this, if I can structure the app in more module way, maybe it's possible to create a function for every page. Because every page start from some data, if user have the data in command line, he can start a page for visualization/outlier detection/time subsetting with data, do something interactively, save the data, then return to command line.

[jmcalabrese]

@xhdong-umd: I don't think we need to do this yet, and I don't want you to do anything that slows down development of the shiny app, I just think it is worth thinking about for the longer term. Modularization may be the way to go.

I would like to hear @chfleming's thoughts on this.

[xhdong-umd]

Yes sure, this is just brain storming. I'm thinking on module a lot because the time subsetting, distance/speed outlier detection will share a lot of common features.

[chfleming]

I would think about modularization and/or having standalone ggplot functions going forward and keep it in mind when making coding choices, but I wouldn't let it slow down current development, which is more important.

The analysis modules would be "wizards", essentially? That might be a very good idea. Lots of people will be using the command line but still need guidance.

[xhdong-umd]

I'm thinking of each page as a suite of functions for one specific analysis stage. All features inside one page are related, so it make sense to launch them all together, like open all plots in one command.

The pages will offer interactive control and some analysis features, user can insert any analysis or data manipulation between pages.

[xhdong-umd]

I'm closing the issue temporarily to focus on active tasks. We can reopen it when time is appropriate. And you can still comment on it anytime even the issue is closed.

[xhdong-umd]

@jmcalabrese @chfleming Previously I said the ideal way is to make each page into a module, later I realized that will need too many changes in app structure and not practical.

I considered the different options of making the interactive map function available to ctmm as follows:

• Wrap the map features into functions and save in single .R file, include it in ctmm. However it will be cumbersome if I need to make change to it. Besides, even the map feature is quite independent from other parts in web app, it still need the merge function to convert multiple telemetry object into single data frame.
• It seemed that the better option is to make webapp into a package, then ctmm can use various functions from it, like the merge function, ggplot function (I also made a function that can make interactive zoomable ggplot in app before), and the cross platform parallel function, zip a folder in all platforms(zip in base R rely on external libraries, zip package don't have function to zip a folder with relative path easily) etc.
  • To use the webapp package will need to install some dependency packages. This is the only negative I can think of.
  • I can also try to submit the package to CRAN at the right time. It should make the package installation easier.

[xhdong-umd]

We can also make a function to start app in command line, take telemetry objects, csv as input. Or we can import the saved session data for further analysis. Making the app as a package should be the most flexible way.

[jmcalabrese]

@xhdong-umd I like the "webapp as a package" idea. Let's discuss the details at our next webapp meeting.

[xhdong-umd]

@jmcalabrese @chfleming I've started to experiment on the package idea. For now I'll create a new repo under my account and test it, then consider to update content in this repo at some time.

What name should we pick for the package? I think we should not use - in package name, then which one do you prefer, or do you have other suggestion?

• ctmmweb
• ctmmwebapp
• ctmmapp

I think ctmmapp is good since we can launch the app in multiple context, not just as a web page. I planned to use function app() to launch the app after package is installed, and app(buffalo) to load the app with some data.

The repo name better match the package name, so we may need to update this repo name later.

[xhdong-umd]

Should I use same GPL-3 license for the package?

[xhdong-umd]

This is my plan of package organization:

• The app code are under /inst/app and will be copied after package installation.
• The package only include functions useful to users, which can be used in command line, like some plot, map function or utility functions.
• The package only require libraries used in these functions. So it's possible to install the package but never use the app and do not need to install app dependencies.
• One exception is there is a app() function can be used to start the web app after package is installed.
• The app have its own dependencies, will be installed at first run, like current setting.

In summary, the package have some functions can be used in R scripts, and the web app is part of package installation and user can start the app easier.

It should be also possible to call the app function with a parameter, so you can load a csv file, move object, telemetry object/list with app. This will make it easier if you are working in R command line and want to inspect your data with the web app.

[xhdong-umd]

I tried to call app with a R data object in memory, then assemble the Shiny app dynamically, but too many things broke in that way since it was too dynamic and far from the Shiny design intentions.

One way to work around it is to create a global variable in global workspace, and the app will read that variable. However I don't like this idea.

I think I need to drop this idea since it require too many substantial changes in app structure.

[xhdong-umd]

I realized to separate core exported functions dependencies and app dependencies is difficult. The exported functions will need to access plot, map, internal data structure, which may actually need most of app dependencies. Trying to separate them is a difficult task and need constant changes.

I assume most users installing the app package will want to run the app at least some time, not just the exported functions, so I'll not try to separate them anymore.

• the package will have same dependency requirement with the app. It should also make the app running/updating easier.
• most utilities functions will be inside package, but only part of them are exported.

[xhdong-umd]

I managed to launch app with data parameter after carefully maintained the app state in different scenarios. By checking data parameter in specific environment I don't have to mess up global environment.

Now you can launch app like this:

library(ctmmapp)
# launch app normally
app()
# launch app with path to movebank csv
app("path to csv")
# launch app with telemetry object or list
app(buffalo)
app(buffalo[[1]])

[xhdong-umd]

My next step is to move utility functions into package. We will need to discuss which functions need to be exported, and what kind of interfaces are desired. Before that I still need to change every function call into full qualified name. I think I can make a tool to make this process easier.

[xhdong-umd]

I'm in the process of qualifying functions which is a cumbersome and time taking task. Because I have to change the repo structure and using a different git setup, now every changes I committed reflect on github repo immediately, so the current status of app may have some bugs while I'm working on it. Previously I can commit the changes to bitbucket, only update the github repo when needed.

Later I may setup the git repo to sync with bitbucket first, then update github less frequently so this will not be a problem.

[xhdong-umd]

I have fixed the bug, which was caused by there is no dAICc column in home range summary table. I also setup bitbucket as the default git remote so not every intermediate changes will be pushed to github.

[xhdong-umd]

I finally completed all the reorganizing work (hopefully).

• To migrate part of an existing app into a package is like pull parts of a running machine into another box, and wrap all the connections between box and app so it can still work.
• I don't want to expose everything so some connections will need different syntax in referencing.
• I also need to test every feature of app to make sure all code paths are covered.
• I try to make sure the app will not modify global environment as much as possible.

There are also a lot of documentations need to be completed. I'll post a interface list later asking for comments.

[xhdong-umd]

I just finished document all exported functions, test on all features, with linux/windows/hosted server, and pushed changes to github, hosted server. You can have a test with the package or hosted app.

Next I will create a package website and we can use it as a starting point to discuss the exported functions and their interfaces. I'll also write some vignettes.

[jmcalabrese]

Great work! I will try to go through this over the next week.

[xhdong-umd]

Since I found some problem in a test shiny host server, and it's difficult to get proper error log (the server log is messed up in order), I spent some time to add an option to check error log in app. This is only needed in hosted mode. User can turn on error log, and click a button to show error messages when error occurred.

There is also an option to turn off parallel mode, which can be used to test if problem is caused by parallel (which usually are difficult to reproduce and fix).

[xhdong-umd]

The package website is live now. I still need to fine tune the organization a little bit.

I also rearranged the new options into a new box to make the organization better, added the help for them.

[xhdong-umd]

I made some more refinements to the package and website. The exported functions are organized and documented. I'll start a new issue to discuss them.