Summary of weekly calls

[anujkhare]

Monday, 5th June 2017

Updates:

• Armadillo + OpenBlas linking issues
• Steps 1-4 in proposal locally done, need to clean and commit

(Some) Next steps:

• For any Armadillo issues, contact the maintainers (Dirk is super responsive)
• Put config for the src in iregnet/src/makevars
• Set up Continuous Integration performance testing using Rperform
• Move the code into a branch on anujkhare/iregnet
• (Anuj will fix Travis tests on anujkhare/iregnet/master)
• Read about git-flow workflow
  • We don't completely follow it, but it is a good read!
• Power on!!

@tdhock @rovervan

[tdhock]

• create a GitHub PR for every new branch. for discussion, even if the branch is not yet ready to merge.
• use commit comments so we can discuss the code.
• Makevars file resolved compilation issue.
• problems with RPerform, please post an issue on https://github.com/analyticalmonk/Rperform/issues with all the versions of the packages that you used (Rperform, iregnet, etc). devtools::session_info() reports versions of packages.
• using new RcppArmadillo data structures such as arma::mat instead of Rcpp::NumericMatrix
• use CXX_STD = CXX11 instead of -std=c++11 flag to avoid compilation errors with g++ (travis)

[anujkhare]

• Rperform problem is in our code: Rover will fix and produce the benchmark!
• Rperform adds some lines to the Rcpp::export?
• Will review and merge once the benchmark is done.

[tdhock]

Rperform issues.

• our master branch is not compatible with Rperform -- need to update our master branch.
• will post an issue on their repo.
• why only 4 commits in Rperform plot?
• sometimes we have a commit which will not work (needs other commits to work).
• eventually want to run all commits in the optimization branch.
• need to post issue on Rperform -- link to one of our branches which causes Rperform to stop with an error -- maybe do this on a fork? (RoverVan/iregnet@optimization) this will help the Rperform developers fix the problem in the long term.
• in the short term, we can use git rebase to get some commits that are working with the current version of Rperform. (anujkhare/iregnet@optimization)
• local master branch has new Rcpp auto-generated code, can push to anujkhare/iregnet@master on github. just make sure the changes are minor, so we can compare the speed improvements with the optimization branch commits.

Optimization by grouping data by censoring type. Currently working for normal distribution only.

[tdhock]

• issues posted to Rperform, they said they will look into it. They haven't fixed the problems yet. I'll send them an email reminder -- the more important problem is the recursive function call error, because that is still a problem after we use git rebase. is there some small commit we can make on master to work around this problem?
• pushed in optimization branch. gaussian function all censoring types coded. logistic coded but not commit/push yet. travis tests passing.
• pushed commit to master branch, working now. very small commit of an automatically generated Rcpp file, so should not affect performance tests when comparing commits in optimization branch.
• next week working more on logistic distribution.
• blog post with microbenchmark comparison? rename packages, maybe iregnetOld and iregnetNew, and then run them both using microbenchmark in the same Rmd, try one or two data sets using the Gaussian distribution, comment on speed improvements.
• next skype at 11am Montreal time on Monday July 31.

[tdhock]

• Rperform student not coding a lot, hasn't had time to look into the issue we posted. Mentor Akash is checking.
• manual benchmark using microbenchmark http://rovervan.com/post/gsoc/iregnet-benchmark results: always faster or the same speed as the old code, and numerically equal up to 1e-3. Try timing glmnet for test one? Maybe make a plot of seconds vs number of rows/columns, geom_line for median and geom_ribbon for 25% and 75% quartiles? One line/ribbon for each of: glmnet, iregnet, oldiregnet.
• Did you try a smaller tolerance than 1e-3? Did you try to compute the cost? = -log(likelihood of data | parameters).
• Next: logistic distribution.
• Rest of GSOC: complete benchmark with other distributions.

[tdhock]

• logistic distribution almost finished (missing interval censoring type)
• benchmark updated http://rovervan.com/post/gsoc/iregnet-benchmark 8-10, new iregnet a little faster
• plot timings of three algos as a function of rows/cols? not yet.
• extreme value distribution? would be great if we have time.
• important to write a summary text/blog describing overall results in GSOC, with benchmark figures.

[tdhock]

• new benchmark plots, 5 times, different number of rows, new iregnet faster
• textual summary, a short paragraph that explains what you did, for each figure.
• same plot for the number of columns?
• add glmnet to the plot of gaussian with no censoring.
• same coefs up to 1e-7.
• what else do we need to do before merging the PR? ok right now. Send a message to @anujkhare asking him if it is OK to merge with master.
• repo with test/benchmark code https://github.com/RoverVan/GSoC-iregnet-Test-Code
• maybe to speed up the benchmark use log scale, 10^seq(2, 4, l=10) and also log-log axes for the plots scale_x_log10, scale_y_log10.
• please use microbenchmark(oldiregnet=, newiregnet=, glmnet=) instead.
• it would be faster to accumulate data frames like this https://github.com/tdhock/idioms#accumulating-dataframes
• check out data.table for computing the min/max/mean --- data.table(time.result)[, list(min=min(mean)), by=expr]

[tdhock]

@RoverVan

• comments/text: good.
• why do for(dist in c(gaussian, logistic)) if you just set dist<-logistic after? why does it take too long?
• use as.integer(10^seq(2, 5, l=10)) or something like that for the grid of row/col sizes.
• glmnet comparison figure: clear that glmnet is still much faster, but asymptotics are not clear. for presentation, use scale_y_log10 and scale_x_log10.
• sometimes the timing curves go down and then up again. but they should always go up for bigger data sets. this is because you are using get_xy to simulate a new data set for each size. Solution is to use get_xy at the beginning to simulate one data set with the maximum size, and then use subset operations to consider only the subset which is important.
• comparison on real data set, e.g. penaltyLearning::neuroblastomaProcessed or iregnet::penalty.learning
• spelling error: martix -> matrix.
• at the end, add a paragraph that discusses what the figures mean. Speed improvements!
• resolve conflicts, ask @anujkhare to approve pr, then merge with master.
• any ideas about what the differences with glmnet?