search

airoldilab / sgd

An R package for large scale estimation with stochastic gradient descent
62 stars 18 forks source link

Models using boost::math::tools::schroeder_iterate can get stuck #108

Open jonlachmann opened 3 years ago

jonlachmann commented 3 years ago

Hi, I am building a package that is using sgd. When doing tests I sometimes encountered models that got stuck in the sgd C++ part for no particular reason, crashing R in the end. I did address sanitizing, but everything checked out. Then I investigated where it got stuck, and it was while running the boost::math::tools::schroeder_iterate function for the update in implicit_sgd.h (rows 79 and 110). I made a fork at https://github.com/jonlachmann/sgd fixing this by just setting the max iterations for boost::math::tools::schroeder_iterate to 1000. I have no idea if this is a reasonable value, but it solves the problem.

Do you want me to create a pull request, or is this something that might need further looking into?

AlbertRapp commented 2 years ago

Had the same problem. Installed your fork and now R doesn't crash anymore while using sgd. Thanks.

ptoulis commented 2 years ago

Hey @jonlachmann Thanks for creating the fork. Appreciate it.

Sounds like a weird numerical issue. Do you have a data/code example to reproduce it by any chance?

jonlachmann commented 2 years ago

Hi! This was during the more intense part of writing my thesis. I do not really remember how I managed to get this to happen, and I ended up coding a more basic SGD from scratch to demonstrate the ideas in the thesis.

I was using different stochastic algorithms in a bayesian model selection setting, so I ran many thousand models, which some of the time gave strange errors that were edge cases of various implementations. Perhaps @AlbertRapp has something that is reproducible?

Anyway, I could try to see if I have any code that relates to this, but I am not too optimistic as the thesis period was very hectic. If you want to apply my fix in your codebase I would be happy to provide a PR.

ptoulis commented 2 years ago

Sure, that would be great! Thanks.