search

pachadotdev / capybara

tldr; If you have a 2-4GB dataset and you need to estimate a (generalized) linear model with a large number of fixed effects, this package is for you.
http://pacha.dev/capybara/
Apache License 2.0
13 stars 0 forks source link

note for myself: deviance values #3

Closed pachadotdev closed 2 months ago

pachadotdev commented 2 months ago

https://github.com/pachadotdev/capybara/blob/reducebackandforth/ uses the same underlying functions as https://github.com/pachadotdev/capybara/blob/main

the differences are

  1. instead of doing R -> C++ -> R -> C++ -> R ... -> R it does R -> C++ -> R
  2. I had to implement my own function for the deviance and link

the tests in https://github.com/pachadotdev/capybara/blob/reducebackandforth/tests/testthat/test-fepoisson.R#L2 https://github.com/pachadotdev/capybara/blob/reducebackandforth/tests/testthat/test-fepoisson.R#L7 https://github.com/pachadotdev/capybara/blob/reducebackandforth/tests/testthat/test-fepoisson.R#L12 give the same results as main

the problem is here

https://github.com/pachadotdev/capybara/blob/reducebackandforth/src/05_glm_fit.cpp#L248

output:

iter: 0 dev: 1.15908e+08 iter: 1 dev: 1.12655e+20

adding a print statement in https://github.com/pachadotdev/capybara/blob/main/R/internals.R#L66 returns this for the same model

[1] "iter: 1" [1] "dev: 119812197.425158" [1] "iter: 2" [1] "dev: 61981346.125582" [1] "iter: 12" [1] "dev: 4265228.57183159"

I need to check the deviance https://github.com/pachadotdev/capybara/blob/reducebackandforth/src/05_glm_fit.cpp#L77

curious thing: felm(), which is just a wrapper for feglm() with a Gaussian link, works ok so the problem must be the transformed values

pachadotdev commented 2 months ago

fixed in 7fe44d093f560b3cb060dea1b272aa2b67c43c36