I made a few more changes to the confidence bands. There seemed to be two issues:
The tolerance for finding changes in alpha was too small in the cpp code (was set to the tolerance parameter, default to 1e-4).
For the Canadian data, the sequence is so long, the lambdas so big, that it wasn’t really “converging” all the time. For the paper, you should run it for more iterations and with smaller tolerance.
I also simplified the computation of the bands. I like the old way better, but it can lead to some issues for some segments if the knots are close together. And the bands get too wide near the knots. There’s still an issue that for large counts (as in the Canadian data), the bands get too narrow. It’s because of 1/yhat^2. If that drops below some threshold, then (1/yhat^2 + lambda * D’D) can have very small eigenvalues. But I can’t figure out the correct threshold. So we should just leave it for now.
I made a few more changes to the confidence bands. There seemed to be two issues:
I also simplified the computation of the bands. I like the old way better, but it can lead to some issues for some segments if the knots are close together. And the bands get too wide near the knots. There’s still an issue that for large counts (as in the Canadian data), the bands get too narrow. It’s because of
1/yhat^2. If that drops below some threshold, then(1/yhat^2 + lambda * D’D)can have very small eigenvalues. But I can’t figure out the correct threshold. So we should just leave it for now.