Need to fit smooth spline based only on degree of freedom, and predict derivatives

[babaq]

Hi,

I am moving some of the R Code to Julia, and this seems to be the only pure Julia package to do smooth spline, the problem is that the original R code use degree of freedoms to do smooth.spline, and predict not only y but also y'.

I am wandering if any of the authors or maintainers still working on this packages, or have plans to update the package and add these features?

[nignatiadis]

Hi @babaq!

I agree, these would be important features to add! I have wanted to implement these for some time, but it's always hard to find the time, especially since I do not actually need this functionality right now...

Let me address both points though:

• Implementing the derivative would be straightforward: in fact, right now the fitted object contains fields g and γ that store respectively the function values and the second derivatives at the design points. Since the fitted function is a natural spline, it is straightforward (perhaps tedious) to figure out how to compute the first derivative on design points as well at other points based on g and γ . I would be happy to provide more feedback/guidance should you or anyone else decide to implement it (before I do). Another point here is the following: what should the interface look like for computing the 1st or 2nd derivatives at a point x? Let me know if you have any suggestion.

• Degrees of freedom: Fitting the smoothing spline based on n data points runs with O(n) computational complexity. A naive implementation of the degrees of freedom could be done easily, but then it would be a O(n^3) implementation. There is a O(n) algorithm that one could implement based on the Silverman book (readme of this repo) that probably could be implemented with tests etc. in at most a couple hours.

[babaq]

Hi @nignatiadis

Thanks for the reply, I thought no one cares this package anymore.

I've looked into the R smooth.spline code and the rbart Fortran code, and the other Fortran library GCVSPL, I could wrap the GCVSPL, but test shows the dof is not the same as in the R, it seems took infinite time to finish when set dof to a very high value, and the performance is worse than R even with the RCall overhead.

So, working on this package seems easier to me, I've started reading the Book you mentioned, and would get to you when i get the basic understanding of the approach.

BTW, the package needs to update to the new Project.toml, to work on current Julia(1.5), if you don't have much time on this, i would like to do a update first, so the ]add and ]dev would work for current Julia.

The other issue is to adjust the code to use BandedMatrices.jl package, I didn't look into yet, but i assume it should be straightforward.

[nignatiadis]

Hi @babaq,

I have not worked on this package in a long time; Helge and Mauro have kept it alive! But I am very happy for any contributions to it and would appreciate if you updated the package to use Project.toml. Also let me know if you have any questions that come up while reading the Green, Silverman book.

Regarding R's smooth.spline, if I recall correctly, it uses a different method compared to this package. It is based on B-Splines; which are actually more numerically stable. My guess is it might be not too hard to implement the B-Spline based method e.g., by using CompactBases.jl (but it's precompile time is very slow right now, so that I'd leave this to the future). As an aside, some feedback I have seen from the Julia community is that one should avoid looking at GPL code (e.g. the R code for smooth.spline) before implementing features for MIT-licensed packages (otherwise it may be considered a violation of the GPL license).

With the Reinsch approach, as you mention, removing the explicit LAPACK calls and using BandedMatrices.jl instead should be straightforward.