(GLR) Avoid copying data with/without intercept

[tlienart]

At the moment there's a rather ugly

X_ = glr.fit_intercept ? hcat(ones(n), X) : X

so if fit_intercept then potentially the hcat creates a new object in memory which is quite inefficient.

@vollmersj suggested that given Julia is column-major, it shouldn't be an issue so I'm guessing there's an easy work-around. Though this may have side-effects on the initial X and, potentially, this should be an option for the user.

What is done in (say) sklearn or mlr etc?

[vollmersj]

It used to be the case that for inplace operations you needed to do add_inplace! = Base.LinAlg.axpy! but this was in 0.5 according to [(https://stackoverflow.com/questions/32610062/blas-axpy-slower-than-in-julia] )this is no longer needed. You simply write .= and .+ .

I checked hover hcat still allocates unreasonable amount of money.

@time M=randn(1000,1000)
@time M=hcat(M,randn(1000,1))

Output

  0.147933 seconds (6 allocations: 7.630 MiB, 87.36% gc time)
  0.032622 seconds (7 allocations: 7.645 MiB)

Bottom line: hcat! is still an open issue [https://github.com/JuliaLang/julia/issues/10546]

It is also interesting so see that [https://github.com/JuliaArrays/StaticArrays.jl] might provide a speed up but should leave this to Flux.jl @MikeInnes

[tlienart]

Hey Seb, please don't close issues too quickly unless something has been implemented or documented.

I think issue https://github.com/JuliaLang/julia/issues/10546 is quite interesting and it looks unlikely that's going to change from what Jeff B says.

However this issue for us is relevant when you're dealing with JuliaDB. In that case it's not a mere question of doing a "hcat" (because you may end up copying something monstrously big in ram). Also some algorithms don't need an explicit representation of hcat(ones(n, 1), X) to work if they don't solve a linear system but rather apply a matmul repeatedly (eg Krylov).

I think here we should

• check what is done in sklearn
• leave the default \ solver use a hcat, mentioning it in the doc
• think about algorithms that do not require a hcat and make sure those are used when dealing with a "very large" feature matrix