search

JuliaStats / Statistics.jl

The Statistics stdlib that ships with Julia.
https://juliastats.org/Statistics.jl/dev/
Other
71 stars 40 forks source link

Make SparseArrays a weak dependency #134

Closed IanButterworth closed 1 year ago

IanButterworth commented 1 year ago

Uses the recently merged https://github.com/JuliaLang/julia/pull/47695 to set up SparseArrays as a weak dependency of Statistics, via a package extension.

All code was copied across to the package extension untouched, with the addition of some imports.

This is backwards compatible. On older julia versions that don't support package extensions, SparseArrays will be loaded as per normal.

What this would look like on julia master, if SparseArrays is removed from the sysimage

julia> @time using Statistics
  0.011658 seconds (13.76 k allocations: 1.030 MiB)

julia> @time using SparseArrays
  4.618848 seconds (7.05 M allocations: 980.989 MiB, 1.46% compilation time)

cc. @KristofferC

codecov-commenter commented 1 year ago

Codecov Report

Base: 96.94% // Head: 97.26% // Increases project coverage by +0.31% :tada:

Coverage data is based on head (8ab238a) compared to base (e9ac70b). Patch has no changes to coverable lines.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #134 +/- ## ========================================== + Coverage 96.94% 97.26% +0.31% ========================================== Files 1 1 Lines 426 365 -61 ========================================== - Hits 413 355 -58 + Misses 13 10 -3 ``` | [Impacted Files](https://codecov.io/gh/JuliaStats/Statistics.jl/pull/134?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats) | Coverage Δ | | |---|---|---| | [src/Statistics.jl](https://codecov.io/gh/JuliaStats/Statistics.jl/pull/134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats#diff-c3JjL1N0YXRpc3RpY3Muamw=) | `97.26% <ø> (+0.31%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaStats)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

IanButterworth commented 1 year ago

@ViralBShah good to go ahead with this, and remove SparseArrays from the sysimage once bumped on julia master? (I don't have merge rights here)

KristofferC commented 1 year ago

good to go ahead with this

Didn't we find out that extensions didn't work right now for stdlibs? To me, it only makes sense to merge this once Statistics.jl is not an stdlib?

vtjnash commented 1 year ago

It seemed to work for me, so I think we should try this

KristofferC commented 1 year ago

Okay, let's go with this then. Thanks for testing!

nalimilan commented 11 months ago

Unfortunately this fails if I bump Statistics to the latest commit on Julia master. And indeed even after using SparseArrays, the cov method defined in the extension isn't used for some reason.

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.473 (2023-09-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 42d98a2ed14* (0 days old master)
|__/                   |

julia> Base.runtests(["Statistics"])
Running parallel tests with:
  nworkers() = 1
  nthreads() = 1
  Sys.CPU_THREADS = 4
  Sys.total_memory() = 15.282 GiB
  Sys.free_memory() = 7.305 GiB

Test   (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
Statistics  (1) |        started at 2023-09-15T19:57:33.267
Statistics  (1) |         failed at 2023-09-15T19:58:48.347
Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:963
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:964
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:963
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:964
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:963
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:964
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:963
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:964
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:978
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 0, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 0 … 0 0; … ; 0 0 … 1 1; 0 0 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:979
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 0, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 0 … 0 0; … ; 0 0 … 1 1; 0 0 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:978
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 0, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 0 … 0 0; … ; 0 0 … 1 1; 0 0 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:979
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 0, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 0 … 0 0; … ; 0 0 … 1 1; 0 0 … 1 1]

Test Summary: | Pass  Fail  Total     Time
  Overall     |  821    12    833  1m16.5s
    Statistics |  821    12    833  1m15.7s
    FAILURE

The global RNG seed was 0x7eb6b98a471ff52f1a5a23d7acd773bb.
KristofferC commented 8 months ago

@nalimilan, I think this is fixed now with https://github.com/JuliaLang/julia/pull/52428