Pkg.test("DynamicHMC") on Julia 1.5.0 gives two fails

[MooersLab]

Running on a MacOS 10.15. I was able to work the worked example in the documentation, so I am unsure if these test failures are important.

Test Summary:     | Pass  Total
simulated meancov |    2      2
Test Summary: |    Pass    Total
test-trees.jl | 2003798  2003798
leapfrog back and forth: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:171
  Expression: ≈(z.p, z1.p, norm = (x->begin
                norm(x, Inf)
            end), rtol = 0.001)
   Evaluated: [-0.7522839018993613, 0.2728057174736127] ≈ [-0.7508500824628771, 0.271725160229739] (norm=#20, rtol=0.001)
Stacktrace:
 [1] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:171
 [2] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
 [3] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:144
leapfrog back and forth: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:171
  Expression: ≈(z.p, z1.p, norm = (x->begin
                norm(x, Inf)
            end), rtol = 0.001)
   Evaluated: [-0.02532229751461036, 0.030292707711202047] ≈ [-0.027146248766058756, 0.03115685279645941] (norm=#20, rtol=0.001)
Stacktrace:
 [1] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:171
 [2] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
 [3] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:144
Test Summary:                                    |  Pass  Fail  Total
test-hamiltonian.jl                              | 13260     2  13262
  Gaussian KE full                               |   400          400
  Gaussian KE diagonal                           |   400          400
  phasepoint internal consistency                |    23           23
  leapfrog calculation                           |   205          205
  leapfrog Hamiltonian invariance                | 10000        10000
  leapfrog back and forth                        |  2198     2   2200
  PhasePoint building blocks and infinite values |     9            9
  Hamiltonian and KE printing                    |     3            3
  unit normal simple HMC                         |     2            2
  normal NUTS HMC transition mean and cov        |    20           20
ERROR: LoadError: Some tests did not pass: 13260 passed, 2 failed, 0 errored, 0 broken.
in expression starting at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/runtests.jl:17
ERROR: Package DynamicHMC errored during testing

[tpapp]

Thanks for reporting. This probably comes from a change in Julia's RNG producing different random values for the tests, which happen to be a bit more ill-conditioned than previously. I will investigate and fix, but this should be innocuous.

[MooersLab]

Thank you for the clarification. I suspected that this might be the case. I will carry on exploring DynamicHMC amusing all is well.

On Thu, Aug 6, 2020 at 6:54 AM Tamas K. Papp notifications@github.com wrote:

Thanks for reporting. This probably comes from a change in Julia's RNG producing different random values for the tests, which happen to be a bit more ill-conditioned than previously. I will investigate and fix, but this should be innocuous.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tpapp/DynamicHMC.jl/issues/132#issuecomment-669882505, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTZEC4LA34LR75KKLO3ZYDR7KKYVANCNFSM4PWALY3Q .

-- Best regards,

Blaine

Blaine Mooers, Ph.D. Associate Professor Department of Biochemistry and Molecular Biology College of Medicine University of Oklahoma Health Sciences Center S.L. Young Biomedical Research Center Rm. 466 975 NE 10th Street https://maps.google.com/?q=975+NE+10th+Street&entry=gmail&source=g, BRC 466 Oklahoma City, OK 73104-5419

office: (405) 271-8300 lab: (405) 271-8313

Faculty webpage http://basicsciences.ouhsc.edu/biochemmolbiol/Faculty/bio_details/TabId/11753/ArtMID/30702/ArticleID/6430/Mooers-Blaine-HM-PhD.aspx X-ray lab (LBSF) http://research.ouhsc.edu/CoreFacilities/LaboratoryofBiomolecularStructureandFunction.aspx SSRL UEC https://www-ssrl.slac.stanford.edu/content/about-ssrl/advisory-panels/ssrl-users-organization/members/ssrluo-2016-executive-committee-members SSURF EasyPyMOL https://github.com/MooersLab/EasyPyMOL Molecular Graphics https://www.oumedicine.com/docs/default-source/ad-biochemistry-workfiles/moleculargraphicslinks.html

https://www.oumedicine.com/docs/default-source/ad-biochemistry-workfiles/MolecularGraphicsLinks.html Small Angle Scattering http://www.oumedicine.com/docs/default-source/ad-biochemistry-workfiles/small-angle-scattering-links-27aug2014.html?sfvrsn=0 office: (405) 271-8300 lab: (405) 271-8313 e-mail: blaine-mooers@ouhsc.edu (or bmooers1@gmail.com)

[tpapp]

Fixed on latest master.

[MooersLab]

Testing of todays version on github in Julia 15.1 on Mac OS X 15.6 progresses further, but fails on the testing of sample-correctness-utitlies.jl.

Test Summary: | Pass Total simulated meancov | 2 2 Test Summary: | Pass Total test-trees.jl | 2003798 2003798 Test Summary: | Pass Total test-hamiltonian.jl | 13262 13262 Test Summary: | Pass Total test-NUTS.jl | 32 32 Test Summary: | Pass Total test-stepsize.jl | 1132 1132 Test Summary: | Pass Total test-mcmc.jl | 13 13 Test Summary: | Pass Total test-diagnostics.jl | 20 20 INFO while testing: mixture of two normals, dimension 3 ALERT some τ = [0.169, 0.184, 0.147] < 0.15 INFO while testing: elongate(1.2, 𝑁), dimension 5 ALERT extreme Anderson-Darling p ≈ 1.66e-14 < 0.0 for coordinate 1 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 1.6572882297252903e-14 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75 ALERT extreme Anderson-Darling p ≈ 7.98e-14 < 0.0 for coordinate 2 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 7.975948103200937e-14 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75 ALERT extreme Anderson-Darling p ≈ 2.39e-14 < 0.0 for coordinate 3 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 2.386491241178068e-14 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75 ALERT extreme Anderson-Darling p ≈ 1.56e-16 < 0.0 for coordinate 4 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 1.555009605115904e-16 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75 ALERT extreme Anderson-Darling p ≈ 5.94e-13 < 0.0 for coordinate 5 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 5.938619232986046e-13 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75 Test Summary: | Pass Fail Total sample-correctness-tests.jl | 153 5 158 NUTS tests with random normal | 96 96 NUTS tests with specific normal distributions | 40 40 NUTS tests with mixtures | 6 6 NUTS tests with heavier tails and skewness | 11 5 16 ERROR: LoadError: Some tests did not pass: 153 passed, 5 failed, 0 errored, 0 broken. in expression starting at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/runtests.jl:27 ERROR: Package DynamicHMC errored during testing

[tpapp]

Thanks. Is this the master branch of DynamicHMC?

[tpapp]

I am happy to address this but I can't replicate on Linux and Julia 1.5 or 1.6 --- I don't have access to OS X.

PRs are appreciated, just extend the bounds that fail by a minor amount.

[tpapp]

Is this still an open issue?

[MooersLab]

I installed julia 1.5.3 on Ubuntu and DyamicHMC and ran test on it. All was well.

On the Mac 10.15.7 with julia 1.5.3 and DymaicHMC from the default distribution installed, test returns

┌ Info: MCMC progress │ step = 9801 │ seconds_per_step = 5.0e-6 └ estimated_seconds_left = 0.00099 ┌ Info: MCMC progress │ step = 9901 │ seconds_per_step = 5.4e-6 └ estimated_seconds_left = 0.00054 [ Info: ┌ Info: Starting MCMC └ total_steps = 1000 [ Info: ┌ Info: MCMC progress │ step = 1 │ seconds_per_step = 0.018 └ estimated_seconds_left = 18.0 ┌ Info: Starting MCMC └ total_steps = 1000 [ Info: [ Info: ┌ Info: MCMC progress │ step = 1 │ seconds_per_step = 4.0e-5 └ estimated_seconds_left = 0.04 Test Summary: | test-logging.jl | No tests ┌ Warning: (t::AbstractTransform)(x) is deprecated, use transform(t, x) instead │ caller = rand_C(::Int64) at common.jl:118 └ @ Main ~/.julia/packages/DynamicHMC/n1LsS/test/common.jl:118 INFO while testing: mixture of two normals, dimension 3 ALERT some τ = [0.169, 0.184, 0.147] < 0.15 INFO while testing: elongate(1.2, 𝑁), dimension 5 ALERT extreme Anderson-Darling p ≈ 1.66e-14 < 0.0 for coordinate 1 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 1.6572882297252903e-14 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75 ALERT extreme Anderson-Darling p ≈ 7.98e-14 < 0.0 for coordinate 2 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 7.975948103200937e-14 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75 ALERT extreme Anderson-Darling p ≈ 2.39e-14 < 0.0 for coordinate 3 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 2.386491241178068e-14 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75 ALERT extreme Anderson-Darling p ≈ 1.56e-16 < 0.0 for coordinate 4 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 1.555009605115904e-16 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75 ALERT extreme Anderson-Darling p ≈ 5.94e-13 < 0.0 for coordinate 5 of 5 NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 Expression: p_AD ≥ pd_fail Evaluated: 5.938619232986046e-13 ≥ 2.0000000000000004e-7 Stacktrace: [1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115 [2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78 [3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75 Test Summary: | Pass Fail Total sample-correctness-tests.jl | 153 5 158 NUTS tests with random normal | 96 96 NUTS tests with specific normal distributions | 40 40 NUTS tests with mixtures | 6 6 NUTS tests with heavier tails and skewness | 11 5 16 ERROR: LoadError: Some tests did not pass: 153 passed, 5 failed, 0 errored, 0 broken. in expression starting at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/runtests.jl:28 ERROR: Package DynamicHMC errored during testing

[tpapp]

Thank you for running these. These are extreme p-values and indicate something suspicious is going on, but I cannot debug further without access to OS X.

Can you please try with Julia 1.6 on OS X and DynamicHMC 3.0.0?

[MooersLab]

I did as requested. All tests passed. There is a warning. I will wait for a response and then I plan to close this issue.

Test Summary: | test-logging.jl | No tests ┌ Warning: (t::AbstractTransform)(x) is deprecated, use transform(t, x) instead │ caller = rand_C(K::Int64) at common.jl:118 └ @ Main ~/.julia/packages/DynamicHMC/J48he/test/common.jl:118 INFO while testing: mixture of two normals, dimension 3 ALERT some τ = [0.164, 0.158, 0.139] < 0.15 Test Summary: | Pass Total sample-correctness-tests.jl | 170 170 Testing DynamicHMC tests passed

[tpapp]

The warnings are OK, some are expected (otherwise the errors would be hair-trigger).

Thanks again for investigating this.

[MooersLab]

I am closing this issue.