search

IQVIA-ML / LightGBM.jl

Julia FFI interface to Microsoft's LightGBM package
Other
93 stars 10 forks source link

ERROR: Package LightGBM errored during testing #117

Closed Frank-III closed 2 years ago

Frank-III commented 2 years ago 
(@v1.7) pkg> test LightGBM
     Testing LightGBM
      Status `C:\Users\frank\AppData\Local\Temp\jl_KYP6sI\Project.toml`
  [324d7699] CategoricalArrays v0.10.5
  [7acf609c] LightGBM v0.5.2
  [a7f614a8] MLJBase v0.18.26
  [e80e1ace] MLJModelInterface v1.3.3
  [2913bbd2] StatsBase v0.33.16
  [ade2ca70] Dates `@stdlib/Dates`
  [8bb1440f] DelimitedFiles `@stdlib/DelimitedFiles`
  [8f399da3] Libdl `@stdlib/Libdl`
  [9a3f8284] Random `@stdlib/Random`
  [8dfed614] Test `@stdlib/Test`
      Status `C:\Users\frank\AppData\Local\Temp\jl_KYP6sI\Manifest.toml`
  [324d7699] CategoricalArrays v0.10.5
  [d360d2e6] ChainRulesCore v1.14.0
  [9e997f8a] ChangesOfVariables v0.1.2
  [3da002f7] ColorTypes v0.11.0
  [34da2185] Compat v3.43.0
  [ed09eef8] ComputationalResources v0.3.2
  [a8cc5b0e] Crayons v4.1.1
  [9a962f9c] DataAPI v1.9.0
  [864edb3b] DataStructures v0.18.11
  [e2d170a0] DataValueInterfaces v1.0.0
  [b429d917] DensityInterface v0.4.0
  [31c24e10] Distributions v0.25.53
  [ffbed154] DocStringExtensions v0.8.6
  [1a297f60] FillArrays v0.12.8
  [53c48c17] FixedPointNumbers v0.8.4
  [59287772] Formatting v0.4.2
  [3587e190] InverseFunctions v0.1.3
  [41ab1584] InvertedIndices v1.1.0
  [92d709cd] IrrationalConstants v0.1.1
  [82899510] IteratorInterfaceExtensions v1.0.0
  [692b3bcd] JLLWrappers v1.4.1
  [7f8f8fb0] LearnBase v0.4.1
  [7acf609c] LightGBM v0.5.2
  [2ab3a3ac] LogExpFunctions v0.3.12
  [30fc2ffe] LossFunctions v0.7.2
  [a7f614a8] MLJBase v0.18.26
  [e80e1ace] MLJModelInterface v1.3.3
  [e1d29d7a] Missings v1.0.2
  [bac558e1] OrderedCollections v1.4.1
  [90014a1f] PDMats v0.11.7
  [d96e819e] Parameters v0.12.3
  [21216c6a] Preferences v1.2.5
  [08abe8d2] PrettyTables v1.3.1
  [92933f4c] ProgressMeter v1.7.2
  [1fd47b50] QuadGK v2.4.2
  [3cdcf5f2] RecipesBase v1.2.1
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [79098fc4] Rmath v0.7.0
  [321657f4] ScientificTypes v2.3.3
  [30f210dd] ScientificTypesBase v2.3.0
  [a2af1166] SortingAlgorithms v1.0.1
  [276daf66] SpecialFunctions v1.8.4
  [64bff920] StatisticalTraits v2.1.0
  [82ae8749] StatsAPI v1.2.2
  [2913bbd2] StatsBase v0.33.16
  [4c63d2b9] StatsFuns v0.9.18
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.7.0
  [3a884ed6] UnPack v1.0.2
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [0dad84c5] ArgTools `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8bb1440f] DelimitedFiles `@stdlib/DelimitedFiles`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads `@stdlib/Downloads`
  [9fa8497b] Future `@stdlib/Future`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [b27032c2] LibCURL `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions `@stdlib/NetworkOptions`
  [44cfe95a] Pkg `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [1a1011a3] SharedArrays `@stdlib/SharedArrays`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [4607b0f0] SuiteSparse `@stdlib/SuiteSparse`
  [fa267f1f] TOML `@stdlib/TOML`
  [a4e569a6] Tar `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll `@stdlib/OpenLibm_jll`
  [83775a58] Zlib_jll `@stdlib/Zlib_jll`
  [8e850b90] libblastrampoline_jll `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll `@stdlib/p7zip_jll`
     Testing Running tests...
[ Info: lib_lightgbm found in system dirs!
LGBM_BoosterSaveModelToString: Test Failed at C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\ffi\booster.jl:337
  Expression: occursin("version=v3", string_repr)
   Evaluated: occursin("version=v3", "tree\nversion=v4\nnum_class=1\nnum_tree_per_iteration=1\nlabel_index=0\nmax_feature_idx=4\nobjective=binary sigmoid:1\nfeature_names=Column_0 Column_1 Column_2 Column_3 Column_4\nfeature_infos=none none none none none\ntree_sizes=240\n\nTree=0\nnum_leaves=1\nnum_cat=0\nsplit_feature=\nsplit_gain=\nthreshold=\ndecision_type=\nleft_child=\nright_child=\nleaf_value=-0.84729786038720356\nleaf_weight=\nleaf_count=\ninternal_value=\ninternal_weight=\ninternal_count=\nis_linear=0\nshrinkage=1\n\n\nend of trees\n\nfeature_importances:\n\nparameters:\n[boosting: gbdt]\n[objective: binary]\n[metric: binary_logloss]\n[tree_learner: serial]\n[device_type: cpu]\n[data: ]\n[valid: ]\n[num_iterations: 100]\n[learning_rate: 0.1]\n[num_leaves: 3]\n[num_threads: 2]\n[max_depth: 5]\n[min_data_in_leaf: 1]\n[min_sum_hessian_in_leaf: 1e-06]\n[bagging_fraction: 1]\n[pos_bagging_fraction: 1]\n[neg_bagging_fraction: 1]\n[bagging_freq: 0]\n[bagging_seed: 3]\n[feature_fraction: 1]\n[feature_fraction_bynode: 1]\n[feature_fraction_seed: 2]\n[early_stopping_round: 0]\n[first_metric_only: 0]\n[max_delta_step: 0]\n[lambda_l1: 0]\n[lambda_l2: 0]\n[min_gain_to_split: 0]\n[drop_rate: 0.1]\n[max_drop: 50]\n[skip_drop: 0.5]\n[xgboost_dart_mode: 0]\n[uniform_drop: 0]\n[drop_seed: 4]\n[top_rate: 0.2]\n[other_rate: 0.1]\n[min_data_per_group: 100]\n[max_cat_threshold: 32]\n[cat_l2: 10]\n[cat_smooth: 10]\n[max_cat_to_onehot: 4]\n[top_k: 20]\n[monotone_constraints: ]\n[feature_contri: ]\n[forcedsplits_filename: ]\n[forcedbins_filename: ]\n[refit_decay_rate: 0.9]\n[cegb_tradeoff: 1]\n[cegb_penalty_split: 0]\n[cegb_penalty_feature_lazy: ]\n[cegb_penalty_feature_coupled: ]\n[verbosity: -1]\n[max_bin: 255]\n[max_bin_by_feature: ]\n[min_data_in_bin: 3]\n[bin_construct_sample_cnt: 200000]\n[histogram_pool_size: -1]\n[data_random_seed: 1]\n[output_model: LightGBM_model.txt]\n[snapshot_freq: -1]\n[input_model: ]\n[output_result: LightGBM_predict_result.txt]\n[initscore_filename: ]\n[valid_data_initscores: ]\n[pre_partition: 0]\n[enable_bundle: 1]\n[max_conflict_rate: 0]\n[is_enable_sparse: 1]\n[sparse_threshold: 0.8]\n[use_missing: 1]\n[zero_as_missing: 0]\n[two_round: 0]\n[save_binary: 0]\n[header: 0]\n[label_column: ]\n[weight_column: ]\n[group_column: ]\n[ignore_column: ]\n[categorical_feature: ]\n[predict_raw_score: 0]\n[predict_leaf_index: 0]\n[predict_contrib: 0]\n[num_iteration_predict: -1]\n[pred_early_stop: 0]\n[pred_early_stop_freq: 10]\n[pred_early_stop_margin: 10]\n[convert_model_language: ]\n[convert_model: gbdt_prediction.cpp]\n[num_class: 1]\n[is_unbalance: 0]\n[scale_pos_weight: 1]\n[sigmoid: 1]\n[boost_from_average: 1]\n[reg_sqrt: 0]\n[alpha: 0.9]\n[fair_c: 1]\n[poisson_max_delta_step: 0.7]\n[tweedie_variance_power: 1.5]\n[max_position: 20]\n[lambdamart_norm: 1]\n[label_gain: ]\n[metric_freq: 1]\n[is_provide_training_metric: 0]\n[eval_at: ]\n[multi_error_top_k: 1]\n[num_machines: 1]\n[local_listen_port: 12400]\n[time_out: 120]\n[machine_list_filename: ]\n[machines: ]\n[gpu_platform_id: -1]\n[gpu_device_id: -1]\n[gpu_use_dp: 0]\n\nend of parameters\n")
Stacktrace:
 [1] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:445 [inlined]
 [2] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\ffi\booster.jl:337 [inlined]
 [3] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [4] top-level scope
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\ffi\booster.jl:333
Test Summary:                       | Pass  Fail  Broken  Total
FFI unit tests                      |  186     1       8    195
  Dataset                           |   37             2     39
  Booster                           |  144     1       6    151
    LGBM_BoosterCreate              |    1                    1
    LGBM_BoosterCreateFromModelfile |    1                    1
    LGBM_BoosterLoadModelFromString |    1                    1
    LGBM_BoosterFree                |    2                    2
    LGBM_BoosterMerge               |    7                    7
    LGBM_BoosterAddValidData        |    3                    3
    LGBM_BoosterResetTrainingData   |    7                    7
    LGBM_BoosterGetNumClasses       |   19                   19
    LGBM_BoosterUpdateOneIter       |    3                    3
    LGBM_BoosterRollbackOneIter     |   60                   60
    LGBM_BoosterGetCurrentIteration |   21                   21
    LGBM_BoosterGetEvalCounts       |                  1      1
    LGBM_BoosterGetEvalNames        |    1                    1
    LGBM_BoosterGetFeatureNames     |    1                    1
    LGBM_BoosterGetNumFeature       |    1                    1
    LGBM_BoosterGetEval             |                  1      1
    LGBM_BoosterGetNumPredict       |                  1      1
    LGBM_BoosterGetPredict          |                  1      1
    LGBM_BoosterCalcNumPredict      |                  1      1
    LGBM_BoosterPredictForMat       |                  1      1
    LGBM_BoosterSaveModel           |    1                    1
    LGBM_BoosterSaveModelToString   |   11     1             12
    LGBM_BoosterFeatureImportance   |    4                    4
  Base Utils                        |    5                    5
ERROR: Package LightGBM errored during testing
yaxxie commented 2 years ago

Hi @Frank-III -- thanks for the report!

It looks like you're using an own LightGBM Binary -- could you state which API version?

When I use the one distributed with this package it doesn't fail at FFI tests, although I can see some other failures.

I will investigate if this is limited to 1.7 or not

yaxxie commented 2 years ago

Hi @Frank-III !

I recently fixed tests on master for 1.7 (which were broken but for other reasons)

Taking a more careful look at the failing testcase presented I can see that you're probably using a master version of the C-library. This has changed the version string from v3 to v4 (although I don't know why)

I'll fix this particular test as well, so that you can run them all through, but I should warn that we can't necessarily support builds from master for the C-API, particularly if there are API changes (which would probably manifest as segfaults during tests). The easiest way to deal with this is to build your binary from a tag (the current 3.3.2 does work)

I'll submit a PR now for the particular part of the test that is failing, and after that if you don't object within a few days, I'll close this issue.

Frank-III commented 2 years ago

Hi @Frank-III !

I recently fixed tests on master for 1.7 (which were broken but for other reasons)

Taking a more careful look at the failing testcase presented I can see that you're probably using a master version of the C-library. This has changed the version string from v3 to v4 (although I don't know why)

I'll fix this particular test as well, so that you can run them all through, but I should warn that we can't necessarily support builds from master for the C-API, particularly if there are API changes (which would probably manifest as segfaults during tests). The easiest way to deal with this is to build your binary from a tag (the current 3.3.2 does work)

I'll submit a PR now for the particular part of the test that is failing, and after that if you don't object within a few days, I'll close this issue.

Thanks! Sorry for not noticing your reply, I would try to install it!

yaxxie commented 2 years ago

I just merged #121 so the master version of this package should not fail the tests. Let me know if thats good for you

Frank-III commented 2 years ago

I just update LightGBM package, and I get fft test to pass but other error occurs

[ Info: lib_lightgbm not found in system dirs, trying fallback
Test Summary:  | Pass  Broken  Total
FFI unit tests |  187       8    195
Binary LightGBM: Test Failed at C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\mlj\binary_classifier.jl:37
  Expression: misclassification_rate < 0.015
   Evaluated: 0.022277227722772276 < 0.015
Stacktrace:
 [1] top-level scope
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:445
 [2] include(fname::String)
   @ Base.MainInclude .\client.jl:451
 [3] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:28 [inlined]
 [4] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [5] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:28 [inlined]
 [6] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [7] top-level scope
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:23
Multiclass LightGBM: Test Failed at C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\mlj\multiclass_classifier.jl:37
  Expression: misclassification_rate < 0.015
   Evaluated: 0.016666666666666666 < 0.015
Stacktrace:
 [1] top-level scope
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:445
 [2] include(fname::String)
   @ Base.MainInclude .\client.jl:451
 [3] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:32 [inlined]
 [4] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [5] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:32 [inlined]
 [6] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [7] top-level scope
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:23
Multiclass LightGBM: Test Failed at C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\mlj\multiclass_classifier.jl:40
  Expression: yhat_with_weights != yhat
   Evaluated: CategoricalArrays.CategoricalValue{String, UInt32}["4", "4", "4", "0", "2", "0", "2", "2", "4", "4"  …  "2", "1", "3", "4", "3", "0", "0", "3", "3", "4"] != CategoricalArrays.CategoricalValue{String, UInt32}["4", "4", "4", "0", "2", "0", "2", "2", "4", "4"  …  "2", "1", "3", "4", "3", "0", "0", "3", "3", "4"]
Stacktrace:
 [1] top-level scope
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:445
 [2] include(fname::String)
   @ Base.MainInclude .\client.jl:451
 [3] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:32 [inlined]
 [4] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [5] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:32 [inlined]
 [6] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [7] top-level scope
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:23
Regression LightGBM: Test Failed at C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\mlj\regression.jl:36
  Expression: !(isapprox(rmse, rmse_weights, rtol = 0.1))
   Evaluated: !(isapprox(0.03184919031849717, 0.03290716103046044; rtol = 0.1))
Stacktrace:
 [1] top-level scope
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:445
 [2] include(fname::String)
   @ Base.MainInclude .\client.jl:451
 [3] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:36 [inlined]
 [4] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [5] macro expansion
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:36 [inlined]
 [6] macro expansion
   @ C:\Users\frank\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\Test\src\Test.jl:1283 [inlined]
 [7] top-level scope
   @ C:\Users\frank\.julia\packages\LightGBM\A7zVd\test\runtests.jl:23
Test Summary:          | Pass  Fail  Total
MLJ interface tests    |   71     4     75
  Parameters           |   18           18
  Binary LightGBM      |    6     1      7
  Multiclass LightGBM  |    6     2      8
  Regression LightGBM  |    6     1      7
  MLJ update interface |   21           21
  MLJ update interface |   14           14
ERROR: Package LightGBM errored during testing
yaxxie commented 2 years ago

A couple of things are slightly weird here:

Test Summary:  | Pass  Broken  Total
FFI unit tests |  217       8    225

This is the FFI tests from the CI but yours is reporting less tests

2nd, the MLJ tests can occasionally fail by chance (random seeds etc). I'd suggest you try again but I have a feeling they would fail again, and the reason I think so is that usually only a single MLJ test fails but here almost all of the ones prone to failing are failing together. Could you try running it again and let me know what happens?

Also actually can I ask you to check you're running it from master of IQVIA-ML/LightGBM.jl ? Could you show the output of ] st

The reason why I ask is that on master one of the tests got revised to this:

@test rmse < 0.1
@test !isapprox(rmse, rmse_weights, rtol=0.01)

But in the trace you showed I saw this: 

  Expression: !(isapprox(rmse, rmse_weights, rtol = 0.1))
   Evaluated: !(isapprox(0.03184919031849717, 0.03290716103046044; rtol = 0.1))

Which is from some older version of the package.

Please just confirm you have something like this:

(@v1.7) pkg> add LightGBM#master
     Cloning git-repo `https://github.com/IQVIA-ML/LightGBM.jl.git`
    Updating git-repo `https://github.com/IQVIA-ML/LightGBM.jl.git`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/.julia/environments/v1.7/Project.toml`
  [7acf609c] ~ LightGBM v0.5.2 ⇒ v0.5.2 `https://github.com/IQVIA-ML/LightGBM.jl.git#master`
    Updating `~/.julia/environments/v1.7/Manifest.toml`
  [7acf609c] ~ LightGBM v0.5.2 ⇒ v0.5.2 `https://github.com/IQVIA-ML/LightGBM.jl.git#master`
    Building LightGBM → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/700f17332b7faca44f607e981fb7c4fe4465a643/build.log`
Precompiling project...
  1 dependency successfully precompiled in 2 seconds (83 already precompiled)

(@v1.7) pkg> st
      Status `~/.julia/environments/v1.7/Project.toml`
  [324d7699] CategoricalArrays v0.10.5
  [7acf609c] LightGBM v0.5.2 `https://github.com/IQVIA-ML/LightGBM.jl.git#master`
  [a7f614a8] MLJBase v0.20.3

(@v1.7) pkg>

and then try to re-run the tests?

Frank-III commented 2 years ago

All tests passed, thanks for your detailed comments. I would close the issue.