search

JuliaAI / MLFlowClient.jl

Julia client for MLFlow.
https://juliaai.github.io/MLFlowClient.jl/
MIT License
42 stars 8 forks source link

main doesn't pass it's own test #42

Closed freemin7 closed 2 months ago

freemin7 commented 3 months ago 
(MLFlowClient) pkg> status
Project MLFlowClient v0.4.6
Status `~/.julia/dev/MLFlowClient/Project.toml`
  [48062228] FilePathsBase v0.9.21
  [cd3eb016] HTTP v1.10.5
  [682c06a0] JSON v0.21.4
  [605ecd9f] ShowCases v0.1.0
  [5c2747f8] URIs v1.5.1
  [ade2ca70] Dates
  [cf7118a7] UUIDs

(MLFlowClient) pkg> test
     Testing MLFlowClient
      Status `/tmp/jl_1EBWsE/Project.toml`
  [48062228] FilePathsBase v0.9.21
  [cd3eb016] HTTP v1.10.5
  [682c06a0] JSON v0.21.4
  [64a0f543] MLFlowClient v0.4.6 `~/.julia/dev/MLFlowClient`
  [605ecd9f] ShowCases v0.1.0
  [5c2747f8] URIs v1.5.1
  [ade2ca70] Dates
  [8dfed614] Test
  [cf7118a7] UUIDs
      Status `/tmp/jl_1EBWsE/Manifest.toml`
  [d1d4a3ce] BitFlags v0.1.8
  [944b1d66] CodecZlib v0.7.4
  [34da2185] Compat v4.14.0
  [f0e56b4a] ConcurrentUtilities v2.4.1
  [460bff9d] ExceptionUnwrapping v0.1.10
  [48062228] FilePathsBase v0.9.21
  [cd3eb016] HTTP v1.10.5
  [692b3bcd] JLLWrappers v1.5.0
  [682c06a0] JSON v0.21.4
  [e6f89c97] LoggingExtras v1.0.3
  [64a0f543] MLFlowClient v0.4.6 `~/.julia/dev/MLFlowClient`
  [739be429] MbedTLS v1.1.9
  [4d8831e6] OpenSSL v1.4.2
  [69de0a69] Parsers v2.8.1
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [605ecd9f] ShowCases v0.1.0
  [777ac1f9] SimpleBufferStream v1.1.0
  [3bb67fe8] TranscodingStreams v0.10.7
  [5c2747f8] URIs v1.5.1
  [458c3c95] OpenSSL_jll v3.0.13+1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [b77e0a4c] InteractiveUtils
  [8f399da3] Libdl
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [de0858da] Printf
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [fa267f1f] TOML v1.0.3
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [c8ffd9c3] MbedTLS_jll v2.28.2+0
  [14a3606d] MozillaCACerts_jll v2022.10.11
  [83775a58] Zlib_jll v1.2.13+0
     Testing Running tests...
Test Summary:    | Pass  Total  Time
createexperiment |    2      2  0.6s
Test Summary:                      | Pass  Total  Time
getexperiment                      |    5      5  0.1s
  getexperiment_by_experiment_id   |    2      2  0.0s
  getexperiment_by_experiment_name |    2      2  0.1s
  getexperiment_not_found          |    1      1  0.0s
Test Summary:         | Pass  Total  Time
getorcreateexperiment |    4      4  0.1s
deleteexperiment: Test Failed at /home/anabrid/.julia/dev/MLFlowClient/test/test_experiments.jl:53
  Expression: length(experiments) == 1
   Evaluated: 2 == 1

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Test/src/Test.jl:478 [inlined]
 [2] macro expansion
   @ ~/.julia/dev/MLFlowClient/test/test_experiments.jl:53 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
 [4] top-level scope
   @ ~/.julia/dev/MLFlowClient/test/test_experiments.jl:48
Test Summary:    | Fail  Total  Time
deleteexperiment |    1      1  0.7s
ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /home/anabrid/.julia/dev/MLFlowClient/test/test_experiments.jl:47
in expression starting at /home/anabrid/.julia/dev/MLFlowClient/test/runtests.jl:3
ERROR: Package MLFlowClient errored during testing

shell> git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

shell> git show HEAD
commit a31b41a91dc32107fa7dcd6ad5a581c116a4ff9c (HEAD -> main, origin/main, origin/gitlab_client, origin/HEAD)
Author: Jose Esparza <28990958+pebeto@users.noreply.github.com>
Date:   Tue Mar 5 11:23:19 2024 -0500

    Bump version

diff --git a/Project.toml b/Project.toml
index 6c4ebe8..202535a 100644
--- a/Project.toml
+++ b/Project.toml
@@ -1,7 +1,7 @@
 name = "MLFlowClient"
 uuid = "64a0f543-368b-4a9a-827a-e71edb2a0b83"
 authors = ["@deyandyankov, @pebeto, and contributors"]
-version = "0.4.5"
+version = "0.4.6"

 [deps]
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"

shell> git log
commit a31b41a91dc32107fa7dcd6ad5a581c116a4ff9c (HEAD -> main, origin/main, origin/gitlab_client, origin/HEAD)
Author: Jose Esparza <28990958+pebeto@users.noreply.github.com>
Date:   Tue Mar 5 11:23:19 2024 -0500

    Bump version

commit 776695b6bb4d92d732b162ae92de72b8ebbf4aba
Merge: de98ac2 60a2535
Author: Jose Esparza <28990958+pebeto@users.noreply.github.com>
Date:   Mon Mar 4 18:14:18 2024 -0500

    Merge pull request #37 from pebeto/improving_code_base

    Improving code base

commit 60a25356276ab61aed12eba7af45b944adc85d44
Author: Jose Esparza <28990958+pebeto@users.noreply.github.com>
Date:   Mon Mar 4 18:11:11 2024 -0500

    Ignoring `deprecated.jl` in coverage

commit dfed176c092e0194d6e220434cfd06a7d346b41c
Author: Jose Esparza <28990958+pebeto@users.noreply.github.com>
Date:   Sun Jan 7 23:29:12 2024 -0500

    Increasing coverage

commit 3190751aef412958151d509d8866c8877636ba38
Author: Jose Esparza <28990958+pebeto@users.noreply.github.com>
Date:   Sun Jan 7 23:13:06 2024 -0500

    Fixing Documenter failing on pipeline

commit 8886aa835e9fb176e1c774e36e69593c983c440d
Author: Jose Esparza <28990958+pebeto@users.noreply.github.com>
Date:   Sun Jan 7 22:22:54 2024 -0500

    Increasing maintainability by nice function separation in files

commit de98ac2fddbd49e00a3724d05184d1462f8a291c
Author: deyandyankov <deyan@dyankov.name>
Date:   Sun Jan 7 21:58:13 2024 +0000

    @JuliaRegistrator register

commit 26674e04bf1313942d78f682924e6683ac6106c7
Author: deyandyankov <deyan@dyankov.name>
Date:   Sun Jan 7 21:54:10 2024 +0000

    bump version

commit e27ac1026627327d58fd4b63aeeef62c738b0977
Author: Jose Esparza <28990958+pebeto@users.noreply.github.com>
Date:   Sun Jan 7 16:53:07 2024 -0500

    Migrating from /api to /ajax-api endpoint (#36)

commit 5a4237fbd28a611c40d37b6cc705a3c0b2de2c50
Author: deyandyankov <deyan@dyankov.name>
Date:   Thu Aug 10 09:54:40 2023 +0100
freemin7 commented 3 months ago

I tried to bisect the issue but output of test changed non deterministically as i walked back the tree and it now works for a31b41a91dc32107fa7dcd6ad5a581c116a4ff9c which is cursed. I have no idea why this would happen. 


(@v1.9) pkg> activate .
  Activating project at `~/.julia/dev/MLFlowClient`

(MLFlowClient) pkg> test
     Testing MLFlowClient
      Status `/tmp/jl_q5UEKV/Project.toml`
  [48062228] FilePathsBase v0.9.21
  [cd3eb016] HTTP v1.10.5
  [682c06a0] JSON v0.21.4
  [64a0f543] MLFlowClient v0.4.6 `~/.julia/dev/MLFlowClient`
  [605ecd9f] ShowCases v0.1.0
  [5c2747f8] URIs v1.5.1
  [ade2ca70] Dates
  [8dfed614] Test
  [cf7118a7] UUIDs
      Status `/tmp/jl_q5UEKV/Manifest.toml`
  [d1d4a3ce] BitFlags v0.1.8
  [944b1d66] CodecZlib v0.7.4
  [34da2185] Compat v4.14.0
  [f0e56b4a] ConcurrentUtilities v2.4.1
  [460bff9d] ExceptionUnwrapping v0.1.10
  [48062228] FilePathsBase v0.9.21
  [cd3eb016] HTTP v1.10.5
  [692b3bcd] JLLWrappers v1.5.0
  [682c06a0] JSON v0.21.4
  [e6f89c97] LoggingExtras v1.0.3
  [64a0f543] MLFlowClient v0.4.6 `~/.julia/dev/MLFlowClient`
  [739be429] MbedTLS v1.1.9
  [4d8831e6] OpenSSL v1.4.2
  [69de0a69] Parsers v2.8.1
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [605ecd9f] ShowCases v0.1.0
  [777ac1f9] SimpleBufferStream v1.1.0
  [3bb67fe8] TranscodingStreams v0.10.7
  [5c2747f8] URIs v1.5.1
  [458c3c95] OpenSSL_jll v3.0.13+1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [b77e0a4c] InteractiveUtils
  [8f399da3] Libdl
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [de0858da] Printf
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [fa267f1f] TOML v1.0.3
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [c8ffd9c3] MbedTLS_jll v2.28.2+0
  [14a3606d] MozillaCACerts_jll v2022.10.11
  [83775a58] Zlib_jll v1.2.13+0
Precompiling project...
  1 dependency successfully precompiled in 1 seconds. 24 already precompiled.
     Testing Running tests...
Test Summary:    | Pass  Total  Time
createexperiment |    2      2  0.9s
Test Summary:                      | Pass  Total  Time
getexperiment                      |    5      5  0.1s
  getexperiment_by_experiment_id   |    2      2  0.0s
  getexperiment_by_experiment_name |    2      2  0.1s
  getexperiment_not_found          |    1      1  0.0s
Test Summary:         | Pass  Total  Time
getorcreateexperiment |    4      4  0.1s
Test Summary:    | Pass  Total  Time
deleteexperiment |    1      1  0.4s
Test Summary:     | Pass  Total  Time
restoreexperiment |    2      2  0.0s
Test Summary:                            | Pass  Total  Time
searchexperiments                        |    6      6  0.1s
  searchexperiments_get_all              |    1      1  0.0s
  searchexperiments_by_filter            |    2      2  0.0s
  searchexperiments_by_filter_attributes |    2      2  0.0s
  searchexperiments_filter_exception     |    1      1  0.0s
Test Summary:                  | Pass  Total  Time
createrun                      |    4      4  0.6s
  createrun_by_experiment_id   |    2      2  0.5s
  createrun_by_experiment_type |    2      2  0.0s
Test Summary: | Pass  Total  Time
getrun        |    2      2  0.1s
Test Summary:                              | Pass  Total  Time
updaterun                                  |   25     25  0.5s
  updaterun_by_run_id                      |    5      5  0.4s
  updaterun_by_run_info                    |    5      5  0.0s
  updaterun_byrun                          |    5      5  0.0s
  updaterun_by_run_info_and_defined_status |    5      5  0.0s
  updaterun_by_run_and_defined_status      |    5      5  0.0s
Test Summary:           | Pass  Total  Time
deleterun               |    2      2  0.2s
  deleterun_by_run_info |    1      1  0.2s
  deleterun_by_run      |    1      1  0.0s
Test Summary:                     | Pass  Total  Time
searchruns                        |   10     10  1.5s
  searchruns_by_experiment_id     |    1      1  0.3s
  searchruns_by_experiment        |    1      1  0.1s
  searchruns_by_experiments_array |    1      1  0.2s
  searchruns_by_filter            |    1      1  0.0s
  searchruns_by_filter_params     |    1      1  0.1s
  searchruns_filter_exception     |    1      1  0.0s
  runs_get_methods                |    4      4  0.1s
Test Summary:                          | Pass  Total  Time
logparam                               |    8      8  0.1s
  logparam_by_run_id_and_key_value     |    2      2  0.0s
  logparam_by_run_info_and_key_value   |    2      2  0.0s
  logparam_by_run_and_key_value        |    2      2  0.0s
  logparam_by_union_and_dict_key_value |    2      2  0.0s
Test Summary:                           | Pass  Total  Time
logmetric                               |   12     12  0.2s
  logmetric_by_run_id_and_key_value     |    3      3  0.1s
  logmetric_by_run_info_and_key_value   |    3      3  0.0s
  logmetric_by_run_and_key_value        |    3      3  0.0s
  logmetric_by_union_and_key_arrayvalue |    3      3  0.0s
Test Summary:                            | Pass  Broken  Total  Time
logartifact                              |    5       1      6  0.3s
  logartifact_by_run_and_filenameanddata |    1              1  0.0s
  logartifact_by_run_id_and_file         |    1              1  0.1s
  logartifact_by_run_and_file            |    1              1  0.0s
  logartifact_by_run_info_and_file       |    1              1  0.0s
  logartifact_using_IOBuffer             |    1              1  0.0s
  logartifact_error                      |            1      1  0.0s
Test Summary:       | Pass  Total  Time
logbatch            |    4      4  0.6s
  logbatch_by_types |    2      2  0.4s
  logbatch_by_dicts |    2      2  0.2s
Test Summary:               | Pass  Total  Time
listartifacts               |    3      3  0.2s
  listartifacts_by_run_id   |    1      1  0.2s
  listartifacts_by_run      |    1      1  0.0s
  listartifacts_by_run_info |    1      1  0.0s
     Testing MLFlowClient tests passed
pebeto commented 2 months ago

Hi.

The error is happening because the previous run doesn't ended correctly. To improve this here, we need to define a post-test routine to delete the experiments even when a test fails.

This can be flagged as an improvement in the next version for the test suite, but it's not a bug.

deyandyankov commented 2 months ago

Right. The tests were written with the assumption that the mlflow test instance will be wiped out after test runs with a separate process and not the tests themselves. This way if the tests fail, one could debug what happened using test output and existing objects in mlflow before wiping out mlflow. So the behaviour was actually intentional. The local development workflow is:

have an mlflow instance running
run the tests
wipe out the instance
run the tests
[...]

The same thing happens when the tests are run in github. A new mlflow instance is spun up and then its automatically wiped out by the job rather than cleaned up by the tests.