mlflow / mlflow-go

Repository for the Go-based MLflow Tracking Server
Apache License 2.0
5 stars 4 forks source link

Fix Python tests on Windows platform #19

Closed dsuhinin closed 3 weeks ago

dsuhinin commented 1 month ago

Remaining failing tests:

======================================== short test summary info =========================================
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_local_path_file_uri_correctly_on_windows[\\my_server/my_path/my_sub_path-file:///{drive}my_server/my_path/my_sub_path/{e}] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmp18le7ubp\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_local_path_file_uri_correctly_on_windows[path/to/local/folder-file://{cwd}/path/to/local/folder/{e}] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmp10ch2yhw\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_local_path_file_uri_correctly_on_windows[/path/to/local/folder-file:///{drive}path/to/local/folder/{e}] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmp1x1hpcrf\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_local_path_file_uri_correctly_on_windows[#path/to/local/folder?-file://{cwd}/{e}#path/to/local/folder?] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpmmfxu5n1\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_uri_path_correctly[s3://bucket/path/to/root-s3://bucket/path/to/root/{e}] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmp2l_z4avt\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_uri_path_correctly[s3://bucket/path/to/root?creds=mycreds-s3://bucket/path/to/root/{e}?creds=mycreds] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmplclqpvus\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_uri_path_correctly[dbscheme+driver://root@host/dbname?creds=mycreds#myfragment-dbscheme+driver://root@host/dbname/{e}?creds=mycreds#myfragment] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpq8cq4uzp\\db'    
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_uri_path_correctly[dbscheme+driver://root:password@hostname.com?creds=mycreds#myfragment-dbscheme+driver://root:password@hostname.com/{e}?creds=mycreds#myfragment] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpqcvwgc15\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_experiment_appends_to_artifact_uri_path_correctly[dbscheme+driver://root:password@hostname.com/mydb?creds=mycreds#myfragment-dbscheme+driver://root:password@hostname.com/mydb/{e}?creds=mycreds#myfragment] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpi_1muedx\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_run_appends_to_artifact_local_path_file_uri_correctly_on_windows[\\my_server/my_path/my_sub_path-file:///{drive}my_server/my_path/my_sub_path/{e}/{r}/artifacts] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojafher process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpbkb7nb44\\db'
her process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpbkb7nb44\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_run_appends_to_artifact_uri_path_correctly[s3://bucket/path/to/root-s3://bucket/path/to/root/{e}/{r}/artifacts] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpd2m6r484\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_run_appends_to_artifact_uri_path_correctly[s3://bucket/path/to/root?creds=mycreds-s3://bucket/path/to/root/{e}/{r}/artifacts?creds=mycreds] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmp1a7wjgn1\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_run_appends_to_artifact_uri_path_correctly[dbscheme+driver://root@host/dbname?creds=mycreds#myfragment-dbscheme+driver://root@host/dbname/{e}/{r}/artifacts?creds=mycreds#myfragment] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmp1f4xxaw1\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_run_appends_to_artifact_uri_path_correctly[dbscheme+driver://root:password@hostname.com?creds=mycreds#myfragment-dbscheme+driver://root:password@hostname.com/{e}/{r}/artifacts?creds=mycreds#myfragment] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpoonnyuam\\db'
FAILED .mlflow.repo\tests\store\tracking\test_sqlalchemy_store.py::test_create_run_appends_to_artifact_uri_path_correctly[dbscheme+driver://root:password@hostname.com/mydb?creds=mycreds#myfragment-dbscheme+driver://root:password@hostname.com/mydb/{e}/{r}/artifacts?creds=mycreds#myfragment] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\nojaf\\AppData\\Local\\Temp\\tmpawgo6tl0\\db'
============ 18 failed, 260 passed, 7 skipped, 80 deselected, 6 warnings in 230.83s (0:03:50) ============
Error: running "pytest --confcutdir=. .mlflow.repo/tests/tracking/test_rest_tracking.py .mlflow.repo/tests/store/tracking/test_sqlalchemy_store.py -k not [file -vv" failed with exit code 3221225477
nojaf commented 1 month ago

The current tests still fail on Windows:

======================================================== test session starts ========================================================
platform win32 -- Python 3.9.20, pytest-8.3.3, pluggy-1.5.0 -- C:\Users\nojaf\Projects\mlflow-go\.venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\nojaf\Projects\mlflow-go\.mlflow.repo
configfile: pytest.ini
collected 1 item                                                                                                                     

.mlflow.repo\tests\tracking\test_rest_tracking.py::test_create_get_search_experiment[sqlalchemy] 2024/10/25 15:10:21 INFO tests.tracking.test_rest_tracking: About to create client

---------------------------------------------------------- live log setup ----------------------------------------------------------- 
INFO     builtins:<string>:36 Initializing stores with backend URI sqlite:///C:/Users/nojaf/AppData/Local/Temp/pytest-of-nojaf/pytest-2/test_create_get_search_experim0/sqlalchemy.db and artifact root file:///C:/Users/nojaf/AppData/Local/Temp/pytest-of-nojaf/pytest-2/test_create_get_search_experim0
2024/10/25 15:10:21 INFO mlflow.tracking._tracking_service.registry: sqlite:///C:/Users/nojaf/AppData/Local/Temp/pytest-of-nojaf/pytest-2/test_create_get_search_experim0/sqlalchemy.db
time="2024-10-25T15:10:21+02:00" level=debug msg="Loaded config: &config.Config{Address:\"localhost:5000\", DefaultArtifactRoot:\"file:///C:/Users/nojaf/AppData/Local/Temp/pytest-of-nojaf/pytest-2/test_create_get_search_experim0\", LogLevel:\"DEBUG\", ModelRegistryStoreURI:\"sqlite:///C:/Users/nojaf/AppData/Local/Temp/pytest-of-nojaf/pytest-2/test_create_get_search_experim0/sqlalchemy.db\", PythonEnv:[]string(nil), PythonAddress:\"\", PythonCommand:[]string(nil), PythonTestsENV:map[string]interface {}{\"MLFLOW_TRUNCATE_LONG_VALUES\":true}, ShutdownTimeout:config.Duration{Duration:60000000000}, StaticFolder:\"\", TrackingStoreURI:\"sqlite:///C:/Users/nojaf/AppData/Local/Temp/pytest-of-nojaf/pytest-2/test_create_get_search_experim0/sqlalchemy.db\", Version:\"dev\"}"
time="2024-10-25T15:10:21+02:00" level=debug msg="SQL trace" app_file="github.com/mlflow/mlflow-go/pkg/sql/sql.go:62" app_func="github.com/mlflow/mlflow-go/pkg/sql.initSqlite()" elapsed=0.000ms rows=0 sql="PRAGMA case_sensitive_like = true;"
time="2024-10-25T15:10:21+02:00" level=debug msg="SQL trace" app_file="github.com/mlflow/mlflow-go/pkg/sql/sql.go:63" app_func="github.com/mlflow/mlflow-go/pkg/sql.initSqlite()" elapsed=1.535ms rows=0 sql="PRAGMA journal_mode=WAL;"
2024/10/25 15:10:21 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2024/10/25 15:10:21 INFO mlflow.store.db.utils: Updating database tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 451aebb31d03, add metric step
INFO  [alembic.runtime.migration] Running upgrade 451aebb31d03 -> 90e64c465722, migrate user column to tags
INFO  [alembic.runtime.migration] Running upgrade 90e64c465722 -> 181f10493468, allow nulls for metric values
INFO  [alembic.runtime.migration] Running upgrade 181f10493468 -> df50e92ffc5e, Add Experiment Tags Table
INFO  [alembic.runtime.migration] Running upgrade df50e92ffc5e -> 7ac759974ad8, Update run tags with larger limit
INFO  [alembic.runtime.migration] Running upgrade 7ac759974ad8 -> 89d4b8295536, create latest metrics table
INFO  [89d4b8295536_create_latest_metrics_table_py] Migration complete!
INFO  [alembic.runtime.migration] Running upgrade 89d4b8295536 -> 2b4d017a5e9b, add model registry tables to db
INFO  [2b4d017a5e9b_add_model_registry_tables_to_db_py] Adding registered_models and model_versions tables to database.
INFO  [2b4d017a5e9b_add_model_registry_tables_to_db_py] Migration complete!
INFO  [alembic.runtime.migration] Running upgrade 2b4d017a5e9b -> cfd24bdc0731, Update run status constraint with killed
INFO  [alembic.runtime.migration] Running upgrade cfd24bdc0731 -> 0a8213491aaa, drop_duplicate_killed_constraint
INFO  [alembic.runtime.migration] Running upgrade 0a8213491aaa -> 728d730b5ebd, add registered model tags table
INFO  [alembic.runtime.migration] Running upgrade 728d730b5ebd -> 27a6a02d2cf1, add model version tags table
INFO  [alembic.runtime.migration] Running upgrade 27a6a02d2cf1 -> 84291f40a231, add run_link to model_version
INFO  [alembic.runtime.migration] Running upgrade 84291f40a231 -> a8c4a736bde6, allow nulls for run_id
INFO  [alembic.runtime.migration] Running upgrade a8c4a736bde6 -> 39d1c3be5f05, add_is_nan_constraint_for_metrics_tables_if_necessary
INFO  [alembic.runtime.migration] Running upgrade 39d1c3be5f05 -> c48cb773bb87, reset_default_value_for_is_nan_in_metrics_table_for_mysql
INFO  [alembic.runtime.migration] Running upgrade c48cb773bb87 -> bd07f7e963c5, create index on run_uuid
INFO  [alembic.runtime.migration] Running upgrade bd07f7e963c5 -> 0c779009ac13, add deleted_time field to runs table
INFO  [alembic.runtime.migration] Running upgrade 0c779009ac13 -> cc1f77228345, change param value length to 500
INFO  [alembic.runtime.migration] Running upgrade cc1f77228345 -> 97727af70f4d, Add creation_time and last_update_time to experiments table
INFO  [alembic.runtime.migration] Running upgrade 97727af70f4d -> 3500859a5d39, Add Model Aliases table
INFO  [alembic.runtime.migration] Running upgrade 3500859a5d39 -> 7f2a7d5fae7d, add datasets inputs input_tags tables
INFO  [alembic.runtime.migration] Running upgrade 7f2a7d5fae7d -> 2d6e25af4d3e, increase max param val length from 500 to 8000        
INFO  [alembic.runtime.migration] Running upgrade 2d6e25af4d3e -> acf3f17fdcc7, add storage location field to model versions
INFO  [alembic.runtime.migration] Running upgrade acf3f17fdcc7 -> 867495a8f9d4, add trace tables
INFO  [alembic.runtime.migration] Running upgrade 867495a8f9d4 -> 5b0e9adcef9c, add cascade deletion to trace tables foreign keys     
INFO  [alembic.runtime.migration] Running upgrade 5b0e9adcef9c -> 4465047574b1, increase max dataset schema size
INFO  [alembic.runtime.migration] Running upgrade 4465047574b1 -> f5a4f2784254, increase run tag value limit to 8000
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
time="2024-10-25T15:10:21+02:00" level=debug msg="SQL trace" app_file="github.com/mlflow/mlflow-go/pkg/tracking/store/sql/experiments.go:307" app_func="github.com/mlflow/mlflow-go/pkg/tracking/store/sql.TrackingSQLStore.SearchExperiments()" elapsed=8.963ms rows=0 sql="SELECT * FROM `experiments` WHERE lifecycle_stage IN (\"active\",\"deleted\") ORDER BY experiments.creation_time DESC,experiments.experiment_id ASC LIMIT 1001"
Windows fatal exception: access violation

Current thread 0x000042d0 (most recent call first):
  File "C:\Users\nojaf\Projects\mlflow-go\mlflow_go\store\_service_proxy.py", line 28 in call_endpoint
  File "C:\Users\nojaf\Projects\mlflow-go\mlflow_go\store\tracking.py", line 222 in search_experiments
  File "C:\Users\nojaf\Projects\mlflow-go\.mlflow.repo\mlflow\store\tracking\sqlalchemy_store.py", line 185 in __init__
  File "C:\Users\nojaf\Projects\mlflow-go\mlflow_go\store\tracking.py", line 65 in __init__
  File "C:\Users\nojaf\Projects\mlflow-go\mlflow_go\store\tracking.py", line 247 in _get_sqlalchemy_store
  File "C:\Users\nojaf\Projects\mlflow-go\.mlflow.repo\mlflow\tracking\_tracking_service\registry.py", line 54 in _get_store_with_resolved_uri
  File "C:\Users\nojaf\Projects\mlflow-go\.mlflow.repo\mlflow\tracking\_tracking_service\registry.py", line 44 in get_store
  File "<string>", line 39 in _init_server
  File "C:\Users\nojaf\AppData\Roaming\uv\python\cpython-3.9.20-windows-x86_64-none\lib\contextlib.py", line 119 in __enter__
  File "C:\Users\nojaf\Projects\mlflow-go\.mlflow.repo\tests\tracking\test_rest_tracking.py", line 83 in mlflow_client
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\fixtures.py", line 891 in call_fixture_func
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\fixtures.py", line 1140 in pytest_fixture_setup
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\fixtures.py", line 1091 in execute
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\fixtures.py", line 617 in _get_active_fixturedef
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\fixtures.py", line 532 in getfixturevalue
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\fixtures.py", line 697 in _fillfixtures
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\python.py", line 1630 in setup
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\runner.py", line 514 in setup
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\runner.py", line 160 in pytest_runtest_setup
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\runner.py", line 242 in <lambda>
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\runner.py", line 341 in from_call
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\runner.py", line 241 in call_and_report
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\runner.py", line 126 in runtestprotocol
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\runner.py", line 113 in pytest_runtest_protocol
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\main.py", line 362 in pytest_runtestloop
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\main.py", line 337 in _main
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\main.py", line 283 in wrap_session
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\main.py", line 330 in pytest_cmdline_main
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\config\__init__.py", line 175 in main
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\lib\site-packages\_pytest\config\__init__.py", line 201 in console_main
  File "C:\Users\nojaf\Projects\mlflow-go\.venv\Scripts\pytest.exe\__main__.py", line 8 in <module>
  File "C:\Users\nojaf\AppData\Roaming\uv\python\cpython-3.9.20-windows-x86_64-none\lib\runpy.py", line 87 in _run_code
  File "C:\Users\nojaf\AppData\Roaming\uv\python\cpython-3.9.20-windows-x86_64-none\lib\runpy.py", line 197 in _run_module_as_main    
Error: running "uv run pytest -s --log-cli-level=DEBUG --confcutdir=. -k not [file .mlflow.repo\tests\tracking\test_rest_tracking.py::test_create_get_search_experiment[sqlalchemy] -vv" failed with exit code 3221225477

Windows fatal exception: access violation occurs in the first read.