search

kaizen-ai / kaizenflow

KaizenFlow is a framework for Bayesian reasoning and AI/ML stream computing
GNU General Public License v3.0
105 stars 74 forks source link

Unit test Master_system_run_debugger #1032

Open DanilYachmenev opened 3 weeks ago

DanilYachmenev commented 3 weeks ago

There is a Master notebook oms/notebooks/Master_system_run_debugger.ipynb

We need to add a unit test for it to Test_run_master_notebooks located at oms/test/test_notebooks.py

For reference use similar unit test methods for other Master notebooks located in the same test class. See unit test doc to follow the code style Also highly recommend to read this doc before submitting 1st PR

FYI @gpsaggese @sonaalKant @samarth9008

samarth9008 commented 3 weeks ago

@mayank922 Since you are unavailable due to other commitments, will assign you a different issue once you are back

heanhsok commented 3 weeks ago

hi @DanilYachmenev

I'm getting The config profile (ck) could not be found error when running the unit test.

user_501@0fb28187a90e:/app/oms/test$ pytest test_notebooks.py
==================================================== test session starts =====================================================
platform linux -- Python 3.9.5, pytest-7.4.2, pluggy-1.3.0 -- /venv/bin/python
cachedir: .pytest_cache
rootdir: /app
configfile: pytest.ini
plugins: xdist-3.3.1, timeout-2.1.0, instafail-0.5.0, rerunfailures-12.0, cov-4.1.0, anyio-4.0.0
collecting 4 items                                                                                                           # Git
  branch_name='SorrTask1032_Unit_test_Master_system_run_debugger'
  hash='465f66e44'
  # Last commits:
    * 465f66e44 Jaydeep  KaizenTask1046_Resume_resampling_Binance_bidask_data (#1047)      (   2 hours ago) Thu Jun 13 23:18:13 2024  (HEAD -> SorrTask1032_Unit_test_Master_system_run_debugger, origin/master, origin/HEAD, master)
    * 52eb5f1c1 Mayank Raj Added Unit test fo dassert_strictly_increasing_index() (#1031)    (    2 days ago) Tue Jun 11 21:28:45 2024
    * 8ca212e0b Surbhi Sharma Unit Test Cases For fill_stats.py added (#1012)                   (    2 days ago) Tue Jun 11 21:24:49 2024
# Machine info
  system=Linux
  node name=0fb28187a90e
  release=6.6.12-linuxkit
  version=#1 SMP Fri Jan 19 08:53:17 UTC 2024
  machine=aarch64
  processor=aarch64
  cpu count=8
  cpu freq=None
  memory=svmem(total=8326987776, available=7182749696, percent=13.7, used=927289344, free=3482656768, active=1136762880, inactive=3278905344, buffers=73879552, cached=3843162112, shared=8691712, slab=291176448)
  disk usage=sdiskusage(total=62671097856, used=22993645568, free=36460736512, percent=38.7)
# Packages
  python: 3.9.5
  cvxopt: 1.3.2
  cvxpy: 1.4.2
  gluonnlp: ?
  gluonts: ?
  joblib: 1.3.2
  mxnet: ?
  numpy: 1.26.0
  pandas: 2.1.1
  pyarrow: 15.0.0
  scipy: 1.11.3
  seaborn: 0.13.0
  sklearn: 1.3.1
  statsmodels: 0.14.0
21:18:52 - INFO  hdbg.py init_logger:1010                               Saving log to file '{'tmp.pytest.log'}'
21:18:52 - INFO  hdbg.py init_logger:1015                               > cmd='/venv/bin/pytest test_notebooks.py'
collected 4 items

test_notebooks.py::Test_run_master_notebooks::test_run_master_bid_ask_execution_analysis (0.44 s) FAILED                        [ 25%]
test_notebooks.py::Test_run_master_notebooks::test_run_master_broker_debugging (0.38 s) FAILED                                  [ 50%]
test_notebooks.py::Test_run_master_notebooks::test_run_master_broker_portfolio_reconciliation (0.37 s) FAILED                   [ 75%]
test_notebooks.py::Test_run_master_notebooks::test_run_master_execution_analysis (0.37 s) FAILED                                [100%]

========================================================== FAILURES ==========================================================
____________________________ Test_run_master_notebooks.test_run_master_bid_ask_execution_analysis ____________________________
Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib/python3.9/unittest/case.py", line 593, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    method()
  File "/app/oms/test/test_notebooks.py", line 136, in test_run_master_bid_ask_execution_analysis
    self._run_notebook(notebook_path, config_builder)
  File "/app/oms/test/test_notebooks.py", line 190, in _run_notebook
    hs3.copy_data_from_s3_to_local_dir(s3_input_dir, scratch_dir, aws_profile)
  File "/app/helpers/hs3.py", line 892, in copy_data_from_s3_to_local_dir
    hsystem.system(cmd, suppress_output=False, log_level="echo")
  File "/app/helpers/hsystem.py", line 303, in system
    rc, _ = _system(
  File "/app/helpers/hsystem.py", line 277, in _system
    raise RuntimeError(
RuntimeError: cmd='(aws s3 sync s3://cryptokaizen-unit-test/outcomes/Test_run_master_notebooks.test_run_master_bid_ask_execution_analysis/input /app/oms/test/outcomes/Test_run_master_notebooks.test_run_master_bid_ask_execution_analysis/tmp.scratch --profile ck) 2>&1' failed with rc='255'
truncated output=

The config profile (ck) could not be found

_________________________________ Test_run_master_notebooks.test_run_master_broker_debugging _________________________________
Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib/python3.9/unittest/case.py", line 593, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    method()
  File "/app/oms/test/test_notebooks.py", line 147, in test_run_master_broker_debugging
    self._run_notebook(notebook_path, config_builder)
  File "/app/oms/test/test_notebooks.py", line 190, in _run_notebook
    hs3.copy_data_from_s3_to_local_dir(s3_input_dir, scratch_dir, aws_profile)
  File "/app/helpers/hs3.py", line 892, in copy_data_from_s3_to_local_dir
    hsystem.system(cmd, suppress_output=False, log_level="echo")
  File "/app/helpers/hsystem.py", line 303, in system
    rc, _ = _system(
  File "/app/helpers/hsystem.py", line 277, in _system
    raise RuntimeError(
RuntimeError: cmd='(aws s3 sync s3://cryptokaizen-unit-test/outcomes/Test_run_master_notebooks.test_run_master_broker_debugging/input /app/oms/test/outcomes/Test_run_master_notebooks.test_run_master_broker_debugging/tmp.scratch --profile ck) 2>&1' failed with rc='255'
truncated output=

The config profile (ck) could not be found

_________________________ Test_run_master_notebooks.test_run_master_broker_portfolio_reconciliation __________________________
Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib/python3.9/unittest/case.py", line 593, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    method()
  File "/app/oms/test/test_notebooks.py", line 180, in test_run_master_broker_portfolio_reconciliation
    self._run_notebook(notebook_path, config_builder)
  File "/app/oms/test/test_notebooks.py", line 190, in _run_notebook
    hs3.copy_data_from_s3_to_local_dir(s3_input_dir, scratch_dir, aws_profile)
  File "/app/helpers/hs3.py", line 892, in copy_data_from_s3_to_local_dir
    hsystem.system(cmd, suppress_output=False, log_level="echo")
  File "/app/helpers/hsystem.py", line 303, in system
    rc, _ = _system(
  File "/app/helpers/hsystem.py", line 277, in _system
    raise RuntimeError(
RuntimeError: cmd='(aws s3 sync s3://cryptokaizen-unit-test/outcomes/Test_run_master_notebooks.test_run_master_broker_portfolio_reconciliation/input /app/oms/test/outcomes/Test_run_master_notebooks.test_run_master_broker_portfolio_reconciliation/tmp.scratch --profile ck) 2>&1' failed with rc='255'
truncated output=

The config profile (ck) could not be found

________________________________ Test_run_master_notebooks.test_run_master_execution_analysis ________________________________
Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib/python3.9/unittest/case.py", line 593, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    method()
  File "/app/oms/test/test_notebooks.py", line 163, in test_run_master_execution_analysis
    self._run_notebook(notebook_path, config_builder)
  File "/app/oms/test/test_notebooks.py", line 190, in _run_notebook
    hs3.copy_data_from_s3_to_local_dir(s3_input_dir, scratch_dir, aws_profile)
  File "/app/helpers/hs3.py", line 892, in copy_data_from_s3_to_local_dir
    hsystem.system(cmd, suppress_output=False, log_level="echo")
  File "/app/helpers/hsystem.py", line 303, in system
    rc, _ = _system(
  File "/app/helpers/hsystem.py", line 277, in _system
    raise RuntimeError(
RuntimeError: cmd='(aws s3 sync s3://cryptokaizen-unit-test/outcomes/Test_run_master_notebooks.test_run_master_execution_analysis/input /app/oms/test/outcomes/Test_run_master_notebooks.test_run_master_execution_analysis/tmp.scratch --profile ck) 2>&1' failed with rc='255'
truncated output=

The config profile (ck) could not be found

==================================================== slowest 3 durations =====================================================
0.45s call     oms/test/test_notebooks.py::Test_run_master_notebooks::test_run_master_bid_ask_execution_analysis
0.39s call     oms/test/test_notebooks.py::Test_run_master_notebooks::test_run_master_broker_debugging
0.37s call     oms/test/test_notebooks.py::Test_run_master_notebooks::test_run_master_broker_portfolio_reconciliation
================================================== short test summary info ===================================================
FAILED test_notebooks.py::Test_run_master_notebooks::test_run_master_bid_ask_execution_analysis - RuntimeError: cmd='(aws s3 sync s3://cryptokaizen-unit-test/outcomes/Test_run_master_notebooks.test_run_master_bid_ask_execution_analysis/input /app/oms/test/outcomes/Test_run_master_notebooks.test_run_master_bid_ask_execution_analysis/tmp.scratch --profile ck) 2>&1' failed with rc='255'
truncated output=

The config profile (ck) could not be found
FAILED test_notebooks.py::Test_run_master_notebooks::test_run_master_broker_debugging - RuntimeError: cmd='(aws s3 sync s3://cryptokaizen-unit-test/outcomes/Test_run_master_notebooks.test_run_master_broker_debugging/input /app/oms/test/outcomes/Test_run_master_notebooks.test_run_master_broker_debugging/tmp.scratch --profile ck) 2>&1' failed with rc='255'
truncated output=

The config profile (ck) could not be found
FAILED test_notebooks.py::Test_run_master_notebooks::test_run_master_broker_portfolio_reconciliation - RuntimeError: cmd='(aws s3 sync s3://cryptokaizen-unit-test/outcomes/Test_run_master_notebooks.test_run_master_broker_portfolio_reconciliation/input /app/oms/test/outcomes/Test_run_master_notebooks.test_run_master_broker_portfolio_reconciliation/tmp.scratch --profile ck) 2>&1' failed with rc='255'
truncated output=

The config profile (ck) could not be found
FAILED test_notebooks.py::Test_run_master_notebooks::test_run_master_execution_analysis - RuntimeError: cmd='(aws s3 sync s3://cryptokaizen-unit-test/outcomes/Test_run_master_notebooks.test_run_master_execution_analysis/input /app/oms/test/outcomes/Test_run_master_notebooks.test_run_master_execution_analysis/tmp.scratch --profile ck) 2>&1' failed with rc='255'
truncated output=

The config profile (ck) could not be found
===================================================== 4 failed in 6.32s ======================================================
21:18:54 - INFO  hcache.py clear_global_cache:294                       Before clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=36.0 KB
21:18:54 - WARN  hcache.py clear_global_cache:295                       Resetting 'global mem' cache '/mnt/tmpfs/tmp.cache.mem'
21:18:54 - WARN  hcache.py clear_global_cache:305                       Destroying '/mnt/tmpfs/tmp.cache.mem' ...
21:18:54 - INFO  hcache.py clear_global_cache:321                       After clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=nan
heanhsok commented 3 weeks ago

I'm also unable to execute this cell in the Master_system_run_debugger notebook. Is this expected?

Cell

target_dir = reconcil.get_target_dir(
    config["dst_root_dir"],
    config["dag_builder_name"],
    config["run_mode"],
    config["start_timestamp_as_str"],
    config["end_timestamp_as_str"],
)
system_log_dir = reconcil.get_prod_system_log_dir(config["mode"])
system_log_dir = os.path.join(target_dir, system_log_dir)
_LOG.info("system_log_dir=%s", system_log_dir)

Error

INFO  dst_root_dir='/shared_data/ecs/preprod/system_reconciliation', dag_builder_name='C5b', run_mode='paper_trading', start_timestamp_as_str='20230713_131000', end_timestamp_as_str='20230714_130500'
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[5], line 1
----> 1 target_dir = reconcil.get_target_dir(
      2     config["dst_root_dir"],
      3     config["dag_builder_name"],
      4     config["run_mode"],
      5     config["start_timestamp_as_str"],
      6     config["end_timestamp_as_str"],
      7 )
      8 system_log_dir = reconcil.get_prod_system_log_dir(config["mode"])
      9 system_log_dir = os.path.join(target_dir, system_log_dir)

File /app/reconciliation/sim_prod_reconciliation.py:963, in get_target_dir(dst_root_dir, dag_builder_name, run_mode, start_timestamp_as_str, end_timestamp_as_str, tag, aws_profile)
    938 """
    939 Return the target dir name to store reconcilation results.
    940 
   (...)
    956 :return: a target dir to store reconcilation results
    957 """
    958 _LOG.info(
    959     hprint.to_str(
    960         "dst_root_dir dag_builder_name run_mode start_timestamp_as_str end_timestamp_as_str"
    961     )
    962 )
--> 963 hs3.dassert_path_exists(dst_root_dir, aws_profile)
    964 hdbg.dassert_isinstance(dag_builder_name, str)
    965 hdbg.dassert_isinstance(start_timestamp_as_str, str)

File /app/helpers/hs3.py:119, in dassert_path_exists(path, aws_profile)
    117     hdbg.dassert(s3fs_.exists(path), f"S3 path '{path}' doesn't exist!")
    118 else:
--> 119     hdbg.dassert_path_exists(path)

File /app/helpers/hdbg.py:754, in dassert_path_exists(path, msg, only_warning, *args)
    752 if not os.path.exists(path):
    753     txt = f"Path '{path}' doesn't exist!"
--> 754     _dfatal(txt, msg, *args, only_warning=only_warning)

File /app/helpers/hdbg.py:142, in _dfatal(txt, msg, only_warning, *args)
    139     _LOG.warning(dfatal_txt)
    140 else:
    141     # Abort.
--> 142     dfatal(dfatal_txt)

File /app/helpers/hdbg.py:71, in dfatal(message, assertion_type)
     69 if assertion_type is None:
     70     assertion_type = AssertionError
---> 71 raise assertion_type(ret)

AssertionError: 
################################################################################
* Failed assertion *
Path '/shared_data/ecs/preprod/system_reconciliation' doesn't exist!
################################################################################
DanilYachmenev commented 3 weeks ago

@heanhsok eh, I see, the problem is that you have yet no access to our aws and shared data dirs

which prevents you to run the script and the notebook completely

However, since you've already invested your time here and there is a lot of reference code, let me reformulate your task: create all the necessary code for the requested test:

You do not need to run anything but pls take a look at the similar functions at every step and try to make equal code structure that will fit your notebook

The file PR and assign me as a reviewer - I will do the rest No need to worry about small bugs or incorrect values - you cant run it so its all good But try to make is as much similar to the similar tests as possible so it will decrease my debugging time

wdyt?

heanhsok commented 3 weeks ago

Got it. I'll do that and file a PR for your review. Thanks!