[Bug, sim] SimEngine._sell() 'Out of tokens, this shouldn't happen' when predicting on Volume

[trentmc]

The issue / to reproduce

In fresh branch of main (as of Jun 21, 10.00 CEST). Latest commit: https://github.com/oceanprotocol/pdr-backend/commit/07791e3f094d142d8e15b7c08b0a18cb3a18470d

Had this setup file: ppss.yaml.txt.

Then ran: pdr sim ppss.yaml

It failed. Summary of traceback:

  File "/Users/trentmc/code/pdr-backend/pdr_backend/sim/sim_engine.py", line 283, in sim_trader
    usd_received = self._sell(curprice, tokcoin_amt_send)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/venv/lib/python3.12/site-packages/enforce_typing/decorator.py", line 29, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/pdr_backend/sim/sim_engine.py", line 358, in _sell
    raise Exception("Out of tokens, this shouldn't happen.")

Towards a Solution

The latest commit -- https://github.com/oceanprotocol/pdr-backend/commit/07791e3f094d142d8e15b7c08b0a18cb3a18470d -- had changed SimEngine:_sell(). So the bug was likely introduced through that.

The change causing the problem was: predict/train on volume not close.

  predict_train_feedsets:
    - predict: binance BTC/USDT v 5m
      train_on:
      - binance BTC/USDT v 5m

Appendix: What ppss.yaml changes fail, vs not

(This section documents some digging, leading to realizing that the problem is in predicting volume.)

The ppss.yaml file I was using had several changes compared to default.

What fails and doesn't

1. FAILS: Setup file shared above. Note it has transform: RelDiff.
2. FAILS: We get the same bug with same setup file, but transform: None
3. OK: We don't get the error with default ppss.yaml setup file

A git diff of setup of (2):

Let's start from default ppss.yaml, and systematically go through the diffs.

• lake_ss.feeds change --> OK
• lake_ss.st_timestr change --> OK
• max_n_train change --> OK
• autoregressive_n change --> OK
• transform change --> OK
• predict_train_feedsets change to predict and train on just v --> FAILS
• predict_train_feedsets change to train on just cv, keep predicting on c --> OK

Full traceback

(venv) trentmc@trents-m3: ~/code/pdr-backend $ pdr sim ppss.yaml 
do_lake_submcommand: parsed_args: ['sim', 'ppss.yaml']
2024-06-21 09:55:01,652 cli INFO pdr sim: Begin
2024-06-21 09:55:01,652 cli INFO Arguments:
2024-06-21 09:55:01,652 cli INFO PPSS_FILE=ppss.yaml
2024-06-21 09:55:01,652 cli INFO Nested args: {}
2024-06-21 09:55:01,671 cli WARNING Multiple predict feeds provided, using the first one
2024-06-21 09:55:01,672 sim_engine INFO Start run
2024-06-21 09:55:01,672 sim_engine INFO Initialize plot data.
2024-06-21 09:55:01,672 ohlcv_data_factory INFO Get historical data, across many exchanges & pairs: begin.
2024-06-21 09:55:01,673 ohlcv_data_factory INFO Data start: timestamp=1703408101673, dt=2023-12-24_08:55:01.673
2024-06-21 09:55:01,673 ohlcv_data_factory INFO Data fin: timestamp=1718956501673, dt=2024-06-21_07:55:01.673
2024-06-21 09:55:01,673 ohlcv_data_factory INFO Update all rawohlcv files: begin
2024-06-21 09:55:01,673 ohlcv_data_factory INFO Update rawohlcv file at exch=binance, pair=BTC/USDT: begin
2024-06-21 09:55:01,673 ohlcv_data_factory INFO filename=/Users/trentmc/code/pdr-backend/lake_data/binance_BTC-USDT_5m.parquet
2024-06-21 09:55:01,673 ohlcv_data_factory INFO File already exists
2024-06-21 09:55:01,679 ohlcv_data_factory INFO File starts at: timestamp=1687183800000, dt=2023-06-19_14:10:00.000
2024-06-21 09:55:01,679 ohlcv_data_factory INFO File finishes at: timestamp=1718955300000, dt=2024-06-21_07:35:00.000
2024-06-21 09:55:01,679 ohlcv_data_factory INFO User-specified start >= file start, so append file
2024-06-21 09:55:01,679 ohlcv_data_factory INFO Aim to fetch data from start time: timestamp=1718955600000, dt=2024-06-21_07:40:00.000
2024-06-21 09:55:01,680 ohlcv_data_factory INFO Fetch up to 1000 pts from timestamp=1718955600000, dt=2024-06-21_07:40:00.000
2024-06-21 09:55:05,559 lake_plutil INFO Just appended 4 df rows to file /Users/trentmc/code/pdr-backend/lake_data/binance_BTC-USDT_5m.parquet
2024-06-21 09:55:05,559 ohlcv_data_factory INFO Update rawohlcv file at exch=binance, pair=BTC/USDT: done
2024-06-21 09:55:05,559 ohlcv_data_factory INFO Update all rawohlcv files: done
2024-06-21 09:55:05,559 ohlcv_data_factory INFO Load rawohlcv file.
2024-06-21 09:55:05,689 ohlcv_data_factory INFO Get historical data, across many exchanges & pairs: done.
Traceback (most recent call last):
  File "/Users/trentmc/code/pdr-backend/./pdr", line 6, in <module>
    cli_module._do_main()
  File "/Users/trentmc/code/pdr-backend/venv/lib/python3.12/site-packages/enforce_typing/decorator.py", line 29, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/pdr_backend/cli/cli_module.py", line 72, in _do_main
    func(args, nested_args)
  File "/Users/trentmc/code/pdr-backend/venv/lib/python3.12/site-packages/enforce_typing/decorator.py", line 29, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/pdr_backend/cli/cli_module.py", line 93, in do_sim
    sim_engine.run()
  File "/Users/trentmc/code/pdr-backend/venv/lib/python3.12/site-packages/enforce_typing/decorator.py", line 29, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/pdr_backend/sim/sim_engine.py", line 111, in run
    self.run_one_iter(test_i, mergedohlcv_df)
  File "/Users/trentmc/code/pdr-backend/venv/lib/python3.12/site-packages/enforce_typing/decorator.py", line 29, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/pdr_backend/sim/sim_engine.py", line 181, in run_one_iter
    profit = self.sim_trader(curprice, pred_up, pred_down, conf_up, conf_down)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/venv/lib/python3.12/site-packages/enforce_typing/decorator.py", line 29, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/pdr_backend/sim/sim_engine.py", line 283, in sim_trader
    usd_received = self._sell(curprice, tokcoin_amt_send)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/venv/lib/python3.12/site-packages/enforce_typing/decorator.py", line 29, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/trentmc/code/pdr-backend/pdr_backend/sim/sim_engine.py", line 358, in _sell
    raise Exception("Out of tokens, this shouldn't happen.")
Exception: Out of tokens, this shouldn't happen.

[trentmc]

Closing, wontfix. Why: as of PR#1212 (merged), sim engine does not try to predict on volume. Adds too much complexity. Therefore this issue is N/A.