Closed trentmc closed 1 week ago
export SHOW_PLOTS=True
pdr_backend/aimodel/test/test_aimodel_prices.py::test_aimodel_prices_dynamic[RelDiff] -s
Runs a test designed explicitly to see wonky residuals.
Gives the following plot. Residuals look sane here. (But still not sane on sim. Must dig deeper)
Summary of problem
Check out the residuals (prediction errors) from this sim run. It's on BTC, RegrLinearRidge, transform=None. Residuals should have average=0. Obviously they don't.![Screenshot 2024-06-21 at 11 32 26](https://github.com/oceanprotocol/pdr-backend/assets/5305452/1524153f-060c-41d9-ab7b-812409565472)
Classifier run
I did a run on latest codebase (Jun 21, 9.00am) with default ppss.yaml settings except:
lake_ss.st_timestr: 180 days ago
aimodel_data_ss.max_n_train: 1500
aimodel_data_ss.autoregressive_n: 6
ClassifLinearRidge
)And the results look sane. Predictoor bots made 3115.63 OCEAN profit.
Regressor run
Then I ran with exactly the same settings except:
approach: RegrLinearRidge
It did horrible. Profit = -18442.56 OCEAN. Profit vs time is almost fully-down (image 1). And the residuals plot had values in the billions, wtf (image 2). (Residuals = prediction errors). Though the regressor's response vs time was sane (image 3). I will look into this
Profit vs time![Screenshot 2024-06-21 at 09 39 35](https://github.com/oceanprotocol/pdr-backend/assets/5305452/cd6d2491-4ae6-453f-abc9-12ffb46d8801)
Residuals plot -- the main symptom is here. See plot at very top of this issue.
Regressor response vs time![Screenshot 2024-06-21 at 09 38 51](https://github.com/oceanprotocol/pdr-backend/assets/5305452/9da08b9d-d157-466c-b6f8-3222b0f31f19)