oceanprotocol / pdr-backend

Instructions & code to run predictoors, traders, more.
Apache License 2.0
22 stars 15 forks source link

[Bug, Sim] RegrLinearRidge has wonky prediction errors #1269

Closed trentmc closed 1 week ago

trentmc commented 1 week ago

Summary of problem

Check out the residuals (prediction errors) from this sim run. It's on BTC, RegrLinearRidge, transform=None. Residuals should have average=0. Obviously they don't. Screenshot 2024-06-21 at 11 32 26

Classifier run

I did a run on latest codebase (Jun 21, 9.00am) with default ppss.yaml settings except:

And the results look sane. Predictoor bots made 3115.63 OCEAN profit.

classif

Regressor run

Then I ran with exactly the same settings except:

It did horrible. Profit = -18442.56 OCEAN. Profit vs time is almost fully-down (image 1). And the residuals plot had values in the billions, wtf (image 2). (Residuals = prediction errors). Though the regressor's response vs time was sane (image 3). I will look into this

Profit vs time Screenshot 2024-06-21 at 09 39 35

Residuals plot -- the main symptom is here. See plot at very top of this issue.

Regressor response vs time Screenshot 2024-06-21 at 09 38 51

trentmc commented 1 week ago
export SHOW_PLOTS=True
pdr_backend/aimodel/test/test_aimodel_prices.py::test_aimodel_prices_dynamic[RelDiff] -s

Runs a test designed explicitly to see wonky residuals.

Gives the following plot. Residuals look sane here. (But still not sane on sim. Must dig deeper) Screenshot 2024-06-21 at 11 30 33