search

oceanprotocol / pdr-backend

Instructions & code to run predictoors, traders, more.
Apache License 2.0
29 stars 22 forks source link

[Sim] When balancing classes, all predictions are 'up'. Weird. #731

Closed trentmc closed 7 months ago

trentmc commented 7 months ago

Background / motivation

We recently introduced balance_classes with possible values of SMOTE | RandomOverSampler | None.

Weirdly, when it runs in sim_engine for 5000 epochs, it never predicts "up". Weird.

Why is this?

Key datapoint: it only does the weird behavior when balance_classes != None.

Examples without up

5000 logreg AR=1 BTC ETH weight_recent=10x_5x, balance_classes=RandomOverSampler

5000 logreg AR=1 BTC ETH weight_recent=10x_5x, balance_classes=SMOTE

Examples with up

5000 logreg AR=1 BTC ETH

800 svc ar=10 BTC

1000 svc AR=1 BTC ETH max-n-train=100

5000 svc AR=3 BTC ETH ocv n-train=200 weight-recnet calibrate-probs

1000 svc ar=3 BTC ETH max-n-train=100

trentmc commented 7 months ago

I just ran this again. It doesn't have this issue anymore; I couldn't reproduce the results from before. It's likely that there was a bug.