search

oceanprotocol / pdr-backend

Instructions & code to run predictoors, traders, more.
Apache License 2.0
22 stars 15 forks source link

[Multisim] Profit reports are duplicate, if classification models used #1292

Open graceful-coder opened 4 days ago

graceful-coder commented 4 days ago

Describe the bug Multisim results show repeating values for pdr_profit_OCEAN and trader_profit_USD when testing ClassifLinearRidge model with the three classifiers None, Isotonic & Sigmoid.

To Reproduce Steps to reproduce the behavior:

  1. Do a fresh pdr-backend clone/install including the lastest Jun 22nd commit that also includes the fixes for #1279, #1267, #1212, #1269, & #1275.

  2. Modify my_ppss.yaml to test the ClassifLinearRidge approach with each classifier (None, Isotonic, Sigmoid). You will need 3 different versions of my_ppss.yaml, one for each classifier.

  3. Download and view the CSV results for ClassifLinearRidge to verify that the pdr_profit_OCEAN & trader_profit_USD values are all the same value.

  4. Compare that the results are similar to the ClassifLinearRidge data on Gdrive here.

Expected behavior Non-repeating pdr_profit_OCEAN & trader_profit_USD values for multisim tests of ClassifLinearRidge using all 3 classifiers.

Attached Files The attached files include the ClassifLinearRidge model approach YAML file & CSV results for the Isotonic classifier as an example. ClassifLinearRidge-Isotonic-multisim_metrics_1719194338513.csv Compressed-YAML-file-for-ClassifLinearRige_and_Isotonic.zip

graceful-coder commented 2 days ago

The repeating values for ClassifLinearRidge persist only intermittently when reproducing the bug on a fresh clone of pdr-backend on a new VM (cloned June 26). CSV results of multisim test attached.

Runs 1 &2 are identical as are runs 8 & 9 in the multisim results. multisim_metrics_1719418929996.csv

The pdr-backend repos cloned on June 22nd produced repeating values for ClassifLinearRidge for all runs even when using the same my_ppss.yaml file as in the June 26 test above.

graceful-coder commented 1 day ago

Multisim results for ClassifLinearLasso model do not show duplicated values. Therefore, not all Classifier models are faulty - as far as we know, it’s just ClassifLinearRidge. The bug may have something to do with the unique handling of ClassifLinearRidge for my_ppss.yaml file default or perhaps a significant digits error.

graceful-coder commented 13 hours ago

Duplicated values are appearing intermittently also in the multisim results for ClassifLinearRidge_Balanced model. Perhaps there is an issue with Predictoor handling a unique return value of Scikit's ClassifLinearRidge model.