search

singnet / reputation

MIT License
11 stars 11 forks source link

Python RS: “predictiveness” #247

Closed akolonin closed 4 years ago

akolonin commented 5 years ago
  1. Implement per 3.3.a in the spec: https://docs.google.com/document/d/1-O7avb_zJKvCXRvD0FmvyoVZdSmkDMlXRB5wsavpSAM/edit#
  2. Provide unit test
  3. Extend list of RS configurations next to "RS=Biased" at https://github.com/singnet/reputation/blob/master/reputation/reputation_buying_test.py#L87 and see if it works better
nejc9921 commented 5 years ago

@akolonin : When I currently run your code, I get error checking unittests:

FAIL: test_predictiveness_java (test_aigents_reputation_api.TestAigentsAPIReputationService)

Traceback (most recent call last): File "/home/nejc/other/git_pull2/reputation/reputation/test_reputation.py", line 995, in test_predictiveness_java self.assertDictEqual(ranks,{'11': 100.0, '13': 100.0, '12': 55.0, '14': 33.0}) AssertionError: {'11': 100.0, '13': 100.0, '14': 100.0, '12': 55.0} != {'11': 100.0, '13': 100.0, '12': 55.0, '14': 33.0}

akolonin commented 5 years ago

@nejc9921 - yes, by some reason it does not work for me on the server but works on local machine, will be sorting this out. Ignore this for now.

akolonin commented 5 years ago

Performance is too way different now:

good_agent = {"buyers":[1,80], "products":[101,180], "qualities":[0.5,0.75,1.0], "transactions": good_transactions} bad_agent = {"buyers":[81,100], "products":[181,200], "qualities":[0.0,0.25], "transactions": bad_transactions} days = 101

...

rs = PythonReputationService() rs.set_parameters({'fullnorm':True,'weighting':False,'logratings':False,'denomination':False,'unrated':False,'default':0.5,'decayed':0.5,'conservatism':0.5,'ratings':1.0,'spendings':0.0}) print("RS=Predictive1.0_c0.1_P", end =" ") rs.set_parameters({'predictiveness':1.0,'conservatism':0.1,'rating_bias':False,'fullnorm':True,'weighting':True ,'logratings':False,'denomination':True ,'unrated':False ,'default':0.5,'decayed':0.5,'ratings':1.0,'spendings':0.0}) reputation_simulate(good_agent,bad_agent, datetime.date(2018, 1, 1), days, True, threshold, 30, rs, verbose)

...

rs = AigentsAPIReputationService('http://localtest.com:1180/', 'john@doe.org', 'q', 'a', False, 'test', True) rs.set_parameters({'fullnorm':True,'weighting':False,'logratings':False,'denomination':False,'unrated':False,'default':0.5,'decayed':0.5,'conservatism':0.5,'ratings':1.0,'spendings':0.0}) print("RS=Predictive1.0_c0.1_J", end =" ") rs.set_parameters({'predictiveness':1.0,'conservatism':0.1,'rating_bias':False,'fullnorm':True,'weighting':True ,'logratings':False,'denomination':True ,'unrated':False ,'default':0.5,'decayed':0.5,'ratings':1.0,'spendings':0.0}) reputation_simulate(good_agent,bad_agent, datetime.date(2018, 1, 1), days, True, threshold, 30, rs, verbose)

...

Output:

RS=Predictive1.0_c0.1_P PLRo=30 agents=80/20/80/20 days=101 Organic: 16830 Sponsored: 4000 Organic2Sponsored: 14720 Organic/Sponsored: 4.21 LTS: 0.87 PFS: 3.68 RS=Predictive1.0_c0.1_J PLRo=30 agents=80/20/80/20 days=101 Organic: 11000 Sponsored: 4000 Organic2Sponsored: 1040 Organic/Sponsored: 2.75 LTS: 0.09 PFS: 0.26

What is interesting, the combination works pretty well in Python:

print("RS=Biased&Predictive1.0_c0.1", end =" ") rs.set_parameters({'predictiveness':1.0,'conservatism':0.1,'rating_bias':True,'fullnorm':True,'weighting':True ,'logratings':False,'denomination':True ,'unrated':False ,'default':0.5,'decayed':0.5,'ratings':1.0,'spendings':0.0}) reputation_simulate(good_agent,bad_agent, datetime.date(2018, 1, 1), days, True, threshold, 30, rs, verbose)

Output:

RS=Biased&Predictive1.0_c0.1 PLRo=30 agents=80/20/80/20 days=101 Organic: 12110 Sponsored: 4000 Organic2Sponsored: 460 Organic/Sponsored: 3.03 LTS: 0.04 PFS: 0.12

akolonin commented 5 years ago

Same issue with another setup:

good_agent = {"buyers":[1,400], "products":[1001,1400], "qualities":[0.5,0.75,1.0], "transactions": good_transactions} bad_agent = {"buyers":[401,500], "products":[1401,1500], "qualities":[0.0,0.25], "transactions": bad_transactions} days = 501

...

RS=Predictive1.0_c0.1_P PLRo=30 agents=400/100/400/100 days=501 Organic: 463620 Sponsored: 100000 Organic2Sponsored: 369900 Organic/Sponsored: 4.64 LTS: 0.8 PFS: 3.7

RS=Predictive1.0_c0.1_J PLRo=30 agents=400/100/400/100 days=501 Organic: 264310 Sponsored: 100000 Organic2Sponsored: 33890 Organic/Sponsored: 2.64 LTS: 0.13 PFS: 0.34

...

RS=Biased&Predictive1.0_c0.1 PLRo=30 agents=400/100/400/100 days=501 Organic: 284340 Sponsored: 100000 Organic2Sponsored: 11040 Organic/Sponsored: 2.84 LTS: 0.04 PFS: 0.11

akolonin commented 5 years ago

@nejc9921 - given the results below, does it sound like Python version is working not good enough?

good_agent = {"buyers":[1,80], "products":[101,180], "qualities":[0.5,0.75,1.0], "transactions": good_transactions}

bad_agent = {"buyers":[81,100], "products":[181,200], "qualities":[0.0,0.25], "transactions": bad_transactions}

days = 101

Java: RS=Predictive0.5_c0.5 PLRo=30 agents=80/20/80/20 days=101 Organic: 17000 Sponsored: 4000 Organic2Sponsored: 15320 Organic/Sponsored: 4.25 LTS: 0.9 PFS: 3.83 RS=Predictive1.0_c0.5 PLRo=30 agents=80/20/80/20 days=101 Organic: 15750 Sponsored: 4000 Organic2Sponsored: 3520 Organic/Sponsored: 3.94 LTS: 0.22 PFS: 0.88 RS=Predictive1.0_c0.9 PLRo=30 agents=80/20/80/20 days=101 Organic: 29210 Sponsored: 4000 Organic2Sponsored: 16000 Organic/Sponsored: 7.3 LTS: 0.55 PFS: 4.0 RS=Predictive1.0_c0.1 PLRo=30 agents=80/20/80/20 days=101 Organic: 11000 Sponsored: 4000 Organic2Sponsored: 1040 Organic/Sponsored: 2.75 LTS: 0.09 PFS: 0.26 RS=Predictive_c0.1_def0.1 PLRo=30 agents=80/20/80/20 days=101 Organic: 11040 Sponsored: 4000 Organic2Sponsored: 1240 Organic/Sponsored: 2.76 LTS: 0.11 PFS: 0.31 RS=Predictive_c0.1_def0.9 PLRo=30 agents=80/20/80/20 days=101 Organic: 10480 Sponsored: 4000 Organic2Sponsored: 820 Organic/Sponsored: 2.62 LTS: 0.08 PFS: 0.2 RS=Predictive_c0.1_dec0.1 PLRo=30 agents=80/20/80/20 days=101 Organic: 9060 Sponsored: 4000 Organic2Sponsored: 2380 Organic/Sponsored: 2.27 LTS: 0.26 PFS: 0.59 RS=Predictive_c0.1_dec0.9 PLRo=30 agents=80/20/80/20 days=101 Organic: 29210 Sponsored: 4000 Organic2Sponsored: 16000 Organic/Sponsored: 7.3 LTS: 0.55 PFS: 4.0

Python: RS=Predictive0.5_c0.5 PLRo=30 agents=80/20/80/20 days=101 Organic: 17540 Sponsored: 4000 Organic2Sponsored: 15200 Organic/Sponsored: 4.38 LTS: 0.87 PFS: 3.8 RS=Predictive1.0_c0.5 PLRo=30 agents=80/20/80/20 days=101 Organic: 17540 Sponsored: 4000 Organic2Sponsored: 15200 Organic/Sponsored: 4.38 LTS: 0.87 PFS: 3.8 RS=Predictive1.0_c0.9 PLRo=30 agents=80/20/80/20 days=101 Organic: 29210 Sponsored: 4000 Organic2Sponsored: 16000 Organic/Sponsored: 7.3 LTS: 0.55 PFS: 4.0 RS=Predictive1.0_c0.1 PLRo=30 agents=80/20/80/20 days=101 Organic: 16830 Sponsored: 4000 Organic2Sponsored: 14720 Organic/Sponsored: 4.21 LTS: 0.87 PFS: 3.68 RS=Predictive_c0.1_def0.1 PLRo=30 agents=80/20/80/20 days=101 Organic: 16760 Sponsored: 4000 Organic2Sponsored: 14890 Organic/Sponsored: 4.19 LTS: 0.89 PFS: 3.72 RS=Predictive_c0.1_def0.9 PLRo=30 agents=80/20/80/20 days=101 Organic: 16780 Sponsored: 4000 Organic2Sponsored: 14670 Organic/Sponsored: 4.2 LTS: 0.87 PFS: 3.67 RS=Predictive_c0.1_dec0.1 PLRo=30 agents=80/20/80/20 days=101 Organic: 17340 Sponsored: 4000 Organic2Sponsored: 15000 Organic/Sponsored: 4.33 LTS: 0.87 PFS: 3.75 RS=Predictive_c0.1_dec0.9 PLRo=30 agents=80/20/80/20 days=101 Organic: 29210 Sponsored: 4000 Organic2Sponsored: 16000 Organic/Sponsored: 7.3 LTS: 0.55 PFS: 4.0

good_agent = {"buyers":[1,400], "products":[1001,1400], "qualities":[0.5,0.75,1.0], "transactions": good_transactions} bad_agent = {"buyers":[401,500], "products":[1401,1500], "qualities":[0.0,0.25], "transactions": bad_transactions} days = 501

Java: RS=Predictive0.5_c0.5 PLRo=30 agents=400/100/400/100 days=501 Organic: 452870 Sponsored: 100000 Organic2Sponsored: 364020 Organic/Sponsored: 4.53 LTS: 0.8 PFS: 3.64 RS=Predictive1.0_c0.5 PLRo=30 agents=400/100/400/100 days=501 Organic: 666060 Sponsored: 100000 Organic2Sponsored: 376120 Organic/Sponsored: 6.66 LTS: 0.56 PFS: 3.76 RS=Predictive1.0_c0.9 PLRo=30 agents=400/100/400/100 days=501 Organic: 689940 Sponsored: 100000 Organic2Sponsored: 400000 Organic/Sponsored: 6.9 LTS: 0.58 PFS: 4.0 RS=Predictive1.0_c0.1 PLRo=30 agents=400/100/400/100 days=501 Organic: 264310 Sponsored: 100000 Organic2Sponsored: 33890 Organic/Sponsored: 2.64 LTS: 0.13 PFS: 0.34 RS=Predictive_c0.1_def0.1 PLRo=30 agents=400/100/400/100 days=501 Organic: 689940 Sponsored: 100000 Organic2Sponsored: 400000 Organic/Sponsored: 6.9 LTS: 0.58 PFS: 4.0 RS=Predictive_c0.1_def0.9 PLRo=30 agents=400/100/400/100 days=501 Organic: 259920 Sponsored: 100000 Organic2Sponsored: 37040 Organic/Sponsored: 2.6 LTS: 0.14 PFS: 0.37 RS=Predictive_c0.1_dec0.1 PLRo=30 agents=400/100/400/100 days=501 Organic: 232060 Sponsored: 100000 Organic2Sponsored: 8960 Organic/Sponsored: 2.32 LTS: 0.04 PFS: 0.09 RS=Predictive_c0.1_dec0.9 PLRo=30 agents=400/100/400/100 days=501 Organic: 689940 Sponsored: 100000 Organic2Sponsored: 400000 Organic/Sponsored: 6.9 LTS: 0.58 PFS: 4.0

Python: RS=Predictive0.5_c0.5 PLRo=30 agents=400/100/400/100 days=501 Organic: 419980 Sponsored: 100000 Organic2Sponsored: 341470 Organic/Sponsored: 4.2 LTS: 0.81 PFS: 3.41 RS=Predictive1.0_c0.5 PLRo=30 agents=400/100/400/100 days=501 Organic: 419980 Sponsored: 100000 Organic2Sponsored: 341470 Organic/Sponsored: 4.2 LTS: 0.81 PFS: 3.41 RS=Predictive1.0_c0.9 PLRo=30 agents=400/100/400/100 days=501 Organic: 675960 Sponsored: 100000 Organic2Sponsored: 386020 Organic/Sponsored: 6.76 LTS: 0.57 PFS: 3.86 RS=Predictive1.0_c0.1 PLRo=30 agents=400/100/400/100 days=501 Organic: 463620 Sponsored: 100000 Organic2Sponsored: 369900 Organic/Sponsored: 4.64 LTS: 0.8 PFS: 3.7 RS=Predictive_c0.1_def0.1 PLRo=30 agents=400/100/400/100 days=501 Organic: 460660 Sponsored: 100000 Organic2Sponsored: 369010 Organic/Sponsored: 4.61 LTS: 0.8 PFS: 3.69 RS=Predictive_c0.1_def0.9 PLRo=30 agents=400/100/400/100 days=501 Organic: 408660 Sponsored: 100000 Organic2Sponsored: 369640 Organic/Sponsored: 4.09 LTS: 0.9 PFS: 3.7 RS=Predictive_c0.1_dec0.1 PLRo=30 agents=400/100/400/100 days=501 Organic: 406190 Sponsored: 100000 Organic2Sponsored: 352230 Organic/Sponsored: 4.06 LTS: 0.87 PFS: 3.52 RS=Predictive_c0.1_dec0.9 PLRo=30 agents=400/100/400/100 days=501 Organic: 689940 Sponsored: 100000 Organic2Sponsored: 400000 Organic/Sponsored: 6.9 LTS: 0.58 PFS: 4.0

good_agent = {"buyers":[1,800], "products":[1001,1800], "qualities":[0.5,0.75,1.0], "transactions": good_transactions} bad_agent = {"buyers":[801,1000], "products":[1801,2000], "qualities":[0.0,0.25], "transactions": bad_transactions} days = 1001

Java: RS=Predictive0.5_c0.5 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1607340 Sponsored: 400000 Organic2Sponsored: 1559620 Organic/Sponsored: 4.02 LTS: 0.97 PFS: 3.9 RS=Predictive1.0_c0.5 PLRo=30 agents=800/200/800/200 days=1001 Organic: 2979460 Sponsored: 400000 Organic2Sponsored: 1600000 Organic/Sponsored: 7.45 LTS: 0.54 PFS: 4.0 RS=Predictive1.0_c0.9 PLRo=30 agents=800/200/800/200 days=1001 Organic: 2979390 Sponsored: 400000 Organic2Sponsored: 1600000 Organic/Sponsored: 7.45 LTS: 0.54 PFS: 4.0 RS=Predictive1.0_c0.1 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1110700 Sponsored: 400000 Organic2Sponsored: 72530 Organic/Sponsored: 2.78 LTS: 0.07 PFS: 0.18 RS=Predictive_c0.1_def0.1 PLRo=30 agents=800/200/800/200 days=1001 Organic: 2979460 Sponsored: 400000 Organic2Sponsored: 1600000 Organic/Sponsored: 7.45 LTS: 0.54 PFS: 4.0 RS=Predictive_c0.1_def0.9 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1091400 Sponsored: 400000 Organic2Sponsored: 79310 Organic/Sponsored: 2.73 LTS: 0.07 PFS: 0.2 RS=Predictive_c0.1_dec0.1 PLRo=30 agents=800/200/800/200 days=1001 Organic: 890610 Sponsored: 400000 Organic2Sponsored: 21790 Organic/Sponsored: 2.23 LTS: 0.02 PFS: 0.05 RS=Predictive_c0.1_dec0.9 PLRo=30 agents=800/200/800/200 days=1001 Organic: 2979460 Sponsored: 400000 Organic2Sponsored: 1600000 Organic/Sponsored: 7.45 LTS: 0.54 PFS: 4.0

Python: RS=Predictive0.5_c0.5 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1765000 Sponsored: 400000 Organic2Sponsored: 1380070 Organic/Sponsored: 4.41 LTS: 0.78 PFS: 3.45 RS=Predictive1.0_c0.5 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1765000 Sponsored: 400000 Organic2Sponsored: 1380070 Organic/Sponsored: 4.41 LTS: 0.78 PFS: 3.45 RS=Predictive1.0_c0.9 PLRo=30 agents=800/200/800/200 days=1001 Organic: 2157310 Sponsored: 400000 Organic2Sponsored: 1434620 Organic/Sponsored: 5.39 LTS: 0.67 PFS: 3.59 RS=Predictive1.0_c0.1 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1865260 Sponsored: 400000 Organic2Sponsored: 1511460 Organic/Sponsored: 4.66 LTS: 0.81 PFS: 3.78 RS=Predictive_c0.1_def0.1 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1617780 Sponsored: 400000 Organic2Sponsored: 1496840 Organic/Sponsored: 4.04 LTS: 0.93 PFS: 3.74 RS=Predictive_c0.1_def0.9 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1617350 Sponsored: 400000 Organic2Sponsored: 1510220 Organic/Sponsored: 4.04 LTS: 0.93 PFS: 3.78 RS=Predictive_c0.1_dec0.1 PLRo=30 agents=800/200/800/200 days=1001 Organic: 1616500 Sponsored: 400000 Organic2Sponsored: 1460010 Organic/Sponsored: 4.04 LTS: 0.9 PFS: 3.65 RS=Predictive_c0.1_dec0.9 PLRo=30 agents=800/200/800/200 days=1001 Organic: 2979460 Sponsored: 400000 Organic2Sponsored: 1600000 Organic/Sponsored: 7.45 LTS: 0.54 PFS: 4.0

nejc9921 commented 5 years ago

I've been stuck on this for a while and I was thinking of some additional logs to include. I documented them in the following document: https://drive.google.com/file/d/1ieGIrgrBzz9kxtO2C9auJKgvaBGn_yNs/view?usp=sharing

If you can include them, it would help me a lot. I might add more changes for anti-biased setting.

nejc9921 commented 5 years ago

Python predictiveness is working, but due to the bugs in both codes, there are some differences currently with Java. Currently running longer tests to determine the size of the differences.

akolonin commented 5 years ago

Java issue is handled here https://github.com/aigents/aigents-java/issues/1 If Python version of "predictiveness" is working as expected?

nejc9921 commented 5 years ago

hmm, should I tests if the unittests converge now?

akolonin commented 5 years ago

1) Just leave the Java RS-specific tests as is. 2) Complete Python RS-specific tests, make sure it is working as expected and let me know.

nejc9921 commented 4 years ago

Can we close this? I did what you said in the last comment, though sync with Java is not tested...