Strategy discussion

[Barigamb738]

Hello. I wanted to create this issue for people to shear their ideas with eachother. For example in one of my earlier attempts i wanted to awoid some of the flaws of ftft by doing 2 in total and not 2 in a row and it worked. But i am going with a different strategy now so i don't need this. I know it's not the best strategy to help your enemies but i did it anyways. If you have any strategy ideas that you don't need anymore but someone could use it, post it here.

[nobody5050]

Interesting thread, I’ll have to watch and see what people post

[jichanbachan]

I'm personally interested in how well any sort of machine learning algorithm works. I have to assume that it would just work as a worse detective, because it would take longer to have any accuracy. It would also probably be beaten by all the other "nice" algorithms since it has the potential to lose massive points against any algorithms with a grim trigger if it defects at any point during training. A text I saw online regarding this showed that it basically just learnt the same thing as TFT, which makes sense. But regardless, I'm still interested in if anyone has managed one that's remotely successful.

[JesseB0rn]

I'm personally interested in how well any sort of machine learning algorithm works. I have to assume that it would just work as a worse detective, because it would take longer to have any accuracy. It would also probably be beaten by all the other "nice" algorithms since it has the potential to lose massive points against any algorithms with a grim trigger if it defects at any point during training. A text I saw online regarding this showed that it basically just learnt the same thing as TFT, which makes sense. But regardless, I'm still interested in if anyone has managed one that's remotely successful.

It acts like tft, but beats it by 0.1 - 0.2 points

[SebastianKorytkowski]

From my experience so far there isn't just one best strategy it all depends on the environment. I think one of the nice strategies will win. What I mean by nice?

1. Do not defect first - this will make you lose the always cooperates bonus, because if you do not defect there is no way to distinguish titfortat and always cooperates and I doubt many people will submit always cooperates, so that's negligible.
2. Try to forgive for defections. (D,D) (D,D) is worse than (D,C) (C,D) and a lot worse than (C,C) (C,C). Try to give the opponent a couple of chances.

In short try to be nice and do your best to optimize for different rude strategies you can encounter. Not trying to win against them (as they have very small chances of winning overall), but trying to get them to cooperate as much as you can.

[nobody5050]

I'm personally interested in how well any sort of machine learning algorithm works. I have to assume that it would just work as a worse detective, because it would take longer to have any accuracy. It would also probably be beaten by all the other "nice" algorithms since it has the potential to lose massive points against any algorithms with a grim trigger if it defects at any point during training. A text I saw online regarding this showed that it basically just learnt the same thing as TFT, which makes sense. But regardless, I'm still interested in if anyone has managed one that's remotely successful.

On the subject of machine learning, here’s a very interesting paper on evolved algorithms for iterated prisoner’s dilemma: https://www.bc.edu/content/dam/files/schools/cas_sites/cs/pdf/academics/honors/06DanielScali.pdf

[Barigamb738]

I am having a problem rn and i was wondering if it happening to other people as well.(if it does it is not really a problem) Whatever i do i always get like 1 or 2 points more than tit for tat and if i improve tit for tat improves. it's like our relative points are fixed. it isn't a problem here with the examples but in the real game it could cause some trouble for me.

[nobody5050]

I think you’re incorrectly responding to tit for tat. I think you’re playing a detective, so if you add a case for if tit for tat happens in your detecting round you could be able to switch to a strategy which earns you points

[Barigamb738]

thanks

[Barigamb738]

also my strat is very different form the detective i just named it that cuz it inspects the opponent.

[Barigamb738]

@nobody5050 i checked the rounds and there was nothing in there that would be bad. detectiveplus (P1) VS. titForTat (P2) C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C Final score for detectiveplus: 3.0 Final score for titForTat: 3.0

[l4vr0v]

TL;DR: "Nice" strategies have an advantage unless "mean" strategies are good at inferring their opponent's patterns and detecting opponents against whom defection has positive long-term net expected value (like ftft and random). If you're able to pick up on random or to find a way to give tats without receiving any tits, you make up for the price you pay to the tit-for-tat variants (grimTrigger and alwaysCooperate more or less cancel each other out, although depending on your olive branch strategy and the like you could get significantly net-negative between these two). Unprovoked defectors aren't non-viable, they're just hard to figure out.

I think one of the nice strategies will win. What I mean by nice?

Do not defect first - this will make you lose the always cooperates bonus, because if you do not defect there is no way to distinguish titfortat and always cooperates and I doubt many people will submit always cooperates, so that's negligible.

Counterpoint: unprovoked defection is heavily penalized and not worth it, unless you can farm it for intelligence. In the base meta (with the 9 defaults), let's discuss the optimal unprovoked defector's behavior and how it would play out, ignoring any attempts to predict when the game is close to ending:

grimTrigger: the optimal unprovoked defector would defect (D/C) for +2, then find itself in a D/D loop for the rest of the game, with an opportunity cost of 2 points per turn relative to a C/C loop. Net gain: 2 points - 2 points per remaining move. Initial defection sequence: D->D...

alwaysCooperate: the optimal unprovoked defector would defect (D/C) for +2, then continue defecting to get 5 points per move instead of the 3 it would've gotten by cooperating. Net gain: 2 points + 2 points per remaining move. Initial defection sequence: D->D...

alwaysDefect: the optimal unprovoked defector would at best defect on the first turn, saving the one point it would lose from a C/D as opposed to the optimal chain of D/Ds against this opponent. Net gain: 1 point. Initial defection sequence: D->D...

titForTat: the optimal unprovoked defector would defect (D/C +5), then cooperate (C/D +0) and return to the cooperation loop. Net gain: -1 point. Initial defection sequence: D->C

joss: similar to titForTat except against joss the cooperation loop doesn't happen consistently since joss also defects without provocation. But if it doesn't defect first and our defector does, and there's no random defection right after the return to cooperation by the optimal defector, it plays out the same as titForTat (since, with joss's 10% chance of unprovoked defection combined with tit-for-tat mechanics, cooperating with joss is better). Net gain: -1 point. Initial defection sequence: D->C

detective: this is also secretly a titForTat variant. Our unprovoked defector would defect on the first 4 turns to exploit the testing schedule, then somehow detect it's encountered a detective, then eat the loss of a C/D to revert to the cooperation loop. We gain 7 points during the testing schedule, then have to eat a loss of 1 point to get back into a C/C loop. Net gain: 6 points. Initial defection sequence: D->D->D->D->C

ftft: our optimal defector defects (+5, +2 better than C/C), cooperates next turn and notices it gets no response, then detects it's playing against ftt and spams DCDCDC to gain on average 1 point per remaining move. Net gain: 2 points + 1 point per remaining move. Initial defection sequence: D->C

simpleton: our optimal unprovoked defector sets off the simpleton with a D/C (+5) then defects again (D/D for +1), then returns to the cooperation loop. Net gain: 0 points. Initial defection sequence: D->D

random: our optimal unprovoked defector defects (expected value increases by 1.5), then immediately detects somehow it's playing against random and gains 1.5 points on average per remaining move. Net gain: 1.5 points per remaining move. Initial defection sequence: D->D

One thing you've probably noticed is that these optimal defectors above are not the same (the initial defection sequences) so you aren't able to get that juicy combined net gain of 11 + 2.5/remaining move. Instead, for just the 9 above, the combined optimal strategy (assuming you're really really good at detecting your opponent) will do significantly worse, although obviously the points per remaining move will have more weight than the constants. I'll leave putting all this together as an exercise for the reader (you just have to weigh the pros/cons of a single initial defection sequence for all these cases + the unknown mix of actual opponents in the meta).

Anyhow, going back to my point: if you just look at grimTrigger, alwaysCooperates, and the tit-for-tat variants then realistically your optimal unprovoked defector sets you back. Unprovoked defection is locally sub-optimal vs. being "nice" and so far "nice" strategies have done really well in competitions like this one. But if you are able to leverage your unprovoked defection into information that allows you to counterplay your opponent (like with ftft or random) highly effectively, that will rapidly outweigh the price you pay to the grimTrigger.

I initially experimented with de-escalating tit-for-tat variants (that break out of the D/D loop but don't get abused by alwaysDefects and the like), and after a point I realized detection and counterplay would have by far the best gains of my remaining options so I have now switched onto the detective variant train. This is, of course, very risky because detection and counterplay logic can get fairly brittle and cost you a lot. And if there's many "grim" strategies in the mix (that disproportionately reward you for the first defection or punish you for it in an unclear way), then your unprovoked defector will either be non-viable (although, fwiw, there's at least one "grim" strategy in the mix- beNice- the maximally punishes titForTat, so it's fairly hard to really figure out what the meta will be like) or will have to be really good about exiting the D/D loop via opportunistic olive branches (C/D's cost you 1 point relative to a D/D but if you get a cooperation in return, you immediately make back that cost).

This is all non-trivial. But "mean" strategies- unprovoked defectors like detective variants- have a place in the meta and are one of the two most viable builds I've seen, along with de-escalating tit-for-tats (that similarly use olive branches to exit D/D loops).

[l4vr0v]

@Barigamb738 if you want to investigate your problem, try pulling in #31 onto your local branch, generating that head-to-head CSV (maybe for runs where you vary the meta), importing it into a spreadsheet application, and then you can see exactly where that performance difference between your titForTat and detectivePlus is coming from. Then you can check results.txt to see the details of that.

[SebastianKorytkowski]

I agree. Though the biggest problem here is predicting your enemy's move. My guess is that the most common strategy will probably be some kind of a titForTat variant. Crafting a detective or something like that that's able to respond to all of these would be very hard. I think maybe some fancy machine learning could pull it off. II can't wait to see the winning strategy. I doubt it will be mine, but I still had some fun learning about game theory.

[l4vr0v]

You could have a very conservative detective that tries to behave the same as a tit-for-tat except when it thinks it's got random, alwaysCooperate, or ftft, and then backs off immediately to return to tit-for-tat if those assumptions don't hold. Unless it gets fooled very often or there's many grims in the meta (that increase the cost of unprovoked defection, although they shouldn't be because unnecessary defections are costly), it should do strictly better than tit-for-tat.

Plus for what it's worth, detective strats themselves are tit-for-tat variants.

[jichanbachan]

I personally think that mean strategies and detective types aren’t going to hold up as well as nice strategies in this tournament. The benefit of betraying an alwaysCooperate vs entering a defect loop with a grim trigger seem to cancel each other out, but no one is going to be submitting more versions of alwaysCooperate. I think it’s far more likely that there are going to be variants of grim trigger.

But again, the winner of this tournament will definitely be decided by the pool of strategies. Maybe there’ll be a ton of algorithms for a mean or detective type to take advantage of. Either that or Random will miraculously win by playing every matchup perfectly. That’ll certainly be something to see.

[l4vr0v]

@nobody5050 just ran a big mosh pit of strats visible in people's repos and a detective variant by @Lasermancer did quite well in there.

The benefit of betraying an alwaysCooperate vs entering a defect loop with a grim trigger seem to cancel each other out, but no one is going to be submitting more versions of alwaysCooperate

alwaysCooperate and grimTrigger don't define the detective meta because they cancel one another out and because grimTrigger has no chance of winning (because it suicides itself into D/D loops; sustained defection is expensive). Tit-for-tat variants and strategies with significant randomization do. Against random moves, defecting doubles your expected value. Against tit-for-tat algorithms, you can keep your unprovoked defection cost as low as 1 point per match. And then you see forgiving tit-for-tat variants (like ftft) and that's what a perfect detective would exploit: you find a way to defect but get the overly generous ones to not respond to some of your tats with tits.

[jichanbachan]

I’ll have to look into that mosh pit and see what sort of things other people are doing.

alwaysCooperate and grimTrigger don't define the detective meta because they cancel one another out and because grimTrigger has no chance of winning (because it suicides itself into D/D loops; sustained defection is expensive).

In regards to this, I meant more that it’s more likely that there’ll be more grim trigger variants than alwaysCooperate in the pool, so defecting first could very well end up in a net loss in points instead of cancelling out. Filling up the pool with grim triggers destroys any detective’s score because nice strategies get ahead (not that that’s going to happen).

But like you said, for detectives it’s going to be up to exploiting variations of tit for tat and random-likes. Likewise, for “nice” strategies, I reckon it might be up to how they can deal with any detectives.

[l4vr0v]

I meant more that it’s more likely that there’ll be more grim trigger variants than alwaysCooperate in the pool

Why though? Punishing your opponent for defection traps you in a D/D loop in most cases and is suicide for any strategy that wants to win.

Yeah, the problem with the "nice" strat is that you're capped by forgiveness and your average-case game is a 3-point match. D/D is very inefficient so you have to throw olive branches every once in a while to an opponent who wrongs you. This means that your worst-case games average less than 1 point per match.

Meanwhile, every single full-exploit a detective is able to pull off gets them a whole point per match. Even if they lose some of their 3-point matches or dip a bit, they can make massive gains from exploitative counterplay. In theory, a detective can do considerably better than any "nice" strat... but it just needs to be almost magically good at adapting to opponents.

[jichanbachan]

Why though? Punishing your opponent for defection traps you in a D/D loop in most cases and is suicide for any strategy that wants to win.

Haha, I know there’s at least two or three grim trigger variants already so I’m just going off what I’ve seen from this tournament so far. I wholeheartedly agree with you, “nice” strats are capped at 3 points for the most part and only strats that initiate defection have a chance to go way higher. It’ll just be up to how well detectives can accomodate for various strats.

[Barigamb738]

Thanks @l4vr0v i will try it

[arielsysdev]

I don't think I'm going to win this but it's nice to share experiences with people. I'll go over the different issues I encountered when developing my algorithm.

Loops It seems any form of loop (c/c, d/c, d/d) wouldn't give you much advantage since at most, you are capped at an average of 3 for c/c, 2.5 for d/c, and 1 for d/d. So the best gains can be had by actively changing your pattern, while checking opponent response and pattern, and evaluating the best exploitative strat for that opponent. I thought my algorithm was pretty good in being able to confidently identify opponent patterns on 5-10 turns or so, but I've seen other strats that are able to do it with 1-3.

Don't fear the grudger I realized this late but the losses you fear you'll have by making unprovoked defects isn't as big as you think than the cost of not identifying your opponent.

Reactive strats won't give much bang for your buck Strats that actively adapt to you won't give you much points overall, this is why tit for tat was so good, no matter what you do your strat won't make much difference in points vs it.

Cash in on unreactive or slow to react strats Strats like random, always cooperate, ftft would give you the biggest gains if you identify them and exploit them as soon as possible. They are pretty easy to identify but it comes down to implementation. So the best strat is one that is able to soonest cash in on those strats, while maintaining a good defense against adaptive strats, and with some great unpredictability to slow adapters to really squeeze out those marginal gains. Most competitor submitted strats would probably be variants of adaptive strats so getting marginal gains on hundreds of these strats will define the winner. If your strat has no way to defend itself when the opponent has gone past your initial reaction, then you're already doomed.

[redtachyon2098]

Is it better to try to take down others, or raise your own score?

[Barigamb738]

I DID IT!!!!!!!!! I finally made a strategy that can get top 6 consistently against @l4vr0v's exampleStrats!! I don't wanna reveal much since this probably will be my final strategy but I am gonna say that it is an iteration of joss and called OmegaJoss. I am currently tweaking the parameters to see what is the best. I am really glad I didn't quit these few days of coding algorithms were my best few days of 2021.

[nobody5050]

I DID IT!!!!!!!!! I finally made a strategy that can get top 6 consistently against @l4vr0v's exampleStrats!! I don't wanna reveal much since this probably will be my final strategy but I am gonna say that it is an iteration of joss and called OmegaJoss. I am currently tweaking the parameters to see what is the best. I am really glad I didn't quit these few days of coding algorithms were my best few days of 2021.

Congratulations!

[redtachyon2098]

I DID IT!!!!!!!!! I finally made a strategy that can get top 6 consistently against @l4vr0v's exampleStrats!! I don't wanna reveal much since this probably will be my final strategy but I am gonna say that it is an iteration of joss and called OmegaJoss. I am currently tweaking the parameters to see what is the best. I am really glad I didn't quit these few days of coding algorithms were my best few days of 2021.

Nice! I couldn't do that, but I'm happy you did. Only two days left! I'm getting hyped. So this is what a deadline feels like if you are working on something you are truly passionate for!

[AndrePinheiroPT]

sorry for my bad english I have just know that @l4vr0v have 36 strategies with I can test mine strategy. Anyway, my strategy is basically a math function with works so well and I wanna test with that 36 strategies! I am not at home now, but I will test today... It's cool see people happy with that tournament and I hope that you guys are having fun haha :)

[Barigamb738]

I DID IT!!!!!!!!! I finally made a strategy that can get top 6 consistently against @l4vr0v's exampleStrats!! I don't wanna reveal much since this probably will be my final strategy but I am gonna say that it is an iteration of joss and called OmegaJoss. I am currently tweaking the parameters to see what is the best. I am really glad I didn't quit these few days of coding algorithms were my best few days of 2021.

Also since then it won against them

[redtachyon2098]

Only one day left!!!!!!!!!!!

[redtachyon2098]

The thing I submitted only has 11 lines of code. I hope it does well.... I couldn't improve upon it.