search

Axelrod-Python / axelrod-fortran

Python wrapper library around TourExec Fortran for Axelrod's second tournament.
Other
4 stars 0 forks source link

Bug with `k61` #62

Closed drvinceknight closed 7 years ago

drvinceknight commented 7 years ago

After taking another look at the discrepancy for k61r (equivalent to Champion) I have found a bug.

TLDR k61r works the very first time it is called in a match but every subsequent match after that a section of it does not behave properly.

k61r should randomly defect after the 25th turn (dependent on the opponent's cooperation ratio).

It behaves as expected the very first time it is used but for every subsequent call it does not. Here is a minimal failing code snippet:

>>> import axelrod as axl
>>> import axelrod_fortran as axlf

>>> def run_champion_v_alternator():
...     player = axlf.Player("k61r")
...     opponent = axl.Alternator()
...     match = axl.Match((player, opponent))
...     axl.seed(0)
...     interactions = match.play()
...     axl.seed(0)
...     assert interactions == match.play()
>>> run_champion_v_alternator()
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-1-70d9b1c75b35> in <module>()
     13     axl.seed(0)
     14     assert interactions == match.play()
---> 15 run_champion_v_alternator()

<ipython-input-1-70d9b1c75b35> in run_champion_v_alternator()
     12 
     13     axl.seed(0)
---> 14     assert interactions == match.play()
     15 run_champion_v_alternator()

AssertionError:

Note however that if I rerun the function in the same session it does not fail:

>>> run_champion_v_alternator()

However, it is indeed the very first play of the match that is correct. Here starting a new Python session:

>>> import axelrod as axl
>>> import axelrod_fortran as axlf
>>> player = axlf.Player("k61r")
>>> opponent = axl.Alternator()
>>> match = axl.Match((player, opponent))
>>> axl.seed(0)
>>> interactions = match.play()
>>> interactions[25:30]  # k61r *does* defect
[(C, D), (C, C), (C, D), (D, C), (C, D)]
>>> axl.seed(0)
>>> interactions = match.play()
>>> interactions[25:30]  # k61r no longer defects
[(C, D), (C, C), (C, D), (C, C), (C, D)]

Modifying this line in the Fortran code https://github.com/Axelrod-Python/TourExec/blob/master/src/strategies/k61r.f#L11:

COPRAT = FLOAT(ICOOP) / FLOAT(ITURN)

to just be a constant:

COPRAT = 0.5

gives consistent (although obviously incorrect) behaviour which makes me believe that somehow something is happening which means we're not "reloading" things correctly (I'm guessing here)?

Things I have tried:

Some good news, I've run the following:

for strategy in axlf.all_strategies:
    player = axlf.Player(strategy)
    for opponent in axl.basic_strategies:
        match = axl.Match((player, opponent()))
        axl.seed(0)
        interactions = match.play()

        axl.seed(0)
        if interactions != match.play():
            print(player, opponent)

and the only player that seems to be affected by this bug is k61r. (FYI, I attempted to add this snippet as a test but that doesn't fail because the test is run out of order, in other words it doesn't detect the bug because k61r has already been called by another test).

meatballs commented 7 years ago

Interesting. I wonder if this is differences between modern and older compilers

On 8 October 2017 16:32:02 BST, Vince Knight notifications@github.com wrote:

After taking another look at the discrepancy for k61r (equivalent to Champion) I have found a bug.

k61r should randomly defect after the 25th turn (dependent on the opponent's cooperation ratio).

It behaves as expected the very first time it is used but for every subsequent call it does not. Here is a minimal failing code snippet:

>>> import axelrod as axl
>>> import axelrod_fortran as axlf

>>> def run_champion_v_alternator():
...     player = axlf.Player("k61r")
...     opponent = axl.Alternator()

...     match = axl.Match((player, opponent))

...     axl.seed(0)
...     interactions = match.play()

...     axl.seed(0)
...     assert interactions == match.play()
>>> run_champion_v_alternator()
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call
last)
<ipython-input-1-70d9b1c75b35> in <module>()
    13     axl.seed(0)
    14     assert interactions == match.play()
---> 15 run_champion_v_alternator()

<ipython-input-1-70d9b1c75b35> in run_champion_v_alternator()
    12 
    13     axl.seed(0)
---> 14     assert interactions == match.play()
    15 run_champion_v_alternator()

AssertionError:

Note however that if I rerun the function in the same session it does not fail:

>>> run_champion_v_alternator()

However, it is indeed the very first play of the match that is correct. Here starting a new Python session:

>>> import axelrod as axl
>>> import axelrod_fortran as axlf
>>> player = axlf.Player("k61r")
>>> opponent = axl.Alternator()
>>> match = axl.Match((player, opponent))
>>> axl.seed(0)
>>> interactions = match.play()
>>> interactions[25:30]  # k61r *does* defect
[(C, D), (C, C), (C, D), (D, C), (C, D)]
>>> axl.seed(0)
>>> interactions = match.play()
>>> interactions[25:30]  # k61r no longer defects
[(C, D), (C, C), (C, D), (C, C), (C, D)]

Modifying this line in the Fortran code https://github.com/Axelrod-Python/TourExec/blob/master/src/strategies/k61r.f#L11:

COPRAT = FLOAT(ICOOP) / FLOAT(ITURN)

to just be a constant:

COPRAT = 0.5

gives consistent (although obviously incorrect) behaviour which makes me believe that somehow something is happening which means we're not "reloading" things correctly (I'm guessing here)?

Things I have tried:

  • Change how our reset method works in axlf to "dump" and reload the CDLL but that's not come to anything (although I've pretty much been fumbling around in the dark).
  • Looked at the make file to see if this could be due to a compilation flag but that's waaaaaay above my understanding of things so who knows...
  • Change the Fortran code in k61r but again with no success - Note that if we could resolve this by modifying axlfthat would be preferable (from the point of view of the narrative of "using the code as is").

Some good news: I've run the following:

for strategy in axlf.all_strategies:
   player = axlf.Player(strategy)
   for opponent in axl.basic_strategies:
       match = axl.Match((player, opponent()))
       axl.seed(0)
       interactions = match.play()

       axl.seed(0)
       if interactions != match.play():
           print(player, opponent)

and the only player that seems to be affected by this bug is k61r. (FYI, I attempted to add this snippet as a test but that doesn't fail because the test is run out of order, in other words it doesn't detect the bug because k61r has already been called).

-- Owen Campbell

This email has been digitally signed. For further information visit http://owencampbell.me.uk/pgp

drvinceknight commented 7 years ago

Interesting. I wonder if this is differences between modern and older compilers

Perhaps :) Certainly a head scratcher, strange that it "in some form or fashion" compiles ok because it definitely works fine the first time it's called... It just seems unhappy with the COPRAT every time after that...

drvinceknight commented 7 years ago

And even stranger that it (seems) to just be k61r: there are various other functions that define variables etc...

marcharper commented 7 years ago

Are some of the variables retaining state somehow? I.e. there's no initialization of ICOOP, so perhaps setting it to zero (it's just the cooperation counter) on reset or when ITURN = 1 will fix the issue.

marcharper commented 7 years ago

(and perhaps older Fortran compilers would zero initialize a variable in a case like this?)

meatballs commented 7 years ago

They all retain state by default

On 8 October 2017 20:00:12 BST, Marc Harper notifications@github.com wrote:

Are some of the variables retaining state somehow? I.e. there's no initialization of ICOOP, so perhaps setting it to zero (it's just the cooperation counter) on reset or when ITURN = 1 will fix the issue.

-- Owen Campbell

This email has been digitally signed. For further information visit http://owencampbell.me.uk/pgp

marcharper commented 7 years ago

So can we modify it as follows (add this line: IF (ITURN .EQ. 1) ICOOP = 0):

      FUNCTION K61R(ISPICK,ITURN,K,L,R, JA)
C BY DANNY C. CHAMPION
C TYPED BY JM 3/27/79
      k61r=ja    ! Added 7/27/93 to report own old value
      IF (ITURN .EQ. 1) ICOOP = 0
      IF (ITURN .EQ. 1) K61R = 0
      IF (ISPICK .EQ. 0) ICOOP = ICOOP + 1
      IF (ITURN .LE. 10) RETURN
      K61R = ISPICK
      IF (ITURN .LE. 25) RETURN
      K61R = 0
      COPRAT = FLOAT(ICOOP) / FLOAT(ITURN)
      IF (ISPICK .EQ. 1 .AND. COPRAT .LT. .6 .AND. R .GT. COPRAT)
     +K61R = 1
      RETURN
END
drvinceknight commented 7 years ago

So can we modify it as follows (add this line: IF (ITURN .EQ. 1) ICOOP = 0):

Already attempted that and it didn't change anything.

drvinceknight commented 7 years ago

Already attempted that and it didn't change anything.

Hold on I lied! I tried something else! Checking now!

drvinceknight commented 7 years ago

Yup: great shout @marcharper,(apologies, I had tried IF (ITURN .EQ. 1) COPRAT = 0).

What should we do in terms of fixing this:

Given the level of magic required I'd suggest fixing TourExec and just clearly documenting this (just as we have for a couple of other things).

marcharper commented 7 years ago

Given the level of magic required I'd suggest fixing TourExec and just clearly documenting this (just as we have for a couple of other things).

That's my vote since it seems like we'd have to mess with the memory address, but maybe @meatballs has another idea.

drvinceknight commented 7 years ago

That's my vote since it seems like we'd have to mess with the memory address, but maybe @meatballs has another idea.

I'll open a couple of PRs, 1 with the fix on TourExec and another with a test that catches this here. Feel free to close @meatballs if you have another idea :+1:

drvinceknight commented 7 years ago

I've opened: