Simulation is not able to call reputation service

[kabirkbr]

I was following https://github.com/singnet/reputation-simulation/blob/master/README.md and ran into some problems.

2) sudo bash scripts/install (otherwise not enough permissions) finishes with an error, which, to my mind, leaves 'reputation' package not installed.

Using /usr/local/lib/python3.6/dist-packages
Finished processing dependencies for reputation-simulation==0.0.1
+ rm -rf src
+ pip3 install -e git+https://github.com/singnet/reputation.git#egg=reputation
Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
ImportError: cannot import name 'main'

3) Running python3 reputation-simulation/snsim/reputation_simulation/ReputationSim.py reputation-simulation/unratedDenomConserv5SAP10.json fails, because there is no file unratedDenomConserv5SAP10.json in the repo and the directory structure seem to be different. Following another readme in reputation-simulation/, I run it with another file:

python3 reputation-simulation/snsim/reputation_simulation/ReputationSim.py reputation-simulation/snsim/noUnratednoDenomConserv5SAP182.json

This one fails, because it tries to call reputation server on localtest.com:1181 which obviously is not yet running:

request http://localtest.com:1181/?my+name+john%40doe.org%2C+surname+john%40doe.org%2C+email+john%40doe.org. <class 'requests.exceptions.ConnectionError'>
Specify proper url to Aigents server or run it locally, eg.
java -cp ./bin/Aigents.jar:./bin/* net.webstructor.agent.Farm store path './al_test.txt', cookie domain localtest.com, console off &
or
sh aigents_server_start.sh
Traceback (most recent call last):
  File "reputation-simulation/snsim/reputation_simulation/ReputationSim.py", line 646, in <module>
    main()
  File "reputation-simulation/snsim/reputation_simulation/ReputationSim.py", line 617, in main
    'john@doe.org','q', 'a', False, dirname, True))
  File "/home/kabir/reputation/reputation/reputation/aigents_reputation_api.py", line 47, in __init__
    self.create_session()
  File "/home/kabir/reputation/reputation/reputation/aigents_reputation_api.py", line 65, in create_session
    assert output == 'What your secret question, secret answer?', 'Expecting secret question, secret answer'
AssertionError: Expecting secret question, secret answer
request http://localtest.com:1181/?Your+trusts+no+john%40doe.org. <class 'requests.exceptions.ConnectionError'>
Specify proper url to Aigents server or run it locally, eg.
java -cp ./bin/Aigents.jar:./bin/* net.webstructor.agent.Farm store path './al_test.txt', cookie domain localtest.com, console off &
or
sh aigents_server_start.sh
Exception ignored in: <bound method AigentsAPIReputationService.__del__ of <reputation.aigents_reputation_api.AigentsAPIReputationService object at 0x7f9c74853ba8>>
Traceback (most recent call last):
  File "/home/kabir/reputation/reputation/reputation/aigents_reputation_api.py", line 53, in __del__
    self.close_session()
  File "/home/kabir/reputation/reputation/reputation/aigents_reputation_api.py", line 74, in close_session
    assert output.split()[0] == 'Ok.', 'Expecting Ok'
AssertionError: Expecting Ok

This is because parameters.use_java is set to true by default; so i set it to false and try again, hoping that it will call the python reputation service which throws the following error:

Traceback (most recent call last):
  File "reputation-simulation/snsim/reputation_simulation/ReputationSim.py", line 646, in <module>
    main()
  File "reputation-simulation/snsim/reputation_simulation/ReputationSim.py", line 615, in main
    ) if not config['parameters']['use_java']else AigentsAPIReputationService(
NameError: name 'PythonReputationService' is not defined

This seem to be related to commented out import statement on line 20 in the reputation-simulation/snsim/reputation_simulation/ReputationSim.py file. After uncommenting I get:

kabir@Ubuntu-1804-bionic-64-minimal:~/reputation$ python3 reputation-simulation/snsim/reputation_simulation/ReputationSim.py reputation-simulation/snsim/noUnratednoDenomConserv5SAP182.json
Traceback (most recent call last):
  File "reputation-simulation/snsim/reputation_simulation/ReputationSim.py", line 20, in <module>
    from reputation import PythonReputationService
ImportError: cannot import name 'PythonReputationService'

Now it seems we are in square zero as python reputation service cannot be found because the reputation package was not correctly installed in the first place...

Could you please guide me into direction of solving the above?

[akolonin]

@kabirkbr - @tiero is our magician of installations

[kabirkbr]

I have also tried to install and run the simulation with local aigents java server running on port 1180 -- cloned from https://github.com/singnet/reputation repo.

In this case I get the following:

kabir@Ubuntu-1804-bionic-64-minimal:~/reputation$ python3 reputation-simulation/snsim/reputation_simulation/ReputationSim.py reputation-simulation/snsim/noUnratednoDenomConserv5SAP182.json
/home/kabir/reputation
Traceback (most recent call last):
  File "reputation-simulation/snsim/reputation_simulation/ReputationSim.py", line 646, in <module>
    main()
  File "reputation-simulation/snsim/reputation_simulation/ReputationSim.py", line 634, in main
    'ratings': config['parameters']['reputation_parameters']['ratings'],
KeyError: 'ratings'

This seems to indicate that something is wrong with the parameters file -- as there are not 'ratings' parameter there. Can you point me to the correct parameter file?

[tiero]

Hey @kabirkbr.

About point number 2:

• Please, Can you post your environment to try to replicate it? Linux/Mac and version of both Python and Pip?
• You are using sudo probably because on your system your user has no access to write in /usr/local/lib/python3.6 where the packages goes. Try something like sudo chown -R $USER /usr/local/lib/python3.6 from your admin user and give to the $USER ownership of that folder, so you don't need sudo after that with non-root users.
• After quick googling your error seems something related to a bug that happens in combination with Ubuntu and pip10, so it's an environment issue I believe. Let's check it https://stackoverflow.com/questions/28210269/importerror-cannot-import-name-main-when-running-pip-version-command-in-windo

About the 3 and other points, I think it's all about application side, not about installation machinery so I cannot be helpful on that 👍

[kabirkbr]

Please, Can you post your environment to try to replicate it? Linux/Mac and version of both Python and Pip?

kabir@Ubuntu-1804-bionic-64-minimal:~$ uname -a
Linux Ubuntu-1804-bionic-64-minimal 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
kabir@Ubuntu-1804-bionic-64-minimal:~$ python3 -V
Python 3.6.6
kabir@Ubuntu-1804-bionic-64-minimal:~$ pip -V
pip 19.0.3 from /home/kabir/.local/lib/python3.6/site-packages/pip (python 3.6)
kabir@Ubuntu-1804-bionic-64-minimal:~$ pip3 -V
pip 19.0.3 from /home/kabir/.local/lib/python3.6/site-packages/pip (python 3.6)

Try something like sudo chown -R $USER /usr/local/lib/python3.6 from your admin user and give to the $USER ownership of that folder, so you don't need sudo after that with non-root users. This is what I get now -- which is not really a problem since I can install things with sudo --:

kabir@Ubuntu-1804-bionic-64-minimal:~/reputation/reputation-simulation$ bash scripts/install
++++ dirname scripts/install
+++ cd scripts
+++ pwd -P
++ dirname /home/kabir/reputation/reputation-simulation/scripts
+ PARENT_PATH=/home/kabir/reputation/reputation-simulation
+ pushd /home/kabir/reputation/reputation-simulation
~/reputation/reputation-simulation ~/reputation/reputation-simulation
+ python3 setup.py develop
running develop
running egg_info
error: [Errno 13] Permission denied

Applying https://stackoverflow.com/questions/28210269/importerror-cannot-import-name-main-when-running-pip-version-command-in-windo to pip and pip3 worked:

sudo bash scripts/install 
[...]
Installing collected packages: reputation
  Found existing installation: reputation 0.0.1
    Can't uninstall 'reputation'. No files were found to uninstall.
  Running setup.py develop for reputation
Successfully installed reputation
+ popd
~/reputation/reputation-simulation

So I suppose all is well on the installation side now... Thanks!

[kabirkbr]

Apart from the point 3 above, I get the following error when running the latest commit 3f2208e:

python3 snsim/reputation_simulation/ReputationSim.py snsim/noUnratednoDenomConserv5SAP182.json 
/home/kabir/reputation/reputation-simulation
Traceback (most recent call last):
  File "snsim/reputation_simulation/ReputationSim.py", line 698, in <module>
    main()
  File "snsim/reputation_simulation/ReputationSim.py", line 658, in main
    config = Adapters(config).translate()
  File "/home/kabir/reputation/reputation-simulation/snsim/reputation_simulation/Adapters.py", line 20, in translate
    return self.antons_view()
  File "/home/kabir/reputation/reputation-simulation/snsim/reputation_simulation/Adapters.py", line 63, in antons_view
    self.config['parameters']["scam_parameters"]["scam_period"] = sp
KeyError: 'scam_parameters'

Which again, in my opinion shows that the configuration files supplied with the repo are out of sync. @deborahduong , could you provide the newest configuration file?

Thanks!

[kabirkbr]

I have tested the fresh clone of the repository in virtual environment with python3.7.2.

1) First, installed and started aigents server from a fresh reputation repository clone after changing port to 1196 in the script. Seems to run well:

kabir@Ubuntu-1804-bionic-64-minimal:~/reputation/repsim3.7.2/reputation/reputation$ sh aigents_server_start.sh 
Aigents server started.

Java process running:

kabir@Ubuntu-1804-bionic-64-minimal:~/reputation/repsim3.7.2/reputation/reputation$ ps aux | grep Aigents
kabir    16317  0.0  0.0 5570436 56312 pts/2   Sl   12:45   0:03 java -cp ./bin/Aigents.jar:./bin/* -Xms1000m -Xmx2000m net.webstructor.agent.Farm store path ./al_test.txt, http port 1196, cookie domain localtest.com, console off
kabir    16633  0.0  0.0  17240  1040 pts/2    S+   14:17   0:00 grep --color=auto Aigents

Java process 16317 listens to 1123 and 1196 ports:

kabir@Ubuntu-1804-bionic-64-minimal:~/reputation/repsim3.7.2/reputation/reputation$ netstat -plnt
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 195.201.197.25:8609     0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:42597         0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:9042            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8787            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8182            0.0.0.0:*               LISTEN      -                   
tcp        0      0 195.201.197.25:7000     0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:7199          0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::1123                 :::*                    LISTEN      16317/java          
tcp6       0      0 127.0.0.1:40165         :::*                    LISTEN      -                   
tcp6       0      0 :::1196                 :::*                    LISTEN      16317/java         
tcp6       0      0 127.0.0.1:61619         :::*                    LISTEN      -                   
tcp6       0      0 :::61620                :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 :::8888                 :::*                    LISTEN      -      

But running simulation throws an error:

(repsim3.7.2) kabir@Ubuntu-1804-bionic-64-minimal:~/reputation/repsim3.7.2/reputation-simulation$ python snsim/reputation_simulation/ReputationSim.py snsim/s2RepHalf_debug100.json 
/home/kabir/reputation/repsim3.7.2/reputation-simulation
s2RepHalf_debug100/ : r_sp30_  port:1196 
s2RepHalf_debug100/ : r_sp30_
time 0
Traceback (most recent call last):
  File "snsim/reputation_simulation/ReputationSim.py", line 707, in <module>
    main()
  File "snsim/reputation_simulation/ReputationSim.py", line 699, in main
    call(config['batch']['parameter_combinations'], config,rs=rs)
  File "snsim/reputation_simulation/ReputationSim.py", line 641, in call
    call(mycombolist, myconfigfile, rs, my_param_str)
  File "snsim/reputation_simulation/ReputationSim.py", line 655, in call
    repsim.go()
  File "snsim/reputation_simulation/ReputationSim.py", line 600, in go
    self.step()
  File "snsim/reputation_simulation/ReputationSim.py", line 580, in step
    self.schedule.step()
  File "/home/kabir/reputation/repsim3.7.2/lib/python3.7/site-packages/Mesa-0.8.5-py3.7.egg/mesa/time.py", line 178, in step
    self.model.random.random.shuffle(agent_keys)
AttributeError: 'builtin_function_or_method' object has no attribute 'shuffle'
request http://localtest.com:1196/?Your+trusts+no+john%40doe.org. <class 'ImportError'>
Specify proper url to Aigents server or run it locally, eg.
java -cp ./bin/Aigents.jar:./bin/* net.webstructor.agent.Farm store path './al_test.txt', cookie domain localtest.com, console off &
or
sh aigents_server_start.sh
Exception ignored in: <function AigentsAPIReputationService.__del__ at 0x7f2771880730>
Traceback (most recent call last):
  File "/home/kabir/reputation/repsim3.7.2/reputation/reputation/aigents_reputation_api.py", line 53, in __del__
  File "/home/kabir/reputation/repsim3.7.2/reputation/reputation/aigents_reputation_api.py", line 75, in close_session
AssertionError: Expecting Ok

I cannot determine if this is the problem with the simulation or the installation -- since it seems to complain both about mesa and aigents server... but aigents server seems to be running well. Changing aigents server port to 1180, 1288 or any does not have effect.

Any ideas how to solve this?

[kabirkbr]

Btw, localtest.com resolves to 127.0.0.1 as required by aigents server installation:

kabir@Ubuntu-1804-bionic-64-minimal:~/reputation/repsim3.7.2/reputation/reputation$ ping localtest.com
PING localtest.com (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=2 ttl=64 time=0.058 ms
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=3 ttl=64 time=0.028 ms
^C
--- localtest.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2029ms
rtt min/avg/max/mdev = 0.028/0.043/0.058/0.012 ms

[deborahduong]

Hi, yes, this can be solved with my patch to mesa. In the setup of the program, my branch which patches this mesa error is installed instead of mesa. See https://github.com/singnet/reputation-simulation/blob/master/setup.py

[deborahduong]

actually, that patch should have installed automatically... @tiero do you know why it wouldn't have been?

[tiero]

@kabirkbr Are you sure the folder where you are running the simulation is the correct one?

Is this right? ~/reputation/repsim3.7.2/reputation-simulation ?

btw Try to force installing debora's fork of mesa pip install -e git+https://github.com/deborahduong/mesa.git#egg=mesa

(But I don't know if the error is actually due mesa package)

[kabirkbr]

Closing.

Just installed all from scratch, reran python snsim/reputation_simulation/ReputationSim.py snsim/s2RepHalf_debug100.json and it worked, both with force installing @deborahduong mesa and without. It also works both with 1288 and 1196 aigents server ports. So no idea what was causing above -- maybe something was changed in the code during the past week or so?

Thanks all.

[kabirkbr]

Just installed a python 3.7.2 virtual environment from scratch and was getting the same error. Yet the error went away after running pip install -e git+https://github.com/deborahduong/mesa.git#egg=mesa as @tiero recommended.

My guess is that the requirement for modified mesa does not get installed from setup.py script possibly because it does not override already existing mesa, while the above command does.

@deborahduong, I would suggest to put the full command in scripts/install rather than using setup_requires in setup.py...