fanntom commented 6 years ago

Hi I was trying out the DeepExploit.py and I ran into some errors that I was able to solve and some errors I couldn't. Regarding the issue before this, I found out that you should use 'python3-pip' and use 'pip3 install libraryname' in order to run DeepExploit.py properly. And you need to type python3 DeepExploit.py in order to run the python file with python3(Kali most up to date version). Also I found a typo in DeepExploit.py which causes a value error. In line 1282 or 1288, there is a period(.) after exploit_tree.json (so it would be 'exploit_tree.json.' instead of 'exploit_tree.json' which causes a value error when running in the training mode.

Now the errors that I couldn't fix:

When I run a training against a metasploitable2 machine, during the training session, there is a KeyError in 'windows/scada/advantech_webaccess_webvrpcs_bof', 'linux/telnet/netgear_telnetenable', 'multi/mysql/mysql_udf_payload' which occured 2 times, and while the finish train and save learned data sequence, there is a KeyError in 'multi/mysql/mysql_udf_payload'.
now with that trained data, if i run DeepExploit in test mode, after: [+]Executing start: local_thread1 Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "DeepExploit.py", line 1613, in job = lambda: worker.run(exploit_tree, target_tree) File "DeepExploit.py", line 1176, in run self.environment.run(exploit_tree, target_tree) File "DeepExploit.py", line 1042, in run target_list = exploit_tree[exploit[8:]]['target_list'] KeyError: 'linux/telnet/netgear_telnetenable' this error occurs.

13o-bbr-bbq commented 6 years ago

Hi, @fanntom Maybe, this causes are no corresponding exploit modules in the "exploit_list.csv". So, you delete all file in the DeepExploit/data/ folder, please retry DeepExploit.

Then, DeepExploit creates these files according to the your Metasploit's environment.

fanntom commented 6 years ago

Thank you @13o-bbr-bbq for the reply. I will try that out and I will comment on this thread about the results.

fanntom commented 6 years ago

Alright here's a followup about the result. The solution you mentioned did solve the problem. Thank you very much! Now I'm looking at a Exception in thread Thread-19: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, *self._kwargs) File "DeepExploit.py", line 1604, in job = lambda: worker.run(exploit_tree, target_tree, saver, env.save_file) File "DeepExploit.py", line 1161, in run self.environment.run(exploit_tree, target_tree) File "DeepExploit.py", line 1101, in run frames) File "DeepExploit.py", line 678, in execute_exploit job_id_list = self.client.get_job_list() File "DeepExploit.py", line 150, in get_job_list jobs = self.call('job.list', []) File "DeepExploit.py", line 65, in call resp = self.client.getresponse() File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse response.begin() File "/usr/lib/python3.6/http/client.py", line 297, in begin version, status, reason = self._read_status() File "/usr/lib/python3.6/http/client.py", line 266, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response but i think this is a problem with the msfrpc end. And I don't know if this is an error but im seeing some [] MsfRPC: Not Authenticated during the saving train data part.

fanntom commented 6 years ago

And when i run it in the test mode, I get this error:

root@kali:~/machine_learning_security/DeepExploit# python3 DeepExploit.py -t 192.168.74.135 -m test Using TensorFlow backend.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 　　　　　██████╗ ███████╗███████╗██████╗

　　　　　██╔══██╗██╔════╝██╔════╝██╔══██╗
　　　　　██║ ██║█████╗ █████╗ ██████╔╝
　　　　　██║ ██║██╔══╝ ██╔══╝ ██╔═══╝
　　　　　██████╔╝███████╗███████╗██║
　　　　　╚═════╝ ╚══════╝╚══════╝╚═╝

 ███████╗██╗  ██╗██████╗ ██╗      ██████╗ ██╗████████╗
 ██╔════╝╚██╗██╔╝██╔══██╗██║     ██╔═══██╗██║╚══██╔══╝
 █████╗   ╚███╔╝ ██████╔╝██║     ██║   ██║██║   ██║   
 ██╔══╝   ██╔██╗ ██╔═══╝ ██║     ██║   ██║██║   ██║   
 ███████╗██╔╝ ██╗██║     ███████╗╚██████╔╝██║   ██║   
 ╚══════╝╚═╝  ╚═╝╚═╝     ╚══════╝ ╚═════╝ ╚═╝   ╚═╝    (beta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Traceback (most recent call last): File "DeepExploit.py", line 1532, in env = Metasploit(rhost) File "DeepExploit.py", line 270, in init self.set_state_os() File "DeepExploit.py", line 310, in set_state_os os_raw = ret.get(b'data').decode('utf-8') AttributeError: 'NoneType' object has no attribute 'decode'

fanntom commented 6 years ago

In the msfconsole window i get this:

<Thread:0x00007f6d5945ac40@/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:93 run> terminated with exception (report_on_exception is true):

Traceback (most recent call last): 22: from /usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:111:in block in spawn' 21: from /usr/share/metasploit-framework/lib/msf/core/db_manager/connection.rb:6:inactive' 20: from /usr/share/metasploit-framework/lib/msf/core/db_manager/connection.rb:123:in connection_established?' 19: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:inwith_connection' 18: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in connection' 17: from /usr/lib/ruby/2.5.0/monitor.rb:226:inmon_synchronize' 16: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in block in connection' 15: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:incheckout' 14: from /usr/lib/ruby/2.5.0/monitor.rb:226:in mon_synchronize' 13: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:inblock in checkout' 12: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:424:in acquire_connection' 11: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:397:inreap' 10: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:397:in each' 9: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:398:inblock in reap' 8: from /usr/lib/ruby/2.5.0/monitor.rb:226:in mon_synchronize' 7: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:400:inblock (2 levels) in reap' 6: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:281:in reset!' 5: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:258:inclear_cache!' 4: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:197:in clear' 3: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:197:ineach_value' 2: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:198:in block in clear' 1: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:215:indealloc' /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:215:in `exec': ERROR: prepared statement "a9" does not exist (PG::InvalidSqlStatementName)

fanntom commented 6 years ago

ah nevermind the AttributeError. It was the problem with the msfrpc, not DeepExploit xD

pieterhouwen commented 6 years ago

Hi Fanntom, how exactly did you fix the msfrpc errors? I'm getting MFGRCP Not Authenticated though the msfrpc is started with the same values as in config.ini

fanntom commented 6 years ago

@pieterhouwen in which phase does that error popup?

pieterhouwen commented 6 years ago

In the training phase when it's saving to the database (at the end)

fanntom commented 6 years ago

@pieterhouwen hmmm i get that error as well. I'll look into the issue myself. @13o-bbr-bbq can you check if the MsfRCP error occurs in your environment as well please?

13o-bbr-bbq commented 6 years ago

@fanntom @pieterhouwen I'm getting MSFRPC Not Authenticated In the training phase when it's saving to the database (at the end) It's error message is also displayed in my environment. But, the trained data is saved normally. Was the trained data saved normally in your environment?

It is a bug that "MSFRPC Not Authenticated" is output, so I will fix it.

pieterhouwen commented 6 years ago

Yup, it saved the data alright, but now I'm getting other errors:

[] Restore learned data. [+] Executing start: local_thread1 Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(self._args, **self._kwargs) File "DeepExploit.py", line 1613, in job = lambda: worker.run(exploit_tree, target_tree) File "DeepExploit.py", line 1176, in run self.environment.run(exploit_tree, target_tree) File "DeepExploit.py", line 1042, in run target_list = exploit_tree[exploit[8:]]['target_list'] KeyError: 'windows/scada/advantech_webaccess_webvrpcs_bof'

Are these exploit-specific errors? The msfconsole window shows nothing

13o-bbr-bbq commented 6 years ago

Hi, @pieterhouwen Maybe, this causes are no corresponding exploit modules in the "exploit_list.csv". So, you delete all file in the DeepExploit/data/ folder, please retry DeepExploit.

Then, DeepExploit creates these files according to the your Metasploit's environment.

pieterhouwen commented 6 years ago

That fixed it, but when using the test option I get the following output on the msfconsole side: (I trained and tested twice):

Traceback (most recent call last): 28: from /usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in block in spawn' 27: from /usr/share/metasploit-framework/lib/rex/thread_factory.rb:22:inblock in spawn' 26: from /usr/share/metasploit-framework/lib/rex/job.rb:40:in block in start' 25: from /usr/share/metasploit-framework/lib/rex/job.rb:39:inensure in block in start' 24: from /usr/share/metasploit-framework/lib/msf/core/exploit_driver.rb:153:in block in run' 23: from /usr/share/metasploit-framework/lib/msf/core/exploit_driver.rb:249:injob_cleanup_proc' 22: from /usr/share/metasploit-framework/lib/msf/core/event_dispatcher.rb:186:in method_missing' 21: from /usr/share/metasploit-framework/lib/msf/core/event_dispatcher.rb:186:ineach' 20: from /usr/share/metasploit-framework/lib/msf/core/event_dispatcher.rb:188:in block in method_missing' 19: from /usr/share/metasploit-framework/lib/msf/core/framework.rb:332:inon_module_complete' 18: from /usr/share/metasploit-framework/lib/msf/core/framework.rb:309:in module_event' 17: from /usr/share/metasploit-framework/lib/msf/core/module.rb:221:inworkspace' 16: from /usr/share/metasploit-framework/lib/msf/core/db_manager/workspace.rb:24:in workspace' 15: from /usr/share/metasploit-framework/lib/msf/core/db_manager/workspace.rb:18:infind_workspace' 14: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in with_connection' 13: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:inconnection' 12: from /usr/lib/ruby/2.5.0/monitor.rb:226:in mon_synchronize' 11: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:inblock in connection' 10: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in checkout' 9: from /usr/lib/ruby/2.5.0/monitor.rb:226:inmon_synchronize' 8: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in block in checkout' 7: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:425:inacquire_connection' 6: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:133:in poll' 5: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:145:insynchronize' 4: from /usr/lib/ruby/2.5.0/monitor.rb:226:in mon_synchronize' 3: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:135:inblock in poll' 2: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:180:in wait_poll' 1: from /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:180:inloop' /usr/share/metasploit-framework/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:189:in `block in wait_poll': could not obtain a database connection within 5.000 seconds (waited 5.900 seconds) (ActiveRecord::ConnectionTimeoutError)

13o-bbr-bbq commented 6 years ago

Hi, @pieterhouwen Maybe, it is connection error between client and Metasploit. Could you check your "config.ini"?

config.ini

[Common]
server_host : "Your IP address of Metasploit"
server_port : "Your port number of Metasploit"
...snip...
[Metasploit]
lhost         : "Your IP address of Metasploit"

fanntom commented 6 years ago

btw should I create a requirements.txt for the python3 libraries?

13o-bbr-bbq commented 6 years ago

Hi, @fanntom Thanks for advice. I'll create the requirements.txt soon.

13o-bbr-bbq commented 6 years ago

I've created the requirement.txt. Please, try it.

13o-bbr-bbq / machine_learning_security

Multiple Errors in DeepExploit.py #5

<Thread:0x00007f6d5945ac40@/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:93 run> terminated with exception (report_on_exception is true):