search

13o-bbr-bbq / machine_learning_security

Source code about machine learning and security.
1.96k stars 648 forks source link

Multiple errors / Fixes #9

Closed someusername123 closed 6 years ago

someusername123 commented 6 years ago

To start off I'm writing this using a fresh install of Kali 2018 4.15.0-kali3-amd64.

After running your install script there is a few workarounds ( I cant remember specifically the issues I had installing to start but I had to do some removal dist-packages that came with Kali using rm -rf and re running pip install -r requirements.txt -I).

Past that the first thing I get when running python DeepExploit.py -t xxx.xxx.xxx.xxx -m train I get "Invalid IP address" which is caused by ipaddress.ip_address(arg) -> ipaddress.ip_address expects a unicode IP see below

ipaddress.ip_address('0.0.0.0') Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.7/dist-packages/ipaddress.py", line 163, in ip_address ' a unicode object?' % address) ipaddress.AddressValueError: '0.0.0.0' does not appear to be an IPv4 or IPv6 address. Did you pass in a bytes (str in Python 2) instead of a unicode object? ipaddress.ip_address(u'0.0.0.0') IPv4Address(u'0.0.0.0') Which can be fixed by converting the rhost to a unicode variable. I changed rhost, mode, port, service = command_parse() to rhost, mode, port, service = command_parse() rhost = unicode(rhost)

When finding a successful exploit a UnicodeDecodeError is thrown from show_banner_bingo which can be resolved by changing
banner = """ to banner = u"""

When using test mode ( python Deepexploit.py -t xx.xx.xx.xx -m test) this error is thrown. I don't have a fix on this one as I do not understand completely the code that is being used for the tensorflow learning algorithms, however it appears that the apply_gradients call is throwing it.

WARNING:tensorflow:From DeepExploit.py:1321: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead Traceback (most recent call last): File "DeepExploit.py", line 1819, in rhost=rhost)) File "DeepExploit.py", line 1626, in init self.environment = Environment(thread_name, thread_type, parameter_server, rhost) File "DeepExploit.py", line 1470, in init self.agent = Agent(name, parameter_server) File "DeepExploit.py", line 1396, in init self.brain = LocalBrain(name, parameter_server) File "DeepExploit.py", line 1296, in init self._build_graph(name, parameter_server) File "DeepExploit.py", line 1337, in _build_graph parameter_server.optimizer.apply_gradients(zip(self.grads, parameter_server.weights_params)) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 598, in applygradients ([str(v) for , _, v in converted_grads_and_vars],)) ValueError: No gradients provided for any variable: ["<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_1/kernel:0' shape=(7, 50) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_1/bias:0' shape=(50,) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_2/kernel:0' shape=(50, 100) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_2/bias:0' shape=(100,) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_3/kernel:0' shape=(100, 200) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_3/bias:0' shape=(200,) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_4/kernel:0' shape=(200, 538) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_4/bias:0' shape=(538,) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_5/kernel:0' shape=(200, 1) dtype=float32_ref>)>", "<_RefVariableProcessor(<tf.Variable 'parameter_server/dense_5/bias:0' shape=(1,) dtype=float32_ref>)>"].

The last thing is that I've always had errors when using the data folder that it came with. You should probably just ship the data folder empty by default and let it enumerate the exploits through Metasploit since your code already does that.

13o-bbr-bbq commented 6 years ago

Hi @someusername123 thanks for advise. maybe, you use the python2, but deep exploit doesn't support the python2. we've checked that Deep Exploit runs on the python3. could you try again using python3 (recommend python3.6)?

someusername123 commented 6 years ago

python3 gives this output ( end result is a illegal instruction code crash)

root@UnknownK:~/shit# git clone https://github.com/13o-bbr-bbq/machine_learning_security.git Cloning into 'machine_learning_security'... remote: Counting objects: 800, done. remote: Total 800 (delta 0), reused 0 (delta 0), pack-reused 800 Receiving objects: 100% (800/800), 12.59 MiB | 5.04 MiB/s, done. Resolving deltas: 100% (479/479), done. root@UnknownK:~/shit/machine_learning_security/DeepExploit# pip3 install -r requirements.txt -I Collecting beautifulsoup4==4.6.0 (from -r requirements.txt (line 1)) Downloading https://files.pythonhosted.org/packages/9e/d4/10f46e5cfac773e22707237bfcd51bbffeaf0a576b0a847ec7ab15bd7ace/beautifulsoup4-4.6.0-py3-none-any.whl (86kB) 100% |████████████████████████████████| 92kB 296kB/s Collecting docopt==0.6.2 (from -r requirements.txt (line 2)) Collecting Jinja2==2.10 (from -r requirements.txt (line 3)) Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl Collecting Keras==2.1.6 (from -r requirements.txt (line 4)) Using cached https://files.pythonhosted.org/packages/54/e8/eaff7a09349ae9bd40d3ebaf028b49f5e2392c771f294910f75bb608b241/Keras-2.1.6-py2.py3-none-any.whl Collecting msgpack-python==0.5.6 (from -r requirements.txt (line 5)) Collecting numpy==1.13.3 (from -r requirements.txt (line 6)) Using cached https://files.pythonhosted.org/packages/57/a7/e3e6bd9d595125e1abbe162e323fd2d06f6f6683185294b79cd2cdb190d5/numpy-1.13.3-cp36-cp36m-manylinux1_x86_64.whl Collecting pandas==0.23.0 (from -r requirements.txt (line 7)) Using cached https://files.pythonhosted.org/packages/69/ec/8ff0800b8594691759b78a42ccd616f81e7099ee47b167eb9bbd502c02b9/pandas-0.23.0-cp36-cp36m-manylinux1_x86_64.whl Collecting tensorflow==1.8.0 (from -r requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/22/c6/d08f7c549330c2acc1b18b5c1f0f8d9d2af92f54d56861f331f372731671/tensorflow-1.8.0-cp36-cp36m-manylinux1_x86_64.whl Collecting MarkupSafe>=0.23 (from Jinja2==2.10->-r requirements.txt (line 3)) Collecting six>=1.9.0 (from Keras==2.1.6->-r requirements.txt (line 4)) Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl Collecting h5py (from Keras==2.1.6->-r requirements.txt (line 4)) Downloading https://files.pythonhosted.org/packages/8e/cb/726134109e7bd71d98d1fcc717ffe051767aac42ede0e7326fd1787e5d64/h5py-2.8.0-cp36-cp36m-manylinux1_x86_64.whl (2.8MB) 100% |████████████████████████████████| 2.8MB 2.9MB/s Collecting pyyaml (from Keras==2.1.6->-r requirements.txt (line 4)) Collecting scipy>=0.14 (from Keras==2.1.6->-r requirements.txt (line 4)) Using cached https://files.pythonhosted.org/packages/a8/0b/f163da98d3a01b3e0ef1cab8dd2123c34aee2bafbb1c5bffa354cc8a1730/scipy-1.1.0-cp36-cp36m-manylinux1_x86_64.whl Collecting pytz>=2011k (from pandas==0.23.0->-r requirements.txt (line 7)) Using cached https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl Collecting python-dateutil>=2.5.0 (from pandas==0.23.0->-r requirements.txt (line 7)) Using cached https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl Collecting tensorboard<1.9.0,>=1.8.0 (from tensorflow==1.8.0->-r requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/59/a6/0ae6092b7542cfedba6b2a1c9b8dceaf278238c39484f3ba03b03f07803c/tensorboard-1.8.0-py3-none-any.whl Collecting grpcio>=1.8.6 (from tensorflow==1.8.0->-r requirements.txt (line 8)) Downloading https://files.pythonhosted.org/packages/1f/ea/664c589ec41b9e9ac6e20cc1fe9016f3913332d0dc5498a5d7771e2835af/grpcio-1.12.1-cp36-cp36m-manylinux1_x86_64.whl (9.0MB) 100% |████████████████████████████████| 9.0MB 1.4MB/s Collecting protobuf>=3.4.0 (from tensorflow==1.8.0->-r requirements.txt (line 8)) Downloading https://files.pythonhosted.org/packages/fc/f0/db040681187496d10ac50ad167a8fd5f953d115b16a7085e19193a6abfd2/protobuf-3.6.0-cp36-cp36m-manylinux1_x86_64.whl (7.1MB) 100% |████████████████████████████████| 7.1MB 2.4MB/s Collecting gast>=0.2.0 (from tensorflow==1.8.0->-r requirements.txt (line 8)) Collecting absl-py>=0.1.6 (from tensorflow==1.8.0->-r requirements.txt (line 8)) Collecting wheel>=0.26 (from tensorflow==1.8.0->-r requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl Collecting astor>=0.6.0 (from tensorflow==1.8.0->-r requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/b2/91/cc9805f1ff7b49f620136b3a7ca26f6a1be2ed424606804b0fbcf499f712/astor-0.6.2-py2.py3-none-any.whl Collecting termcolor>=1.1.0 (from tensorflow==1.8.0->-r requirements.txt (line 8)) Collecting werkzeug>=0.11.10 (from tensorboard<1.9.0,>=1.8.0->tensorflow==1.8.0->-r requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl Collecting bleach==1.5.0 (from tensorboard<1.9.0,>=1.8.0->tensorflow==1.8.0->-r requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/33/70/86c5fec937ea4964184d4d6c4f0b9551564f821e1c3575907639036d9b90/bleach-1.5.0-py2.py3-none-any.whl Collecting html5lib==0.9999999 (from tensorboard<1.9.0,>=1.8.0->tensorflow==1.8.0->-r requirements.txt (line 8)) Collecting markdown>=2.6.8 (from tensorboard<1.9.0,>=1.8.0->tensorflow==1.8.0->-r requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/6d/7d/488b90f470b96531a3f5788cf12a93332f543dbab13c423a5e7ce96a0493/Markdown-2.6.11-py2.py3-none-any.whl Collecting setuptools (from protobuf>=3.4.0->tensorflow==1.8.0->-r requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/7f/e1/820d941153923aac1d49d7fc37e17b6e73bfbd2904959fffbad77900cf92/setuptools-39.2.0-py2.py3-none-any.whl Installing collected packages: beautifulsoup4, docopt, MarkupSafe, Jinja2, six, numpy, h5py, pyyaml, scipy, Keras, msgpack-python, pytz, python-dateutil, pandas, werkzeug, setuptools, protobuf, html5lib, bleach, markdown, wheel, tensorboard, grpcio, gast, absl-py, astor, termcolor, tensorflow Successfully installed Jinja2-2.10 Keras-2.1.6 MarkupSafe-1.0 absl-py-0.2.2 astor-0.6.2 beautifulsoup4-4.6.0 bleach-1.5.0 docopt-0.6.2 gast-0.2.0 grpcio-1.12.1 h5py-2.8.0 html5lib-0.9999999 markdown-2.6.11 msgpack-python-0.5.6 numpy-1.14.3 pandas-0.23.0 protobuf-3.6.0 python-dateutil-2.7.3 pytz-2018.4 pyyaml-3.12 scipy-1.1.0 setuptools-39.2.0 six-1.11.0 tensorboard-1.8.0 tensorflow-1.8.0 termcolor-1.1.0 werkzeug-0.14.1 wheel-0.31.1 root@UnknownK:~/shit/machine_learning_security/DeepExploit# gedit ~/.keras/keras.json

modified json file to what was stated in the installation documentation here

root@UnknownK:~/shit/machine_learning_security/DeepExploit# python3 DeepExploit.py Using TensorFlow backend. Illegal instruction root@UnknownK:~/shit/machine_learning_security/DeepExploit# python3 DeepExploit.py -t 127.0.0.1 -m train Using TensorFlow backend. Illegal instruction

someusername123 commented 6 years ago

Python3.6 gives the same illegal instruction throw

13o-bbr-bbq commented 6 years ago

maybe, it is problem of tensorflow version.
https://github.com/tensorflow/tensorflow/issues/17411

please, try tensorflow 1.5. 

> pip uninstall tensorflow
> pip install tensorflow==1.5
someusername123 commented 6 years ago

I actually moved to a Kali 2018 light version and it seems to work fine for the most part, I get errors when it is attempting to exploit. See below File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "DeepExploit.py", line 2033, in job = lambda: worker.run(exploit_tree, target_tree, saver, env.save_file) File "DeepExploit.py", line 1827, in run self.environment.run(exploit_tree, target_tree) File "DeepExploit.py", line 1733, in run target_tree) File "DeepExploit.py", line 953, in reset_state service_name = target_tree[port_num]['prod_name'] TypeError: string indices must be integers

I'm using python3.6 -t xx.xx.xx.xx -m train

13o-bbr-bbq commented 6 years ago

thanks for you retry.
i repaired above problems, please try again.

someusername123 commented 6 years ago

Retried with same command line as before, getting an error on enumerating the exploits now, see below [*] 139/1161 exploit:linux/imap/imap_uw_lsub, targets:1 Traceback (most recent call last): File "DeepExploit.py", line 1970, in exploit_tree = env.get_exploit_tree() File "DeepExploit.py", line 409, in get_exploit_tree payload_list = self.client.get_target_compatible_payload_list(exploit, int(target)) File "DeepExploit.py", line 178, in get_target_compatible_payload_list ret = self.call('module.target_compatible_payloads', [module_name, target_num]) File "DeepExploit.py", line 102, in call resp = self.client.getresponse() File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse response.begin() File "/usr/lib/python3.6/http/client.py", line 297, in begin version, status, reason = self._read_status() File "/usr/lib/python3.6/http/client.py", line 266, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response

13o-bbr-bbq commented 6 years ago

i added the function of re-connection. please, try again.