Closed ccsplit closed 7 years ago
marco-lancini
commented
7 years ago Hi @ccsplit, the recursion error has been fixed with the latest commit in develop, but it was related to a faulty decorator.
Regarding the bad file descriptor issue, what version of the agent are you running?
ccsplit
commented
7 years ago Hello @marco-lancini, I am using Needle agent v1.0.3 and have tested with the needle release 1.1.0 and also master. I will test with the develop branch to see if the Bad file descriptor issue is still present.
Edit: The develop branch after removing the cli.Mode.Console no longer has the recursive issue but I am still getting the Bad file descriptor issue. I would also suggest adding the inner exception to the Exception message or having a debug print statement prior for debugging the inner issue as to why the connection failed. When I get a chance I will see if I can resolve the issue or create a scenario/test case that causes the issue.
marco-lancini
commented
7 years ago Hi @ccsplit, are you still experiencing this issue even if using the latest version from the develop branch?
ccsplit
commented
7 years ago @marco-lancini, It still occurs but is less frequent than it was before. Usually I can now restart the Needle-agent, or reconnect the iOS device and the issue will resolve. There have also been times where it has resolved itself when re-attempting a connection. If you'd like you can close it, and whenever I get a chance to take a look at it further I will try to create a test and a fix for the issue.
chengteck
commented
7 years ago Hi guys. Is there any update on the fix for [Errno 9] Bad file descriptor with Python 2.7.13? I have issues running modules and I got "[Errno 9] Bad file descriptor" error for every module I run.
marco-lancini
commented
7 years ago Hi @chengteck, @ccsplit: I currently can't reproduce the issue at all :S Are you able to provide me with more info? (anything could be useful now)
chengteck
commented
7 years ago Hi @marco-lancini Thanks a lot for assisting.
I am using Needle Agent v.1.0.4 and Needle v1.2.0.
I have attached a series of screenshots for your reference.
Hope you are able to reproduce this issue.
Are you using Python 2.7.13 as well?
Thanks alot for helping!
marco-lancini
commented
7 years ago what happens if you:
use device/agent_client
set COMMAND LIST_APPS
run(yes, I'm on the same version of python)
chengteck
commented
7 years ago Hi there.
I got the same "[Errno 9] Bad file descriptor" error and my NeedleAgent crashed.
May I know what iOS version are you running on? I am running on iOS 9.0.2 and running needle in a kali VM.
I have attached the screenshots for your reference.
marco-lancini
commented
7 years ago are you able to provide the full list of apps installed on your device?
any app installed manually by putting the .app in the /Applications folder?
nvisium-john-poulin
commented
7 years ago I was having similar issues and got it to work by removing several unnecessary applications and rebooting the agent. Not sure why/how this fixed the issue. If you guys haven't tried rebooting the agent yet give that a go.
floyd-fuh
commented
7 years ago Just wanted to leave a "me too" here:
[needle] > set DEBUG True
DEBUG => True
[needle] > shell
[*] Spawning a shell...
[*] Checking connection with device...
[V] Connection not present, creating a new instance
[V] [AGENT] Connecting to agent (10.211.0.147:4444)...
[+] [AGENT] Successfully connected to agent (10.211.0.147:4444)...
[D] [AGENT] Executing command: os_version
[D] [AGENT] Parsing result (are you sure the agent is in the foreground?)
[!] [Errno 9] Bad file descriptor
[V] [SSH] Disconnecting...
[V] [AGENT] Disconnecting from agent...
[?] Resetting connection to device...
[V] [AGENT] Connecting to agent (10.211.0.147:4444)...
[+] [AGENT] Successfully connected to agent (10.211.0.147:4444)...
[D] [AGENT] Executing command: os_version
[D] [AGENT] Parsing result (are you sure the agent is in the foreground?)
[!] [Errno 9] Bad file descriptor
[V] [SSH] Disconnecting...
[V] [AGENT] Disconnecting from agent...
[?] Resetting connection to device...
[V] [AGENT] Connecting to agent (10.211.0.147:4444)...
[+] [AGENT] Successfully connected to agent (10.211.0.147:4444)...
[D] [AGENT] Executing command: os_version
[D] [AGENT] Parsing result (are you sure the agent is in the foreground?)
[!] [Errno 9] Bad file descriptor
[V] [SSH] Disconnecting...
[V] [AGENT] Disconnecting from agent...
[?] Resetting connection to device...
[V] [AGENT] Connecting to agent (10.211.0.147:4444)...
[+] [AGENT] Successfully connected to agent (10.211.0.147:4444)...
[!] Problem establishing connection: Exception - An error occurred and it was not possible to restore it (3 attempts failed)
------------------------------------------------------------
Traceback (most recent call last):
File "/opt/needle/needle/core/framework/framework.py", line 641, in _connection_new
self.device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 203, in wrapper
device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 203, in wrapper
device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 203, in wrapper
device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 207, in wrapper
raise Exception("An error occurred and it was not possible to restore it ({} attempts failed)".format(self.tries))
Exception: An error occurred and it was not possible to restore it (3 attempts failed)
------------------------------------------------------------
[!] Exception: An error occurred and it was not possible to restore it (3 attempts failed)
[V] [SSH] Disconnecting...
[V] [AGENT] Disconnecting from agent...
[needle] > show options
Name Current Value Required Description
------------------------ ------------- -------- -----------
AGENT_PORT 4444 yes Port on which the Needle Agent is listening
APP no Bundle ID of the target application (e.g., com.example.app). Leave empty to launch wizard
DEBUG True yes Enable debugging output
HIDE_SYSTEM_APPS False yes If set to True, only 3rd party apps will be shown
IP 10.211.0.147 yes IP address of the testing device (set to localhost to use USB)
OUTPUT_FOLDER /Users/user/.needle/output yes Full path of the output folder, where to store the output of the modules
PASSWORD ******** yes SSH Password of the testing device
PORT 22 yes Port of the SSH agent on the testing device (needs to be != 22 to use USB)
PUB_KEY_AUTH True yes Use public key auth to authenticate to the device. Key must be present in the ssh-agent if a passphrase is used
SAVE_HISTORY True yes Persists command history across sessions
SKIP_OUTPUT_FOLDER_CHECK False no Skip the check that ensures the output folder does not already contain other files. It will automatically overwrite any file
USERNAME root yes SSH Username of the testing device
VERBOSE True yes Enable verbose outputI actually would have tried to debug the code, as I am familiar with python... but that error message doesn't even show the root cause when debugging is enabled and it is far to generic.
On the device everything seems fine, SSH is running on 22, agent on 4444, I see Client connections and disconnecting.
Agent version 1.0.5 git describe --tags says v1.3.0 git 530099ae7caed465bd25baf4d7016e01c463c0e3
ccsplit
commented
7 years ago Hello @floyd-fuh, the main issue appears to be happening within needle/core/device/agent.py#handle_read in asyncore.dispatcher.setblocking (wifi) | select.select (usb). At least that is where it was occurring for me when I was taking a look at it initially, this diff can be applied to get more information, if you want to take a look at it.
marco-lancini
commented
7 years ago Hi @floyd-fuh, thank you for reporting this. I would be immensely grateful if you could apply the diff proposed by @ccsplit, so to get more information about the error!
floyd-fuh
commented
7 years ago I was on a client-separated Wifi so I thought I give USB a chance. There I was at least able to use the "shell" command to get a root shell. However, same error again:
[needle] > use binary/info/compilation_checks
[needle][compilation_checks] > run
[...]
------------------------------------------------------------
Traceback (most recent call last):
File "/opt/needle/needle/core/framework/module.py", line 111, in do_run
pre = self.module_pre()
File "/opt/needle/needle/core/framework/module.py", line 139, in module_pre
self.device.setup()
File "/opt/needle/needle/core/device/device.py", line 235, in setup
self.remote_op.dir_create(self.TEMP_FOLDER)
File "/opt/needle/needle/core/device/remote_operations.py", line 63, in dir_create
if not self.dir_exist(path):
File "/opt/needle/needle/core/device/remote_operations.py", line 56, in dir_exist
out = self.command_blocking(cmd, internal=True)
File "/opt/needle/needle/core/device/remote_operations.py", line 94, in command_blocking
out, err = self._device._exec_command_ssh(cmd, internal)
File "/opt/needle/needle/core/utils/utils.py", line 203, in wrapper
device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 203, in wrapper
device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 203, in wrapper
device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 203, in wrapper
device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 207, in wrapper
raise Exception("An error occurred and it was not possible to restore it ({} attempts failed)".format(self.tries))
Exception: An error occurred and it was not possible to restore it (3 attempts failed)
------------------------------------------------------------
[!] Exception: An error occurred and it was not possible to restore it (3 attempts failed)
[...]Then, well about your patch... it's broken, the lines are cut, see https://gist.github.com/ccsplit/6284feb4bb259d83f08b7994f04624f4#file-needle-diff-L45
So I fixed the patch first as I think it should work. The patch is OKish, but much better if we just let Python do the job it's supposed to do (btw. please change the "RETRY DECORATOR" aka Retry class, you should never ever catch all Exceptions). So changing
class Retry(object):
default_exceptions = (Exception)to:
class Retry(object):
default_exceptions = ()lets us do real debugging:
[!] Problem establishing connection: error -
------------------------------------------------------------
Traceback (most recent call last):
File "/opt/needle/needle/core/framework/framework.py", line 641, in _connection_new
self.device.connect()
File "/opt/needle/needle/core/device/device.py", line 218, in connect
self._connect_agent()
File "/opt/needle/needle/core/device/device.py", line 160, in _connect_agent
self.agent.exec_command_agent(Constants.AGENT_CMD_OS_VERSION)
File "/opt/needle/needle/core/utils/utils.py", line 197, in wrapper
return func(obj, *args, **kwargs)
File "/opt/needle/needle/core/device/agent.py", line 95, in exec_command_agent
return self.read_result()
File "/opt/needle/needle/core/device/agent.py", line 100, in read_result
return self.client.handle_read()
File "/opt/needle/needle/core/device/agent.py", line 41, in handle_read
ready = select.select([self], [], [], Constants.AGENT_TIMEOUT_READ)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 174, in _dummy
raise error(EBADF, 'Bad file descriptor')
error: [Errno 9] Bad file descriptor
------------------------------------------------------------
[!] error: [Errno 9] Bad file descriptorThat's better, now we can at least Google for the problem. Seems like sometimes it happens when you close the socket and then reuse it:
https://stackoverflow.com/questions/4642345/python-client-server-question
So actually it seems that is really the problem. When we replace self.close() with print("Closing handle") in agent.py (AsyncClient) we get the following output with the patch:
asyncore.dispatcher.setblocking(True) start
asyncore.dispatcher.setblocking(True) end
select.select() start
2017-06-14 18:14:38,173| ERROR | Secsh channel 0 open FAILED: Connection refused: Connect failed
2017-06-14 18:14:38,174| ERROR | Could not establish connection from ('127.0.0.1', 4444) to remote side of the tunnel
finished select.select()
Closing handle
finished self.recv()
select.select() start
finished select.select()
Closing handle
finished self.recv()
select.select() start
finished select.select()
Closing handle
finished self.recv()
select.select() start
finished select.select()
Closing handleSo I guess you need to look into your socket handling, you can not use the socket after closing it...
Btw. whenever the exception occurs, the mobile app crashes as well.
marco-lancini
commented
7 years ago Hi @floyd-fuh, first of all thanks for the info you provided, they really helped tracking down the issue.
This might prove difficult to fix to be fair. Reason is asyncore decides to close a socket as soon as a recv returns nothing: https://github.com/enthought/Python-2.7.3/blob/master/Lib/asyncore.py#L388
I tried to force a new re-negotiation of the session, but the client ends up hanging...
Any suggestion would be much appreciated here!
floyd-fuh
commented
7 years ago fyi, asyncore nearly got removed from python 3, is deprecated in python 3.6 and people better at python than me tell me that twisted or asyncio might be better options.
It might be a OSX only problem as well.
My suggestion is to research the error message a little bit further... https://duckduckgo.com/?q=asyncore+[Errno+9]+Bad+file+descriptor
floyd-fuh
commented
7 years ago Probably a workaround: Change a line in you OSX python installation https://bytes.com/topic/python/answers/589612-asyncore-select-statement-problem Fix: Move away from asyncore
marco-lancini
commented
7 years ago @floyd-fuh: I had a look at the source code of asyncore and I see why is gonna be deprecated....
Also, it's not feasible to ask users to make amendments to their python installation.
Going forward I think the only way will involve a replacement of asyncore with twisted
floyd-fuh
commented
7 years ago I replaced the asyncore with regular sockets. No need to do anything fancy asynchronous if not necessary.
marco-lancini
commented
7 years ago Thank you @floyd-fuh for taking the time to address this. I quickly checked this and it now seems to be fixed in v1.3.1 (https://github.com/mwrlabs/needle/releases/tag/v1.3.1).
Closing the issue for now.
floyd-fuh
commented
7 years ago No problem, did you test it? As you know I'm on OSX and I'm unsure if it all works fine on other environments
Issue
Consistently I receive an error when attempting to connect Needle with Python 2.7.13 on Mac OSX. It appears the issue may be caused by an issue within
asyncore.dispatcherregarding Python 2.7.13, which may be similar to the issue discussed within fail2ban issue #161. Which appears to be resolved through the following commit.The issue occurs when trying to connect through wifi and usb with the only difference being where the
Errno 9 Bad file descriptoris thrown. The issue also persists through reboot/reconnecting the device.Expected behaviour
Needle should connect to the device, and bring up a shell or execute the commands.
Actual behaviour
Needle throws a
RuntimeErrorfor recursive depth exceeded. Which appears to be caused by an Exception happening within needle/core/device/agent.py#handle_read of aErrno 9 Bad file descriptorwithinasyncore.dispatcher.setblocking (wifi) | select.select (usb). Additional logging was added within these files to debug the issue.Steps to reproduce
set IP <ipad ip>set PORT 22set PASSWORD passwordset PUB_KEY_AUTH False(Doesn't work either way)shellneedle error logs
... Truncated for brevity full output ... maximum recursion depth exceeded RuntimeError below:
Environment
Workstation Operating System
Mac OSX 10.12.4 (Sierra)
Python Version
Python 2.7.13
Python Packages (
pip freeze)From within a virtual environment:
Device iOS Version
iPad Air 2 running 8.1