MemoryError on nla_parse_nested()

[sophron]

Hi @wraith-wireless,

As you know we are trying to include PyRIC on Wifiphisher. On my system, I'm getting a MemoryError. Apparently, the script goes in an infinite loop when executing nla_parse_nested method.

Here's the traceback:

Traceback (most recent call last): File "bin/wifiphisher", line 12, in run() File "/root/wifiphisher/wifiphisher/pywifiphisher.py", line 786, in run args.apinterface) File "/root/wifiphisher/wifiphisher/interfaces.py", line 246, in init self._check_compatibility(interface_object) File "/root/wifiphisher/wifiphisher/interfaces.py", line 270, in _check_compatibility modes = pyric.devmodes(card) File "/root/wifiphisher/PyRIC/pyric/pyw.py", line 456, in devmodes return nlstub(devmodes,card) File "/root/wifiphisher/PyRIC/pyric/pyw.py", line 957, in nlstub return fct(*argv) File "/root/wifiphisher/PyRIC/pyric/pyw.py", line 458, in devmodes return phyinfo(card,nlsock)['modes'] File "/root/wifiphisher/PyRIC/pyric/pyw.py", line 555, in phyinfo rmsg = nl.nl_recvmsg(nlsock) File "/root/wifiphisher/PyRIC/pyric/lib/libnl.py", line 243, in nl_recvmsg msg = nlmsg_fromstream(sock.recv()) File "/root/wifiphisher/PyRIC/pyric/lib/libnl.py", line 448, in nlmsg_fromstream nla_parse(msg,l,t,stream,nlh.NLMSGHDRLEN + genlh.GENLMSGHDRLEN) File "/root/wifiphisher/PyRIC/pyric/lib/libnl.py", line 478, in nla_parse if dt == nlh.NLA_NESTED: a = nla_parse_nested(a) File "/root/wifiphisher/PyRIC/pyric/lib/libnl.py", line 530, in nla_parse_nested ns.append(nested[idx+1:idx+(alen-1)]) MemoryError

[wraith-wireless]

I tried with three different cards but cannot recreate this error. Have you cloned or downloaded the newest version of pyw although this shouldn't be the problem since libnl as not been modified over any recent period of time.

Could you supply me with your card type and I think, I will have to look at the raw stream to run this agains the code. If you could modify the function nl_recvmsg in libnl. chjange the first line of code

msg = nlmsg_fromstream(sock.recv())

to

stream = sock.recv()
fin = open('memerrorstream','w')
fin.write(stream)
fin.close()

and then paste the output here or on pastebin or something. That way I can check it out.

[wraith-wireless]

Hey so looking into Memory Errors. It appears that this more likely due to running out of memory while storing the nested attributes. See http://stackoverflow.com/questions/11283220/memory-error-in-python I still cannot see this as being a problem, since the number of strings being stored is rather small.

First off, are you running Python 64bit and are you running Kali or some other live distro?

Second this may be an inherent problem in PyRIC taking up too much space. Should be unlikely since as I said the number of strings being appended for any one function is small.

If it seems to be that PyRIC is a memory hog especially on live systems, I'll have to look at reworking storing all the netlink response in memoryviews. For now I'm going to do some memory profiling and see what I get.

[sophron]

@wraith-wireless The PR that we are introducing PyRIC lies here. I believe the PyRIC code was cloned almost a month ago. I can try the latest version if you want to.

I emailed you the dump of the stream since this is binary data.

I'm using this on a native Kali Linux 32bit.

Let me know how it goes.

[wraith-wireless]

Ok I won't be able to take a look at it until this afternoon but I'll get right on it then. About the latest code - wouldn't be the problem libnl has been pretty stable, you don't need to reclone.

From what I read 32-bit Python limits memory to 4GB. However, this should be more than enough I think the phyinfo dump as a whole maxes out at 1700.

[wraith-wireless]

So, I don't get a memory error on this. On a side-note the 'offending' stream you emailed is the retrieval of the family id, meaning, on your system at least, retrieving the family id is too big.

Did some calcs, the size of the parsed message 7706 bytes which shouldn't cause a memory error in itself. The memory error has to come from the substrings being stored. I'll go ahead and starting recoding that to use a memory view. Hopefully that will help.

[sophron]

I debugged a little more. The script runs out of memory because the idx is always zero and the l (length of nested) is 512, hence it goes in an infinite loop here.

Apparently, alen is also always zero, meaning that struct.unpack_from fails to unpack the binary data.

[wraith-wireless]

Hmm, so the question becomes how to handle a 0 alen? skip the 0 byte and hope it recovers? Can you paste the output of (from a shell)

iw phy <phy> info

Also since based on your stream the error was occurring during pyw.familyid(), I changed the way some of the familyid attributes were parsed. Can you go ahead and use the policy.py and libnl.py I just pushed up and show me any errors with output that occurs?

[wraith-wireless]

OK, disregard previous instructions. I didn't like it as was a solving the problem as much as hiding it. The newest push throws an exception on alen == 0 which causes the nested attribute to be stored as is i.e. packed bytes. Of course if this occurs in say supported commands, we'll get an empty list. But, I'm hoping for now it won't. This will also allow us to determine exactly what attribute was causing the error i.e supported freqs, supported commands etc.

To apply, please clone the newest PyRIC .

Also, although PyRIC does not handle anything with nmcli. It does have native support now for rfkill. In pyw, I have implemented block(card) and unblock(card). And, support for rand mac address generation

[sophron]

Good, this solved it. From a quick test it runs OK now.

[sophron]

Do you think we'll have these fixes on PyPI version soon enough?

[wraith-wireless]

Awesome, I ran the unittest and everything came back good on my end. I just added a new minor release v 0.1.1-1 to PyPi, https://pypi.python.org/pypi/PyRIC/0.1.1.post1 I wanted to make some more changes to PyRIC before i release v 0.1.2.

[wraith-wireless]

PyRIC 0.1.2 is ready on PyPI. Had to fix some install errors that @blackHatMonkey identified.

[wraith-wireless]

PyRIC 0.1.2.1 should be stable "relatively so"