Install fails with error: Failed building wheel for M2Crypto

[Strazzom]

I have followed the installation steps from here. Everything works fine up until I try to run pip install -r requirements.txt. After pip downloads all dependencies, it fails with the error in the title.

The full log can be found here: https://gist.github.com/Strazzom/c9b1859afd6e478b505e8310f88d8ae7

This is using the same environment as in issue #1, minus the fact that it is not running in a docker container.

I have tried the following to correct the issue. Neither worked:

1) Installed libssl-dev from apt. This did not fix the issue, so I uninstalled it with --purge.

2) pip uninstall m2crypto. It was not installed. I then installed from the Debian repositories with apt install python-m2crypto. My reasoning was based on the following:

There is (was?) known issue with Debian/M2Crypto, so in case something related to M2Crypto will cause exceptions just uninstall one from pip and install one which is shipped with distro.

This is quoted from issue #619 in the main branch.

I will keep testing and update this issue with progress.

[alxchk]

Installed libssl-dev from apt. This did not fix the issue, so I uninstalled it with --purge.

Well, according to the log this is not that issue with M2Crypto. That issue was after successful build. For some reason it just can't find the header. In my system openssl/err.h belongs to libssl-dev, which you tried to install, so this looks strange.

dpkg -S openssl/err.h
libssl-dev:amd64: /usr/include/openssl/err.h

[Strazzom]

After diffing the two error messages, and using sed to change some paths, it appears the errors from this issue and #1 are the same.

The only thing that is actually different is the following (from diff output):

user@pupy-testbded:~$ diff error-docker error-baremetal 
1c1
< No handlers could be found for logger "pupy.network"
---
> No handlers could be found for logger "pupy.network.pss"

This suggests that m2crypto is also failing to install in the docker build, (which I assume is automated?) and these are actually the same issue. As such, I have tried to add a duplicate tag to the first one (not sure if it worked). EDIT: Closed first issue.

EDIT 2:

I have compared the error messages thrown when trying to install requirements.txt using pip both with and without libssl-dev.

Snippet here: https://gist.github.com/Strazzom/e0bc259c0dbceb94597338bd063d4601

When installing without libssl-dev, it fails because it can't find err.h. When installing with libssl-dev, it generates a much longer error. My guess is that something is going wrong with libssl-dev independent of it not being installed.

My output for dpkg -S openssl/err.h is the same. What platform do you use? It is possible that there was a change in libssl-dev.

[alxchk]

Try to use libssl1.0-dev. In D9 libssl-dev defaults to libssl1.1, which M2 can't use

[Strazzom]

Progress report:

I installed libssl1.0-dev. It mitigated the error described above, but now I am seeing errors from kcp after doing an install with pip install -r requirements.txt.

Error in question: https://gist.github.com/Strazzom/2a72aa36104a6f6d72815b0d86f6d558

It may be difficult to figure out what is wrong, seeing as more extended output is cut off by the segmentation fault.

After running build-docker.sh, I see a similar error (again with KCP): https://gist.github.com/Strazzom/9f1157c036a9b4f1fb128bfb68d0ae84

For whatever reason, the newest docker pull from alxchk/pupy:unstable seems to work without issue. I am not sure what changed between versions, but whatever it is seems to have corrected the first error when pupysh.py is run.

The only issue in the latest docker pull (created ~17 hours ago as of now), is this error after running the pupysh.py script:

WARNING:scapy.loading:can't import layer netflow: DADict instance has no attribute 'tcp'

The above error does not seem to actually break anything, but I will test functionality to verify. This may be difficult, seeing as this is the first time I have actually gotten pupy to work.

What platform are you building on? I suspect very highly that the KCP error is a result of my environment. Perhaps it is because the install of Debian 9 is dirty from the testing thus far; if you can confirm that you are building successfully on Debian 9, I will retry on a clean VM.

[alxchk]

WARNING:scapy.loading:can't import layer netflow: DADict instance has no attribute 'tcp'

This thing well-known, but I'm not sure what I'm going to do with that. The reason - some strange people place meta-information (which is used in runtime) to docstrings. And during docker generation all things are compiled with -O. Thus docstrings excluded. Likely I'll improve PupyCompile to handle such cases and will use it instead of python -O.

KCP is another story, I didn't test it for a while but it should work. Can you please build extension standalone (https://github.com/alxchk/pykcp) and try to load it with gdb -a python -c 'import kcp'. Also if you can please provide strace log of installation process. Maybe the problem is not with KCP, but with some other thing.

Compilation in docker failed because submodule was not synced for some reason:

pykcp.c:10:22: kcp/ikcp.h: No such file or directory
pykcp.c:11:22: kcp/ikcp.c: No such file or directory

What platform are you building on?

There are two completely different things.

1. pupysh deps
2. pupy client

I'm using pupysh in gentoo testing and debian stretch+sid. Docker image based on debian:stretch-slim (https://github.com/alxchk/pupy/blob/unstable/pupy/Dockerfile#L1).

Modules (clients) for Linux are compiled inside docker toolchain (https://github.com/alxchk/docker-old-tc-bootstrap, https://github.com/alxchk/docker-old-tc) with modified debian lenny/etch to make ABI compatible with as much distros as much as possible.

When you run build-docker.sh you are using these toolchains, so only two things in environment which do matter are working docker and enabled vsyscall support|emulation|whatever for amd64 builds (because of ancient glibc).

Docker build mounts pupy sources to container, so in case something was not initialized properly or modified this may be the reason of error. In your case you should try to sync all submodules first. At least pupy/external/pykcp is not synced properly (pupy/external/pykcp/kcp likely missing).

Please note that latest commit in master breaks Docker build because of this - https://github.com/boppreh/keyboard/issues/187. This only matters in case you are building your own docker images locally which you are not doing right now.

[Strazzom]

Before doing any of the above, I will try an install on a clean VM. I would like to rule out that possibility. Failing that, I will go through the procedure outlined above.

EDIT: Still having issues with the installation on clean VM. Couple of questions as a sanity check:

I am attempting to install without docker. I should in fact be attempting to install pupy per the original instructions here, correct? Is there a new installation procedure for your branch?

Given that this is on Debian 9, should I install the dependencies specified in the Dockerfile before attempting to run pip install -r requirements.txt?

[alxchk]

I am attempting to install without docker. I should in fact be attempting to install pupy per the original instructions here, correct? Is there a new installation procedure for your branch?

Generally speaking you can use Dockerfile as a reference. Note that you still need to have docker to build clients. There is no repo with precompiled clients here.

I also wrote a script which initializes work directory for pupy with virtualenv, but it was never fully tested.

I'll write installation steps to wiki likely today, but in the end it will contain same info as Dockerfile and script.

[alxchk]

https://github.com/alxchk/pupy/wiki/Installation-(Debian-9)

[Strazzom]

I am testing out the new installation instructions for Debian 9. The script create-workspace.py fails upon attempting to run the build-docker.sh script. This is probably some kind of permission error, or some quirk of subprocess.

I have executed the build-docker.sh script manually, will report back if it throws errors.

Here is the exception raised when it tries to run said script:

[+] Build sources-linux with toolchain alxchk/tc-linux64
Traceback (most recent call last):
  File "./create-workspace.py", line 78, in <module>
    ], env=env, cwd=pupy)
  File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./build-docker.sh']' returned non-zero exit status 139

EDIT 1: This seems to be as a result of the last command in that script failing (or at least not succeeding).

daemonize.o: In function `daemonize':
/build/workspace/project/client/sources-linux/daemonize.c:214: warning: fexecve is not implemented and will always fail
[+] Build complete

EDIT 2:

Command "/root/pupy/pupyws/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-oXNO3Y/psutil/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-8_1Ej_-record/install-record.txt --single-version-externally-managed --compile --install-headers /root/pupy/pupyws/include/site/python2.7/psutil" failed with error code 1 in /tmp/pip-build-oXNO3Y/psutil/
Traceback (most recent call last):
  File "./create-workspace.py", line 91, in <module>
    ], cwd=os.path.join(pupy, 'pupy'))
  File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/root/pupy/pupyws/bin/pip', 'install', '-r', 'requirements.txt']' returned non-zero exit status 1
root@pupy-testbed:~/pupy# 

This is after removing lines 76-78 in create-workspace.py, and running build-docker.sh manually as root.

[kefkahacks]

I believe the script does that because it uses -D to daemonize (perhaps his docker didn't come from the repo or rather my docker came from the kali repo which pulls from sid). In my environment it's -d

I didn't verify that fully but manually running through the steps and tweaking a bit, I was able to build binary clients for linux32 and 64 and infect myself. Didn't test any modules, beyond some shell commands.. I moved on to android and jarsigner errors.. but, that issue doesn't belong here, so I'll save it.

[Strazzom]

@kefkahacks: What exactly did you tweak? I stopped trying to fix this issue immediately because I can use the docker container, but it is still an issue.

Could you post a script log or asciinema recording?

EDIT: After more recent testing, it seems that the build process is still broken for me. Trying to record it with asciinema actually crashes a clean VM.

EDIT 2: Not an asciinema issue. VM crashes as soon as it tries to install python deps after building "toolchain alxchk/tc-windows".

Exact step: [+] Build sources with toolchain alxchk/tc-windows

[Strazzom]

@alxchk: Followup on the above.

I created a fresh Debian 9 VM and ran an apt update/upgrade. I then followed the wiki page for Debian 9 installation to the letter, (note that this was after I edited it; that should not matter however, as it was only the docker install script section of it).

Upon attempting to run ./create-workspace.py pupyws as a regular user (also happens with root), I get the following before the VM crashes:

[+] Build clients
/home/user/pupy

[+] Build sources with toolchain alxchk/tc-windows
[+] Install python packages
Collecting https://github.com/secdev/scapy/archive/master.zip
  Downloading https://github.com/secdev/scapy/archive/master.zip (3.4MB)

No error, nothing. The VM just halts suddenly and the window disappears.

Full strace can be found here.

I am running this on Qubes 3.2.

This is actually a serious impediment at this point, as I can't build the docker image from this without the platform specific payloads. This is getting in the way of testing Docker Compose.

[alxchk]

The VM just halts suddenly and the window disappears.

Well.. This does looks like an VM issue :) Maybe you are out of space?

In case you are doing strace, do -ff or at least -f.

[Strazzom]

I have tested this on a Virtualbox VM of Debian 9. After enabling vsyscall in the kernel options, I can no longer reproduce this issue.

I believe it was caused in part by vsyscall not being enabled, and also by a misunderstanding of the instructions. I will update the Debian 9 install instructions, and post the full asciinema log of the process.