search

fkie-cad / FACT_docker

Dockerfile for building the FACT container
GNU General Public License v3.0
21 stars 10 forks source link

radare link not working #4

Closed frakman1 closed 2 years ago

frakman1 commented 2 years ago

When I click on the "View in radare" button, it opens a new page with the error:

Error! HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /v1/retrieve (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))

Is radare web ui supposed to be part of FACT or is it supposed to be installed elsewhere? I don't see it running or exposing that port within the docker container so I don't know how it is supposed to work.

jstucke commented 2 years ago

The radare web gui actually runs in another docker container. As FACT docker is configured at the moment (with "docker in docker"), the radare container should be located outside the FACT docker container with your other docker images and show up when you run docker images and get started when you run the FACT container. Since FACT docker is still a bit experimental, there might be problems with the URL and/or port (maybe the port is just missing in FACT docker config). We will look into that.

frakman1 commented 2 years ago

Thank you. I checked and the only radare image I see is radare_server. Is that it?

While troubleshooting,I even built my own docker-radare-web-gui, based on your Dockerfile and ran it on the host with mapped container port 8080 to host 8000, then volume mapped the main.cfg and nginx.cfg file and changed localhost to the IP address of the host. That didn't work either and gave an error like:

Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

I then ran out of ideas and gave up

maringuu commented 2 years ago

There are two things failing here. The first thing is that the installer does not install docker-compose properly in the container. The fix for this is here.

The other thing is that in the config the url for the radare container is set to localhost in the config. In the docker container this is wrong and should point to the host where docker is running. I'm searching for a clean solution for this.

frakman1 commented 2 years ago

Thank you for the update @maringuu I would love to use a pre-made, working docker image because I never know if mine is correctly configured or not.

jstucke commented 2 years ago

I'm sorry for your poor experience running FACT in docker. It is still in a more or less experimental state and when running it on the host or inside a VM, everything should work as intended. It is a good chance to root out some of the most severe issues and make it more stable, though. A pre-built image would sadly also not fix everything by itself, because you still need to set configs, mounts, folders on the host, etc. for it to work correctly.

jstucke commented 2 years ago

If you need radare at the moment, the easiest workaround would be to use the "raw download" option to download the respective files and analyze them with radare locally

jstucke commented 2 years ago

Actually I got it to work with adding option --net=host to the start (docker run) of the container. I also needed to start the radare container from within the FACT container manually but if you use the branch from https://github.com/fkie-cad/FACT_core/pull/669 this should work directly.

FACT needs to POST the file to the radare container and we need this option or some other workaround for the container to be reachable from inside the FACT container. --net=host is not the optimal solution, since it has some security implications.

frakman1 commented 2 years ago

I re-ran the fact container with --net=host but not sure which docker to start for radare I saw two radare related images (radare_server, docker-radare-web-gui) and started them within the fact container One gave me an error

fact@Kubuntu18:~$ docker run radare_server
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/10/28 14:11:37 [emerg] 1#1: host not found in upstream "app:8080" in /etc/nginx/nginx.conf:14
nginx: [emerg] host not found in upstream "app:8080" in /etc/nginx/nginx.conf:14
fact@Kubuntu18:~$ docker run frakman1/docker-radare-web-gui
 * Serving Flask app 'server.flask_forwarding' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
[2021-10-28 14:09:45][_internal][WARNING]:  * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
[2021-10-28 14:09:45][_internal][INFO]:  * Running on http://172.17.0.2:8080/ (Press CTRL+C to quit)

I still got an error in the WebUI when I click on the "View in radare" button

jstucke commented 2 years ago

Sorry that part was not really clear. What I meant was navigating to the folder src/install/radare/ and calling docker-compose up (again with docker exec).

jstucke commented 2 years ago

But that is not the best solution, since you would have to do it each time. Since https://github.com/fkie-cad/FACT_core/pull/669 is merged you should also be able to rebuilt the image or do this:

frakman1 commented 2 years ago

Thank you. I tried the docker-compose command but ran into multiple errors. It also complains about port 8000 being in use despite netstat showing nothing listening on it. Normally using 0.0.0.0: in docker run fixes this kind of error. No idea why though.

fact@Kubuntu18:/opt/FACT_core/src/install/radare$ docker-compose up
Starting radare_server_1 ...
radare_app_1 is up-to-date

ERROR: for radare_server_1  a bytes-like object is required, not 'str'

ERROR: for server  a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.22/containers/84b5511d3b3c799b71715fe6c914e363b8e24531ee960622b6cd274ea675e07c/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/compose/service.py", line 625, in start_container
    container.start()
  File "/usr/lib/python3/dist-packages/compose/container.py", line 241, in start
    return self.client.start(self.id, **options)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 1109, in start
    self._raise_for_status(res)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.22/containers/84b5511d3b3c799b71715fe6c914e363b8e24531ee960622b6cd274ea675e07c/start: Internal Server Error ("b'driver failed programming external connectivity on endpoint radare_server_1 (b08864896688b9434d01725c3bf91e9a1bcc5d14d72126aca95fba0a3217a3e0): Error starting userland proxy: listen tcp [::]:8000: bind: address already in use'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 11, in <module>
    load_entry_point('docker-compose==1.25.0', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 72, in main
    command()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 128, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1107, in up
    to_attach = up(False)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1088, in up
    return self.project.up(
  File "/usr/lib/python3/dist-packages/compose/project.py", line 565, in up
    results, errors = parallel.parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/project.py", line 548, in do
    return service.execute_convergence_plan(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 567, in execute_convergence_plan
    return self._execute_convergence_start(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 506, in _execute_convergence_start
    _, errors = parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 508, in <lambda>
    lambda c: self.start_container_if_stopped(c, attach_logs=not detached, quiet=True),
  File "/usr/lib/python3/dist-packages/compose/service.py", line 620, in start_container_if_stopped
    return self.start_container(container)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 627, in start_container
    if "driver failed programming external connectivity" in ex.explanation:
TypeError: a bytes-like object is required, not 'str'
fact@Kubuntu18:/opt/FACT_core/src/install/radare$ sudo netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.1:45963         0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:26001         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:4242          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:44597         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:7001          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:20670         0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:4000            0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:12001         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:9191          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:25001         0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:27018           0.0.0.0:*               LISTEN      -
tcp6       0      0 ::1:45963               :::*                    LISTEN      -
tcp6       0      0 :::139                  :::*                    LISTEN      -
tcp6       0      0 :::1716                 :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 ::1:631                 :::*                    LISTEN      -
tcp6       0      0 ::1:7001                :::*                    LISTEN      -
tcp6       0      0 :::1180                 :::*                    LISTEN      -
tcp6       0      0 :::445                  :::*                    LISTEN      -
tcp6       0      0 :::4000                 :::*                    LISTEN      -
tcp6       0      0 :::6600                 :::*                    LISTEN      -

Just saw the second comment. Will try to git pull approach next

jstucke commented 2 years ago

does radare maybe already run? Does docker ps on the host show anything?

Edit: maybe it was still running from before. Then you should docker stop the radare containers (on the host) before trying docker-compose up again.

frakman1 commented 2 years ago

I checked and there was no radare running anywhere. I stopped any container that I had run before and there was nothing radare related when doing docker ps

I attempted the git pull solution but I also got the same error as before when docker-compose was run automatically. This time, it prevents the fact webui from running although the docker container is still up.

[2021-10-28 14:35:37][install][ERROR]: Failed to run docker-compose -f /opt/FACT_core/src/install/radare/docker-compose.yml up -d:
jstucke commented 2 years ago

What happens if you try to run it as before from the venv?

frakman1 commented 2 years ago

I don't see a compose-env folder: I also had to manually stop the radare app that failed but still hanging around: docker stop radare_app_1 


fact@Kubuntu18:~$ cd /opt/FACT_core/src/install/
fact@Kubuntu18:/opt/FACT_core/src/install$ source compose-env/bin/activate
bash: compose-env/bin/activate: No such file or directory
fact@Kubuntu18:/opt/FACT_core/src/install$ ls -lsath
total 92K
8.0K drwxr-xr-x 1 fact fact 4.0K Oct 28 14:34 ..
4.0K drwxr-xr-x 1 fact fact 4.0K Oct 28 14:34 .
4.0K -rw-r--r-- 1 fact fact 2.8K Oct 28 14:34 common.py
 12K -rw-r--r-- 1 fact fact 8.9K Oct 28 14:34 frontend.py
4.0K -rwxr-xr-x 1 fact fact 2.5K Oct 28 14:34 pre_install.sh
4.0K -rw-r--r-- 1 fact fact  311 Oct 28 14:34 requirements_frontend.txt
4.0K drwxr-xr-x 2 fact fact 4.0K Oct 20 16:21 __pycache__
   0 -rw-r--r-- 1 fact fact    0 Oct 20 16:19 __init__.py
4.0K -rw-r--r-- 1 fact fact  113 Oct 20 16:19 apt-pkgs-backend.txt
4.0K -rw-r--r-- 1 fact fact  192 Oct 20 16:19 apt-pkgs-common.txt
8.0K -rw-r--r-- 1 fact fact 6.4K Oct 20 16:19 backend.py
4.0K -rw-r--r-- 1 fact fact 2.7K Oct 20 16:19 db.py
4.0K -rw-r--r-- 1 fact fact   75 Oct 20 16:19 dnf-pkgs-backend.txt
4.0K -rw-r--r-- 1 fact fact  313 Oct 20 16:19 dnf-pkgs-common.txt
4.0K -rw-r--r-- 1 fact fact   90 Oct 20 16:19 fact_env.sh
4.0K -rwxr-xr-x 1 fact fact 1.6K Oct 20 16:19 pre_install_fedora.sh
4.0K drwxr-xr-x 3 fact fact 4.0K Oct 20 16:19 radare
4.0K -rw-r--r-- 1 fact fact  234 Oct 20 16:19 requirements_backend.txt
4.0K -rw-r--r-- 1 fact fact  602 Oct 20 16:19 requirements_common.txt
4.0K -rw-r--r-- 1 fact fact  205 Oct 20 16:19 requirements_pre_install.txt
jstucke commented 2 years ago

If you don't have that folder, then you probably had the new version (where it was removed) already and it should normally start without issues. The problems seems still to be caused by the port being in use. Did you find what blocked the port?

and ran it on the host with mapped container port 8080 to host 8000

Could it be that the container itself blocks the port? Don't start the fact container with -p 8000 as the radare container needs that port (or it would need to be configured to a different port)

frakman1 commented 2 years ago

I didn't see port 8000 or 8080 used anywhere (netstat from host and container)

OK, I started from scratch with a new project and rebuilt and pulled all the docker containers again. I re-used the mongodb firmware database in /media/data

I found that my host was running apache2 and messing up my binds in a non-obvious way. I stopped the apache2 service and it seems to run better now.

However, the docker compose command still fails for radare and I found that I had to change the port format to this: - "0.0.0.0:8000:8000" in /opt/FACT_core/src/install/radare/docker-compose.yml within the fact container before it would work. I then re-started fact (docker start -ia fact) because docker run would undo my change.

So despite using the --net=host option, the radare link still wants to go to localhost:8000/radare1/m/ when I click on the "View in radare" link. When I changed "localhost" to my host's IP address in the browser URL, it then took me to the radare page.

The fix was to get into the container and modify the last line in /opt/FACT_core/src/config/main.cfg to set radare2_host to my hosts IP address and restart the fact container.

So now, finally, radare works for me. Thank you for your support!

image

frakman1 commented 2 years ago

My initial comment/observation is that the file says /tmp/tmpxxx instead of the name of the file I clicked on.

jstucke commented 2 years ago

Glad to hear that you could get it to work! We are also working on a more permanent solution for the radare problems.