search

nus-apr / cerberus

Research Acceleration Platform that provides interface to multiple state-of-the-art program analysis tools including but not limited to fuzzing, static analysis and program repair.
MIT License
65 stars 30 forks source link

Cerberus not working in docker container #30

Closed LouisSY closed 1 year ago

LouisSY commented 1 year ago

I'm trying to run cerberus in a docker container but it doesn't work. May I ask if cerberus can be used in docker container?

My container is created from the image of "ubuntu:20.04". (I also tried ubuntu:latest") When I run the command

cerberus --tool=vulnfix --benchmark=vulnloc --bug-index=22

it gives the following error.

====================================================================================================

Starting Cerberus (Program Repair Framework)

====================================================================================================

Bootstrapping framework
__________________________________________________________________________________________

reading profile values
reading configuration values from arguments
updating configuration values

Initializing setup
__________________________________________________________________________________________

loading repair tool
Runtime Error
Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Run time statistics:

Experiment Count: -1

Cerberus exited with an error after 0.000 minutes

Could you please help me to see why this is happening??? Thank you so much!

Marti2203 commented 1 year ago

Hi, you can run Cerberus in a container but this will require the container to be able to run Docker (The no such file or directory is caused by the Docker Python SDK trying to find the unix socket, which represents the connection to the Docker Deamon). Is the container you have made provided with capabilities to interact with Docker?

Marti2203 commented 1 year ago

I just looked into this a bit more - it is not really recommended to run Docker inside of Docker, if you want to run Cerberus in a container, you will have to pass the -local flag, which will take in the tools and data from the local environment and not try to spin up a docker container.

LouisSY commented 1 year ago

Thank you so much for your help! I will have a try and see if that works!

rshariffdeen commented 1 year ago

@LouisSY if you need to run cerberus on a container you have to link the docker socket into the container so cerberus can communicate with the host docker to create new docker environments. 

docker run -d -ti  -v  /var/run/docker.sock:/var/run/docker.sock IMAGE_ID
LouisSY commented 1 year ago

Thank you so much for your reply! I have tried this method and this time the error message is changed.

The error is shown by the following:

Starting Cerberus (Program Repair Framework)

====================================================================================================

Bootstrapping framework
__________________________________________________________________________________________

reading profile values
reading configuration values from arguments
updating configuration values

Initializing setup
__________________________________________________________________________________________

loading repair tool
[warning] docker image not found in Docker registry
pulling docker image
loading benchmark
    [benchmark] benchmark environment not found
    [benchmark] building benchmark environment
        [benchmark] building docker image
Repair Failed
[error] Unable to build image: Dockerfile not found
Runtime Error
Error. Exiting...

Run time statistics:

Experiment Count: -1

Cerberus exited with an error after 1.471 minutes

Does it mean that Cerberus cannot find the dockerfile of fixing-bug tool?

Marti2203 commented 1 year ago

Hi, I have experimented with this yesterday but did not have time to look more into it - the problem lays in some interaction in the file binds - the docker container being constructed is unable to access the file on the container you are running Cerberus on. I am not the expert on this and will look into it more in the following days. I also tried the sysbox runtime ( a different docker runtime, compared to the default runc, it allows you to run docker in docker seamelessly which here would be useful) but had some issues with it - I suggest checking it out but at your own risk as I temporarily broke my docker installation while trying it out.

LouisSY commented 1 year ago

Thank you so much for you help!!!!

Apart from running cerberus in a docker container, I also tried to run it directly on my machine but It still gives an error message. The followings are the error message:

====================================================================================================

Starting Cerberus (Program Repair Framework)

====================================================================================================

Bootstrapping framework
__________________________________________________________________________________________

reading profile values
reading configuration values from arguments
updating configuration values

Initializing setup
__________________________________________________________________________________________

loading repair tool
Runtime Error
Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Run time statistics:

Experiment Count: -1

Cerberus exited with an error after 0.000 minutes

I did the following steps to encounter this problem: git clone --recurse-submodules https://github.com/nus-apr/cerberus to init when cloning the project cd /my-path-to-cerberus/cerberus to get into cerberus folder pip install -r requirments.txt to install all dependencies source activate cerberus --tool=vulnfix --benchmark=vulnloc --bug-index=22

My system info:

If it is convenient for you, could you please help me to check if there is anything wrong with my Installation Procedure? I followed the steps shown on Readme.md but it gives this error. Thank you so much!!!

Marti2203 commented 1 year ago

Can you run with --debug flag? This will help me get a better idea about the whole execution

LouisSY commented 1 year ago

This is what I got when running with --debug flag.

Input: cerberus --tool=vulnfix --benchmark=vulnloc --bug-index=22 --debug

Output:

====================================================================================================

Starting Cerberus (Program Repair Framework)

====================================================================================================

Bootstrapping framework
__________________________________________________________________________________________

reading profile values
reading configuration values from arguments
updating configuration values

Initializing setup
__________________________________________________________________________________________

loading repair tool
        [debug]      using tool: vulnfix
Runtime Error
Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Run time statistics:

Experiment Count: -1

Cerberus exited with an error after 0.000 minutes

It seems that the error messages are almost same.

Marti2203 commented 1 year ago

There is a file ./logs/log-error (in the directory of cerberus), can you show that as well?

LouisSY commented 1 year ago

The file ./logs/log-error is shown in the following:

2023-02-15 16:32:15,567 ERROR Runtime Error
2023-02-15 16:32:15,567 ERROR Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
2023-02-15 16:32:15,568 ERROR Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shuai/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/transport/unixconn.py", line 30, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/shuai/.local/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/home/shuai/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/shuai/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/shuai/.local/lib/python3.10/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shuai/Desktop/COMP0138/cerberus/app/core/main.py", line 284, in main
    repair_tool_list, benchmark, setup = initialize()
  File "/home/shuai/Desktop/COMP0138/cerberus/app/core/main.py", line 265, in initialize
    tool.check_tool_exists()
  File "/home/shuai/Desktop/COMP0138/cerberus/app/drivers/tools/AbstractTool.py", line 197, in check_tool_exists
    if not container.is_image_exist(repo_name, tag_name):
  File "/home/shuai/Desktop/COMP0138/cerberus/app/core/container.py", line 11, in is_image_exist
    client = docker.from_env()
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/shuai/.local/lib/python3.10/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

2023-02-15 16:32:15,569 ERROR 
Cerberus exited with an error after 0.000 minutes
Marti2203 commented 1 year ago

Is your user added to the group for docker? Maybe this is the reason? Worst case is changing file permissions highly not recommended if you are not sure what you are doing

Marti2203 commented 1 year ago

Please do not touch the file permissions, the user group thing should work @LouisSY

LouisSY commented 1 year ago

I didn't add any users in docker group for now. I thought docker creates the docker group, but also any sudoers can use Docker.

By using getent group docker and getent group sudo I got:

docker:x:999:
sudo:x:27:shuai

I didn't make any changes on the file permissions. May I ask if I need to add user "shuai" to docker group?

Marti2203 commented 1 year ago

Yes, just add shuai to the docker group as the socket is accessible from that group. IIRC you only need to sign out and login in order for the group changes to work (worst case is a restart). This should make Cerberus be able to interact with docker and hopefully everything else will work without hiccups :D

Marti2203 commented 1 year ago

Sudoers can use sudo, which makes you root so CLI works properly as you have full permissions over everything but the docker group allows the user to interact with the socket and you are running Cerberus as a normal user

LouisSY commented 1 year ago

I added user to the docker group and cerberus works well now!!!!

Thank you so much for your patience and help :D

Marti2203 commented 1 year ago

Nice! Happy to help out :)