Usage example issues

[jose]

Hi @HakonHarnes,

First of all thanks for sharing this repository.

I've been trying to run the usage examples but with little to no success so far.

Setup

Your setup mentions

docker create network db_network 
docker create network mn_network 
docker create network wasim_network 

but docker v24.0.7 requires

docker network create db_network 
docker network create mn_network 
docker network create wasim_network 

Usage

Building the dataset

I'd to made a few changes to the Dockerfile to make it run.

diff --git a/dataset/Dockerfile b/dataset/Dockerfile
index 6856cee..114f786 100644
--- a/dataset/Dockerfile
+++ b/dataset/Dockerfile
@@ -1,4 +1,4 @@
-FROM archlinux:base-devel-20230312.0.133040
+FROM archlinux:base-devel

 RUN pacman -Syyu vim git nodejs gcc cmake ninja python3 npm python-pip --noconfirm

@@ -52,6 +52,6 @@ RUN echo 'int main() { return 0; }' | \
 WORKDIR /app
 COPY requirements.txt ./
 COPY tigress.h /usr/local/bin/tigress/3.3.2/tigress.h
-RUN pip install -r requirements.txt
+RUN pip install -r requirements.txt --break-system-packages
 COPY . .
 CMD ["python", "main.py"]

but now, I get the following error when I run docker compose run build-dataset

Traceback (most recent call last):
  File "/app/main.py", line 83, in <module>
    main()
  File "/app/main.py", line 73, in main
    code = build_wasm(dir_path, folder)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/main.py", line 59, in build_wasm
    update_db(name, code, file_out, folder)
  File "/app/main.py", line 34, in update_db
    add_document('unobfuscated', data)
  File "/app/mongodb/utils.py", line 54, in add_document
    result = collection.replace_one(filter_, document, upsert=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/collection.py", line 965, in replace_one
    self._update_retryable(
  File "/usr/lib/python3.12/site-packages/pymongo/collection.py", line 872, in _update_retryable
    return self.__database.client._retryable_write(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1569, in _retryable_write
    return self._retry_with_session(retryable, func, s, bulk, operation, operation_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1455, in _retry_with_session
    return self._retry_internal(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/_csot.py", line 108, in csot_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1501, in _retry_internal
    ).run()
      ^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/mongo_client.py", line 2347, in run
    return self._read() if self._is_read else self._write()
                                              ^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/mongo_client.py", line 2450, in _write
    self._server = self._get_server()
                   ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/mongo_client.py", line 2433, in _get_server
    return self._client._select_server(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1316, in _select_server
    server = topology.select_server(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/topology.py", line 368, in select_server
    server = self._select_server(
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/topology.py", line 346, in _select_server
    servers = self.select_servers(
              ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/topology.py", line 253, in select_servers
    server_descriptions = self._select_servers_loop(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pymongo/topology.py", line 303, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: mongodb:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: <TopologyDescription id: 667189d933ee8152d9954e4a, topology_type: Unknown, servers: [<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('mongodb:27017: [Errno -2] Name or service not known (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>

Any idea?

-- Best, Jose

[HakonHarnes]

Hi Jose!

Seems like pinning the Arch Linux version and installing the latest versions of the packages from the AUR causes issues. I think your approach with removing the specific container version should work (fingers crossed).

Regarding the other issue, this is occurs because the dataset container can't communicate with the mongodb server. Looking at the error, it seems like the connection is timing out. Have you started the mongodb server? It needs to run while conducting the experiments.

If it the mongodb server is running and it still doesn't work, it's likely an issue with the docker network. To debug this, see if you can try to ping the mongodb container from the dataset container.

The docker network commands are indeed wrong. I've fixed that in the README now. Thanks for pointing that out.

Hope that helps some.

[jose]

Seems like pinning the Arch Linux version and installing the latest versions of the packages from the AUR causes issues. I think your approach with removing the specific container version should work (fingers crossed).

The archlinux:base-devel-20230312.0.133040 version was given some key errors while installing some packages.

Regarding the other issue, this is occurs because the dataset container can't communicate with the mongodb server. Looking at the error, it seems like the connection is timing out. Have you started the mongodb server? It needs to run while conducting the experiments.

I think I've. I ran

cd mongodb
docker compose run mongodb

that should be enough right?

If it the mongodb server is running and it still doesn't work, it's likely an issue with the docker network. To debug this, see if you can try to ping the mongodb container from the dataset container.

How would one do that? (Not a docker expert.)

The docker network commands are indeed wrong. I've fixed that in the README now. Thanks for pointing that out

👍

-- Best, Jose

[HakonHarnes]

I think I've fixed the dockerfiles now. I've removed the specific the specific arch linux versions, so now it just uses the latest version. For the issue with installing python packages, I've configured a virtual environment for each container now (prior versions of Arch Linux allowed you to installed packages globally. This is no longer allowed so a virtual env is needed).

Regarding the mongodb issue, I think that should be resolved now. The issue was that the docker compose file was missing the hostname property so the other containers weren't able to connect to the mongodb container. Everything works on my end now.

Are you able to get it working now?