search

ADS-Testing / SAMOTA

Replication of "(ICSE 22) Efficient Online Testing for DNN-based Systems using Surrogate-Assisted and Many-Objective Optimization"
MIT License
1 stars 0 forks source link

readme

Replication of "(ICSE 22) Efficient Online Testing for DNN-based Systems using Surrogate-Assisted and Many-Objective Optimization"

Modified version of the original replication package to support the followings:

Requirements

Hardware

Software

Directory Structure

Setup

Install Python Libraries

(Option 1: venv) Initialise venv and install the required packages:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

(Option 2: pipenv) Initialise pipenv and install the required packages:

pipenv install

(Option 3: conda) Initialise a conda environment and install the required packages (Useful if multiple python versions are locally installed):

conda create -n samota python=3.8
conda activate samota
pip install -r requirements.txt

Download & Setup Pylot (docker; 30-50 mins)

(Recommandation: Run the next step, i.e., Download CARLA, whlie you are doing the pylot setup below because downloading CALRA is time-consuming)

Check the docker:

docker images

If there is no docker, use the script from Pylot to install it.

If docker gives you a permission error, follow below (otherwise skip):

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Download the pylot docker image:

docker pull erdosproject/pylot:v0.3.2
nvidia-docker run -itd --name pylot -p 20022:22 erdosproject/pylot:v0.3.2 /bin/bash

Create ssh-keys using following command and press enter twice when prompted

ssh-keygen

Setup ssh using the keys

nvidia-docker cp ~/.ssh/id_rsa.pub pylot:/home/erdos/.ssh/authorized_keys
nvidia-docker exec -i -t pylot sudo chown erdos /home/erdos/.ssh/authorized_keys
nvidia-docker exec -i -t pylot sudo service ssh start

Download CARLA (20-40 mins)

Download the simulators from the following command and extract them to a folder name Carla_Versions (due to the figshare upload limit, the compressed file is divided into three)

./setup_carla.sh

Integrate CARLA and Pylot

Move the customised CARLA into the Pylot container

docker cp Carla_Versions pylot:/home/erdos/workspace/
rm -rf Carla_Versions # optional, to save some storage space

Move customised pylot scripts into the Pylot container

ssh -p 20022 -X erdos@localhost  # answer 'yes'

# now you should be inside the container
cd /home/erdos/workspace
mkdir results  # check, Results?
cd pylot
rm -rf pylot scripts
logout

# outside the container
docker cp implementation/pylot pylot:/home/erdos/workspace/pylot/
docker cp implementation/scripts pylot:/home/erdos/workspace/pylot/

Note: in case error while running the simulator, log in to docker and change the permissions of scripts by using following commands

ssh -p 20022 -X erdos@localhost
cd /home/erdos/workspace/pylot/scripts
chmod +x run_simulator.sh
chmod +x run_simulator_without_b.sh
chmod +x run_simulator_without_t.sh
chmod +x run_simulator_without_t_b.sh

Create an empty directory for saving results

mkdir {PATH_TO_THIS_REPO}/SAMOTA/implementation/runner/Results
mkdir {PATH_TO_THIS_REPO}/SAMOTA/implementation/runner/output
mkdir {PATH_TO_THIS_REPO}/SAMOTA/implementation/runner/output/temp

How to run

run the search algorithm using the following code

cd implementation/runner
python3 run_{search_algorithm}.py

Note: Ignore No such container:path: pylot:/home/erdos/workspace/results/finished.txt.

(Tip) How to start/stop docker an existing container

If the pylot container is not properly stopped during the execution of search algorithms due to unexpected errors (or keyboard inturruptions), you can use the following docker commands to start|stop the existing container: 

nvidia-docker start|stop pylot

If needed, you can check the status of the container(s):

nvidia-docker ps --all

Troubleshooting wiki links

TODO

License

MIT License