Unknown error when submitting docker image to leaderboard benchmark

[friendship1]

Hello, I am having a hard time submitting docker images to carla leaderboard right now. It may not be an issue caused by this repo, since this repository is written as a baseline on the official alphadrive site(https://leaderboard.carla.org/get_started/), I'm asking here. Any help would be appreciated.

If I run it on my local machine it works fine with no issues. (Tested on g3.8xlarge machine) But as a test, when I submit it to the CARLA benchmark leaderboard, it spins for 7 minutes and then terminated without any results. Because it is a benchmark environment, I cannot see the error message.

I used carla from here (https://github.com/carla-simulator/carla/releases/tag/0.9.10.1) and this is the Dockerfile.master I used.

If the Docker image you submitted remains on your local machine, we would appreciate it if you could share it. If this is difficult, I'm curious how the printenv results in a working docker image.

FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04

ARG HTTP_PROXY
ARG HTTPS_PROXY
ARG http_proxy

RUN apt-get update && apt-get install --reinstall -y locales && locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US
ENV LC_ALL en_US.UTF-8

RUN apt-get update && apt-get install -y --no-install-recommends \
         build-essential \
         cmake \
         git \
         curl \
         vim \
         ca-certificates \
         libjpeg-dev \
             libpng16-16 \
             libtiff5 \
         libpng-dev \
         python-dev \
         python3.5 \
         python3.5-dev \
         python-networkx \
         python-setuptools \
         python3-setuptools \
         python-pip \
         python3-pip && \
         pip install --upgrade "pip < 21.0" && \
         pip3 install --upgrade "pip < 21.0" && \
         rm -rf /var/lib/apt/lists/*

# installing conda
RUN curl -o ~/miniconda.sh -LO https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
     chmod +x ~/miniconda.sh && \
     ~/miniconda.sh -b -p /opt/conda && \
     rm ~/miniconda.sh && \
     /opt/conda/bin/conda clean -ya && \
     /opt/conda/bin/conda create -n python37 python=3.7 numpy networkx scipy six requests

RUN packages='py_trees==0.8.3 shapely six dictor requests' \
        && pip3 install ${packages}

WORKDIR /workspace
COPY .tmp/PythonAPI /workspace/CARLA/PythonAPI
ENV CARLA_ROOT /workspace/CARLA

ENV PATH "/workspace/CARLA/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg":/opt/conda/envs/python37/bin:/opt/conda/envs/bin:$PATH

# adding CARLA egg to default python environment
RUN pip install --user setuptools py_trees==0.8.3 psutil shapely six dictor requests

ENV SCENARIO_RUNNER_ROOT "/workspace/scenario_runner"
ENV LEADERBOARD_ROOT "/workspace/leaderboard"
ENV TEAM_CODE_ROOT "/workspace/team_code"
ENV PYTHONPATH "/workspace/CARLA/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg":"${SCENARIO_RUNNER_ROOT}":"${CARLA_ROOT}/PythonAPI/carla":"${LEADERBOARD_ROOT}":${PYTHONPATH}

COPY .tmp/scenario_runner /workspace/scenario_runner
COPY .tmp/leaderboard /workspace/leaderboard
COPY .tmp/team_code ${TEAM_CODE_ROOT}

RUN mkdir -p /workspace/results
RUN chmod +x /workspace/leaderboard/scripts/run_evaluation.sh

########################################################################################################################
########################################################################################################################
############                                BEGINNING OF USER COMMANDS                                      ############
########################################################################################################################
########################################################################################################################

COPY .tmp/carla_project /workspace/carla_project

ENV PYTHONPATH "/workspace/CARLA/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg":${PYTHONPATH}
ENV PYTHONPATH "/workspace/":${PYTHONPATH}
ENV TEAM_AGENT /workspace/team_code/image_agent.py
ENV TEAM_CONFIG /workspace/team_code/model.ckpt
ENV CHALLENGE_PHASE_CODENAME SENSORS
ENV CHALLENGE_TRACK_CODENAME SENSORS

RUN apt-get update && apt-get install -y --no-install-recommends \
         libgtk2.0-dev

ENV PATH="/opt/conda/bin:${PATH}"
RUN /opt/conda/envs/python37/bin/pip install -r /workspace/carla_project/requirements.txt
RUN /opt/conda/bin/conda install -c anaconda libgcc=7.2.0
RUN /opt/conda/bin/conda init bash

RUN cp /opt/conda/lib/libstdc++.so.6.0.28 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
ENV CONDA_DEFAULT_ENV python37
ENV PATH /opt/conda/envs/python37/bin:$PATH

RUN /opt/conda/envs/python37/bin/pip install tabulate ephem

########################################################################################################################
########################################################################################################################
############                                   END OF USER COMMANDS                                         ############
########################################################################################################################
########################################################################################################################

ENV SCENARIOS ${LEADERBOARD_ROOT}/data/all_towns_traffic_scenarios_public.json
ENV ROUTES ${LEADERBOARD_ROOT}/data/routes_training.xml
ENV REPETITIONS 1
ENV CHECKPOINT_ENDPOINT /workspace/results/results.json
ENV DEBUG_CHALLENGE 0

ENV HTTP_PROXY ""
ENV HTTPS_PROXY ""
ENV http_proxy ""
ENV https_proxy ""

CMD ["/bin/bash"]

[bradyz]

Since your submission terminates without any results I'd guess something is breaking on startup - maybe some compatibility error.

Try editing the dockerfile to more closely match the current leaderboard repo, in particular the path to the python egg

I would give this a try and resubmit, and if it still fails, try contacting someone on the CARLA discord https://discord.com/invite/vNVHXfb since this will be extremely difficult to debug without knowing error messages

[friendship1]

Thank you very much. When I inquired about alphadrive, there was no reply, so I couldn't solve the problem for about 2 weeks, but thank you for your reply. I fixed the problem by fixing the Dockerfile. I think the problem is caused by the python egg path not being updated, as you mentioned. Below is the Dockerfile I used to create a working Docker image.

As we progress further, if we find out exactly what the problem was, we'll update it here. I will close this issue. thank you.

FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04

ARG HTTP_PROXY
ARG HTTPS_PROXY
ARG http_proxy

RUN apt-get update && apt-get install --reinstall -y locales && locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US
ENV LC_ALL en_US.UTF-8

RUN apt-get update && apt-get install -y --no-install-recommends \
         build-essential \
         cmake \
         git \
         curl \
         vim \
         ca-certificates \
         libjpeg-dev \
             libpng16-16 \
             libtiff5 \
         libpng-dev \
         python-dev \
         python3.5 \
         python3.5-dev \
         python-networkx \
         python-setuptools \
         python3-setuptools \
         python-pip \
         python3-pip && \
         pip install --upgrade "pip < 21.0" && \
         pip3 install --upgrade "pip < 21.0" && \
         rm -rf /var/lib/apt/lists/*

# installing conda
RUN curl -o ~/miniconda.sh -LO https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
     chmod +x ~/miniconda.sh && \
     ~/miniconda.sh -b -p /opt/conda && \
     rm ~/miniconda.sh && \
     /opt/conda/bin/conda clean -ya && \
     /opt/conda/bin/conda create -n python37 python=3.7 numpy networkx scipy six requests

RUN packages='py_trees==0.8.3 shapely six dictor requests ephem tabulate' \
        && pip3 install ${packages}

WORKDIR /workspace
COPY .tmp/PythonAPI /workspace/CARLA/PythonAPI
ENV CARLA_ROOT /workspace/CARLA

ENV PATH "/workspace/CARLA/PythonAPI/carla/dist/carla-leaderboard-py3x.egg":/opt/conda/envs/python37/bin:/opt/conda/envs/bin:$PATH

# adding CARLA egg to default python environment
RUN pip install --user setuptools py_trees==0.8.3 psutil shapely six dictor requests ephem tabulate

ENV SCENARIO_RUNNER_ROOT "/workspace/scenario_runner"
ENV LEADERBOARD_ROOT "/workspace/leaderboard"
ENV TEAM_CODE_ROOT "/workspace/team_code"
ENV PYTHONPATH "/workspace/CARLA/PythonAPI/carla/dist/carla-leaderboard-py3x.egg":"${SCENARIO_RUNNER_ROOT}":"${CARLA_ROOT}/PythonAPI/carla":"${LEADERBOARD_ROOT}":${PYTHONPATH}

COPY .tmp/scenario_runner ${SCENARIO_RUNNER_ROOT}
COPY .tmp/leaderboard ${LEADERBOARD_ROOT}
COPY .tmp/team_code ${TEAM_CODE_ROOT}

RUN mkdir -p /workspace/results
RUN chmod +x /workspace/leaderboard/scripts/run_evaluation.sh

########################################################################################################################
########################################################################################################################
############                                BEGINNING OF USER COMMANDS                                      ############
########################################################################################################################
########################################################################################################################

ENV TEAM_AGENT ${TEAM_CODE_ROOT}/npc_agent.py
ENV TEAM_CONFIG ${TEAM_CODE_ROOT}/YOUR_CONFIG_FILE

COPY .tmp/carla_project /workspace/carla_project

ENV PYTHONPATH "/workspace/CARLA/PythonAPI/carla/dist/carla-leaderboard-py3x.egg":${PYTHONPATH}
ENV PYTHONPATH "/workspace/":${PYTHONPATH}
ENV TEAM_AGENT /workspace/team_code/image_agent.py
ENV TEAM_CONFIG /workspace/team_code/model.ckpt
ENV CHALLENGE_TRACK_CODENAME SENSORS

RUN apt-get update && apt-get install -y --no-install-recommends \
         libgtk2.0-dev

ENV PATH="/opt/conda/bin:${PATH}"
RUN /opt/conda/envs/python37/bin/pip install -r /workspace/carla_project/requirements.txt
RUN /opt/conda/bin/conda install -c anaconda libgcc=7.2.0
RUN /opt/conda/bin/conda init bash

RUN cp /opt/conda/lib/libstdc++.so.6.0.28 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
ENV CONDA_DEFAULT_ENV python37
ENV PATH /opt/conda/envs/python37/bin:$PATH

RUN /opt/conda/envs/python37/bin/pip install tabulate ephem

########################################################################################################################
########################################################################################################################
############                                   END OF USER COMMANDS                                         ############
########################################################################################################################
########################################################################################################################

ENV SCENARIOS ${LEADERBOARD_ROOT}/data/all_towns_traffic_scenarios_public.json
ENV ROUTES ${LEADERBOARD_ROOT}/data/routes_training.xml
ENV REPETITIONS 1
ENV CHECKPOINT_ENDPOINT /workspace/results/results.json
ENV DEBUG_CHALLENGE 0

ENV HTTP_PROXY ""
ENV HTTPS_PROXY ""
ENV http_proxy ""
ENV https_proxy ""

CMD ["/bin/bash"]