BDonnot
commented
1 year ago The DockerFile used for building the image of the competition is (it contains all package versions and how to install them in a virtualenv if you prefer):
bdonnot/l2rpn:delft.2023.0
# Use an official Python runtime as a parent image
FROM python:3.10.12
MAINTAINER Benjamin DONNOT <benjamin.donnot@rte-france.com>
ENV DEBIAN_FRONTEND noninteractive
# generic install
RUN apt-get update && \
apt-get install -y \
less \
apt-transport-https \
build-essential \
git \
ssh \
tar \
gzip
# prep python install packages
RUN python3 -m pip install --upgrade setuptools pip wheel
# install some free solvers
RUN apt install -y "coinor-cbc" \
"coinor-clp" \
"glpk-utils=5.0-1" \
"libglpk-dev=5.0-1" \
"ffmpeg" \
libsm6 \
libxext6 \
"graphviz" \
"graphviz-dev"
# # used by all
RUN --mount=type=cache,target=/root/.cache python3 -m pip install numpy==1.23.5 scipy==1.10.1
# # solver for cvxpy (better if installed beforehand)
RUN --mount=type=cache,target=/root/.cache python3 -m pip install "cvxopt==1.3.1" "cylp==0.91.6" # CVXOPT and GLPK ?
RUN --mount=type=cache,target=/root/.cache python3 -m pip install absl-py==1.4.0 \
accel-brain-base==1.1.0 \
acme==2.6.0 \
aiosignal==1.3.1 \
alabaster==0.7.13 \
ale-py==0.8.1 \
alembic==1.11.1 \
antlr4-python3-runtime==4.9.3 \
anyio==3.7.0 \
apache-beam==2.48.0 \
argon2-cffi==21.3.0 \
argon2-cffi-bindings==21.2.0 \
arrow==1.2.3 \
asttokens==2.2.1 \
astunparse==1.6.3 \
async-timeout==4.0.2 \
attrs==21.4.0 \
autopep8==2.0.2 \
AutoROM==0.6.1 \
AutoROM.accept-rom-license==0.6.1 \
Babel==2.12.1 \
backcall==0.2.0 \
bayesian-optimization==1.4.3 \
BayesianOptimization==0.0.0 \
beautifulsoup4==4.12.2 \
bleach==6.0.0 \
blinker==1.6.2 \
blosc2==2.0.0 \
Bottleneck==1.3.7 \
branca==0.6.0 \
cached-property==1.5.2 \
cachetools==5.3.1 \
certifi==2023.5.7 \
cffi==1.15.1 \
cftime==1.6.2 \
charset-normalizer==3.1.0 \
chex==0.1.7 \
click==8.1.3 \
click-plugins==1.1.1 \
cligj==0.7.2 \
cloudpickle==2.2.1 \
cma==3.3.0 \
colorama==0.4.6 \
colorlover==0.3.0 \
comm==0.1.3 \
contextlib2==21.6.0 \
contourpy==1.0.7 \
crcmod==1.7 \
cryptography==41.0.1 \
cufflinks==0.17.3 \
cvxopt==1.3.1 \
cvxpy==1.3.1 \
cycler==0.11.0 \
cylp==0.91.6 \
Cython==0.29.35 \
dask==2023.6.0 \
databricks-cli==0.17.7 \
debugpy==1.6.7 \
decorator==5.1.1 \
deepdiff==6.3.0 \
defusedxml==0.7.1 \
deprecation==2.1.0 \
dgl==1.1.0 \
dglgo==0.0.2 \
dill==0.3.1.1 \
dm-acme==0.4.0 \
dm-env==1.6 \
dm-launchpad==0.5.0 \
dm-sonnet==2.0.1 \
dm-tree==0.1.8 \
dnspython==2.3.0 \
docker==6.1.3 \
docopt==0.6.2 \
docutils==0.20.1 \
ecos==2.0.12 \
entrypoints==0.4 \
et-xmlfile==1.1.0 \
etils==1.3.0 \
exceptiongroup==1.1.1 \
executing==1.2.0 \
Farama-Notifications==0.0.4 \
fastavro==1.7.4 \
fasteners==0.18 \
fastjsonschema==2.17.1 \
filelock==3.12.1 \
Fiona==1.9.4.post1 \
fire==0.5.0 \
Flask==2.3.2 \
flatbuffers==23.5.26 \
flax==0.6.11 \
folium==0.14.0 \
fonttools==4.40.0 \
fqdn==1.5.1 \
frozenlist==1.3.3 \
fsspec==2023.6.0 \
gast==0.4.0 \
geographiclib==2.0 \
geonetworkx==0.5.3 \
geopandas==0.13.2 \
geopy==2.3.0 \
gitdb==4.0.10 \
GitPython==3.1.31 \
google-auth==2.19.1 \
google-auth-oauthlib==1.0.0 \
google-pasta==0.2.0 \
google-vizier==0.1.6 \
googleapis-common-protos==1.59.0 \
greenlet==2.0.2 \
Grid2Op==1.9.1 \
grpcio==1.51.3 \
grpcio-tools==1.51.3 \
grpcio-reflection==1.51.3 \
gunicorn==20.1.0 \
gym==0.26.2 \
gym-notices==0.0.8 \
gymnasium==0.28.1 \
gymnasium-notices==0.0.1 \
h5py==3.8.0 \
hdfs==2.7.0 \
httplib2==0.22.0 \
huggingface-hub==0.16.2 \
hydra-core==1.3.2 \
idna==3.4 \
imageio==2.31.1 \
imagesize==1.4.1 \
importlib-metadata==6.6.0 \
importlib-resources==5.12.0 \
iniconfig==2.0.0 \
ipykernel==6.23.1 \
ipython==8.14.0 \
ipython-genutils==0.2.0 \
ipywidgets==8.0.6 \
isoduration==20.11.0 \
isort==5.12.0 \
itsdangerous==2.1.2 \
jax==0.4.12 \
jax-jumpy==1.0.0 \
jaxlib==0.4.12 \
jedi==0.18.2 \
jill==0.11.3 \
Jinja2==3.1.2 \
joblib==1.2.0 \
josepy==1.13.0 \
jraph==0.0.6.dev0 \
json5==0.9.14 \
jsonpointer==2.3 \
jsonschema==4.17.3 \
jupyter==1.0.0 \
jupyter-console==6.6.3 \
jupyter-events==0.6.3 \
jupyter_client==8.2.0 \
jupyter_core==5.3.0 \
jupyter_server==2.6.0 \
jupyter_server_terminals==0.4.4 \
jupyterlab-pygments==0.2.2 \
jupyterlab-widgets==3.0.7 \
keras==2.12.0 \
kiwisolver==1.4.4 \
l2rpn-baselines==0.6.0.post1 \
lazy_loader==0.2 \
libclang==16.0.0 \
lightgbm==3.3.5 \
LightSim2Grid==0.7.2 \
linopy==0.1.5 \
littleutils==0.2.2 \
llvmlite==0.40.1rc1 \
locket==1.0.0 \
lxml==4.9.2 \
lz4==4.3.2 \
Mako==1.2.4 \
Markdown==3.4.3 \
markdown-it-py==2.2.0 \
MarkupSafe==2.1.3 \
matplotlib==3.7.1 \
matplotlib-inline==0.1.6 \
mdurl==0.1.2 \
mirakuru==2.5.1 \
mistune==2.0.5 \
ml-collections==0.1.1 \
ml-dtypes==0.2.0 \
mlflow==2.4.1 \
mock==5.0.2 \
mpmath==1.3.0 \
msgpack==1.0.5 \
nbclassic==1.0.0 \
nbclient==0.8.0 \
nbconvert==7.4.0 \
nbformat==5.9.0 \
nest-asyncio==1.5.6 \
netCDF4==1.6.4 \
networkx==3.1 \
nevergrad==0.6.0 \
nose==1.3.7 \
notebook==6.5.4 \
notebook_shim==0.2.3 \
numba==0.57.0 \
numexpr==2.8.4 \
numpy==1.23.5 \
numpydoc==1.5.0 \
oauthlib==3.2.2 \
objsize==0.6.1 \
ogb==1.3.6 \
omegaconf==2.3.0 \
open-spiel==1.3 \
opencv-python==4.7.0.72 \
openpyxl==3.1.2 \
opt-einsum==3.3.0 \
optax==0.1.5 \
orbax-checkpoint==0.2.5 \
ordered-set==4.1.0 \
orjson==3.9.1 \
ortools==9.4.1874 \
osqp==0.6.3 \
outdated==0.2.2 \
overrides==7.3.1 \
packaging==23.1 \
pandapower==2.13.1 \
pandas==2.0.2 \
pandocfilters==1.5.0 \
parso==0.8.3 \
partd==1.4.0 \
patsy==0.5.3 \
pexpect==4.8.0 \
pickleshare==0.7.5 \
Pillow==9.5.0 \
platformdirs==3.5.3 \
plotly==5.15.0 \
pluggy==1.0.0 \
ply==3.11 \
port-for==0.6.3 \
portpicker==1.5.2 \
prometheus-client==0.8.0 \
prompt-toolkit==3.0.38 \
proto-plus==1.22.2 \
protobuf==4.23.2 \
psutil==5.9.5 \
ptyprocess==0.7.0 \
pure-eval==0.2.2 \
py-cpuinfo==9.0.0 \
pyarrow==11.0.0 \
pyasn1==0.5.0 \
pyasn1-modules==0.3.0 \
pybind11==2.10.4 \
pycodestyle==2.10.0 \
pycparser==2.21 \
pydantic==1.10.9 \
pydot==1.4.2 \
pygame==2.4.0 \
pygifsicle==1.0.7 \
Pygments==2.15.1 \
PyJWT==2.7.0 \
pymongo==4.3.3 \
Pyomo==6.6.1 \
pyOpenSSL==23.2.0 \
pyparsing==3.0.9 \
pyproj==3.6.0 \
pypsa==0.23.0 \
pyqlearning==1.2.6 \
pyRFC3339==1.1 \
pyrsistent==0.19.3 \
pytest==7.3.2 \
pytest-redis==3.0.2 \
pytest-timeout==2.1.0 \
pytest-tornasync==0.6.0.post2 \
python-dateutil==2.8.2 \
python-gnupg==0.5.0 \
python-json-logger==2.0.7 \
pytz==2023.3 \
PyUtilib==6.0.0 \
PyWavelets==1.4.1 \
PyYAML==6.0 \
pyzmq==25.1.0 \
qdldl==0.1.7 \
qtconsole==5.4.3 \
QtPy==2.3.1 \
querystring-parser==1.2.4 \
rdkit-pypi==2022.9.5 \
redis==4.5.5 \
regex==2023.6.3 \
requests==2.31.0 \
requests-futures==1.0.1 \
requests-oauthlib==1.3.1 \
rfc3339-validator==0.1.4 \
rfc3986-validator==0.1.1 \
rich==13.4.1 \
rpds-py==0.8.7 \
rsa==4.9 \
ruamel.yaml==0.17.31 \
ruamel.yaml.clib==0.2.7 \
safetensors==0.3.1 \
scikit-image==0.21.0 \
scikit-learn==1.2.2 \
scipy==1.10.1 \
scs==3.2.3 \
seaborn==0.12.2 \
semantic-version==2.10.0 \
Send2Trash==1.8.2 \
shap==0.41.0 \
shap2==0.42.0 \
shapely==2.0.1 \
six==1.16.0 \
slicer==0.0.7 \
smmap==5.0.0 \
sniffio==1.3.0 \
snowballstemmer==2.2.0 \
soupsieve==2.4.1 \
spektral==1.3.0 \
Sphinx==7.0.1 \
sphinxcontrib-applehelp==1.0.4 \
sphinxcontrib-devhelp==1.0.2 \
sphinxcontrib-htmlhelp==2.0.1 \
sphinxcontrib-jsmath==1.0.1 \
sphinxcontrib-qthelp==1.0.3 \
sphinxcontrib-serializinghtml==1.1.5 \
SQLAlchemy==1.4.0 \
sqlparse==0.4.4 \
stable-baselines3==2.0.0 \
stack-data==0.6.2 \
statsmodels==0.14.0 \
support-developer==1.0.5 \
sympy==1.12 \
tables==3.8.0 \
tabulate==0.9.0 \
tenacity==8.2.2 \
tensorboard==2.12.3 \
tensorboard-data-server==0.7.0 \
tensorboard-plugin-wit==1.8.1 \
tensorboardX==2.5 \
tensorflow==2.12.0 \
tensorflow-addons==0.20.0 \
tensorflow-estimator==2.12.0 \
tensorflow-gnn==0.5.0 \
tensorflow-io-gcs-filesystem==0.32.0 \
tensorflow-probability==0.20.1 \
tensorstore==0.1.37 \
termcolor==2.3.0 \
terminado==0.17.1 \
threadpoolctl==3.1.0 \
tianshou==0.5.0 \
tifffile==2023.4.12 \
tinycss2==1.2.1 \
tokenizers==0.13.3 \
tomli==2.0.1 \
toolz==0.12.0 \
torch==2.0.1+cpu \
torchaudio==2.0.2+cpu \
torchvision==0.15.2+cpu \
--extra-index-url https://download.pytorch.org/whl/cpu \
tornado==6.3.2 \
tqdm==4.65.0 \
traitlets==5.9.0 \
transformers==4.30.2 \
typeguard==2.13.3 \
typer==0.9.0 \
typing_extensions==4.6.3 \
tzdata==2023.3 \
uri-template==1.2.0 \
urllib3==1.26.16 \
validators==0.20.0 \
wcwidth==0.2.6 \
webcolors==1.13 \
webencodings==0.5.1 \
websocket-client==1.5.3 \
Werkzeug==2.3.6 \
wget==3.2 \
widgetsnbextension==4.0.7 \
wrapt==1.14.1 \
xarray==2023.5.0 \
xbart==0.1.8 \
xgboost==1.7.5 \
zipp==3.15.0 \
zstandard==0.21.0
# need to install chronix2grid "alone" from a github commit (not on pypi yet)
RUN --mount=type=cache,target=/root/.cache python3 -m pip install git+https://github.com/bdonnot/chronix2grid.git@3228bc276ef54d73bcfb46dd7f8e408f2b93bc90
# due to possible incompatibility with gymnasium, I install ray without deps
RUN --mount=type=cache,target=/root/.cache python3 -m pip install ray[data,train,rllib,tune]==2.5.1 --no-deps referencing --no-deps
# need to be i nstalled after torch !
RUN --mount=type=cache,target=/root/.cache python3 -m pip install torch-geometric==2.3.1 \
torch-cluster==1.6.1 \
torch-scatter==2.1.1 \
torch-sparse==0.6.17 \
torch-spline-conv==1.2.2 \
--extra-index-url https://data.pyg.org/whl/torch-2.0.1+cpu.html
# julia
RUN jill install --upstream Official --confirm
RUN julia --eval 'using Pkg; Pkg.add("JuMP"); Pkg.add("KNITRO")'
WORKDIR /
# Make port 80 available to the world outside this container
EXPOSE 80
The full list of available packages (with the DockerFile) and a simple script to install everything will be detailed in the next messages.
The docker image (as of writing) if
l2rpn.idf.2023.1that you can use withdocker pull bdonnot/l2rpn:idf.2023.1Main installed packages
The main packages available on codalab are:
We tried to installed the latest version available of all packages that did not cause conflicts with one another (it's a NP-hard problem... see https://pip.pypa.io/en/stable/topics/more-dependency-resolution/#the-dependency-resolution-problem).
And this is why we had to remove some packages.
Conflicting packages
Some packages could not be installed correctly, among them:
gym: is installed in its latest version, but the maintainer explicitly advise to replacegymwithgymnasium.gymis no more maintained. It is probably the last time we use it for the competition.KerasRLdoes not appear to be on pypiray[rllib]: this is an interesting one... The RL community is switching (more or less smoothly) between gym (no more maintained) to gymnasium (the "fork" from gym maintained by the people maintaining gym for the past few years). For this switch there are two options: a huge work to support latest gymnasium version (0.28.1), this is the path taken by stable-baselines3 or adopting a more gradual approach and moving from gym 0.26.2 to gymnasium 0.26.4 (this is the path taken by ray). Unfortunately, bot packages decides to impose a gymnasium version. And as a gymnasium version cannot be both 0.26.3 AND 0.28.1 at the same time we had to make a choice. We decided to support stable-baselines3 (because for us using the latest gymnasium version is better). We decided however to still installray[[data,train,rllib,tune]==2.5.1without their dependencies. Some things there might not work as expected. Let us know if that's a problem for you.cogmentsupports only protobuf ~3.7 which was released in 2019 (almost 4 years ago) and appears to be incompatible with some other packages needed, especially with some recent tensorflow installation.Other software available
Compared to other competitions and at the request to some participants we also added some other extra things
Other informations
The python version used is cpython "Python 3.10.12 (main, Jul 4 2023, 05:58:45) [GCC 12.2.0] on linux", default one from the official python 3.10.12 docker image on dockerhub.
Some info about the system: "Linux 953e32c3b027 5.11.0-40-generic #44~20.04.2-Ubuntu SMP Tue Oct 26 18:07:44 UTC 2021 x86_64 GNU/Linux"