esnet / sense-rtmon

Custom Scripts for Dynamic End-To-End Flow-Specific Grafana Dashboards
5 stars 4 forks source link

Containerize the main loop scripts #57

Closed xi-yang closed 8 months ago

xi-yang commented 11 months ago

The main loops and associated artifacts should be packaged into a controller Docker image. This will then be started as part of the docker stack. Validation of the containerization must consider:

  1. autonomous run and handling of arbitrary flows that are added and removed I dynamically
  2. health monitoring and crash proof / auto restart for continuous 24/7 operations.
  3. logging with meaningful messages for operators observations / monitoring
  4. online global reconfiguration change utils (without shutdown of the service)
sunami09 commented 10 months ago

BUILD

[root@ip-172-31-72-189 cloud]# docker images
REPOSITORY                   TAG       IMAGE ID       CREATED         SIZE
nginx                        <none>    2a36393edaf1   2 weeks ago     187MB
grafana/grafana-enterprise   <none>    e52ba16e8c2e   2 weeks ago     416MB
ricoberger/script_exporter   v2.16.0   e04dc11e3586   4 weeks ago     27.8MB
grafana/grafana-enterprise   <none>    a12118839714   6 weeks ago     416MB
nginx                        latest    d453dd892d93   2 months ago    187MB
prom/pushgateway             latest    a30191ce4597   3 months ago    20.8MB
ricoberger/script_exporter   v2.5.2    16a9b2fc9798   20 months ago   22.3MB
prom/prometheus              v2.2.1    cc866859f8df   5 years ago     113MB
[root@ip-172-31-72-189 cloud]# docker build -t mainloop .
[+] Building 38.2s (13/13) FINISHED                                                                                                                                          docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                   0.0s
 => => transferring dockerfile: 490B                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/python:3.6                                                                                                                          0.3s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                          0.0s
 => [1/7] FROM docker.io/library/python:3.6@sha256:f8652afaf88c25f0d22354d547d892591067aa4026a7fa9a6819df9f300af6fc                                                                   25.3s
 => => resolve docker.io/library/python:3.6@sha256:f8652afaf88c25f0d22354d547d892591067aa4026a7fa9a6819df9f300af6fc                                                                    0.0s
 => => sha256:f8652afaf88c25f0d22354d547d892591067aa4026a7fa9a6819df9f300af6fc 1.86kB / 1.86kB                                                                                         0.0s
 => => sha256:cb5b7ae361722f070eca53f35823ed21baa85d61d5d95cd5a95ab53d740cdd56 10.87MB / 10.87MB                                                                                       0.2s
 => => sha256:d097a4907a8ec079df5ac31872359c2de510f82214c0448e926393b376d3b60d 2.22kB / 2.22kB                                                                                         0.0s
 => => sha256:54260638d07c5e3ad24c6e21fc889abbc8486a27634c0892086ff71f3f44b104 9.27kB / 9.27kB                                                                                         0.0s
 => => sha256:0e29546d541cdbd309281d21a73a9d1db78665c1b95b74f32b009e0b77a6e1e3 54.92MB / 54.92MB                                                                                       0.7s
 => => sha256:9b829c73b52b92b97d5c07a54fb0f3e921995a296c714b53a32ae67d19231fcd 5.15MB / 5.15MB                                                                                         0.2s
 => => sha256:6f9f74896dfa93fe0172f594faba85e0b4e8a0481a0fefd9112efc7e4d3c78f7 196.51MB / 196.51MB                                                                                     2.8s
 => => sha256:6494e4811622b31c027ccac322ca463937fd805f569a93e6f15c01aade718793 54.57MB / 54.57MB                                                                                       0.9s
 => => extracting sha256:0e29546d541cdbd309281d21a73a9d1db78665c1b95b74f32b009e0b77a6e1e3                                                                                              4.4s
 => => sha256:5e3b1213efc56598e78bd602983945c164de2a37205e06a62dada823124dc743 6.29MB / 6.29MB                                                                                         0.9s
 => => sha256:9fddfdc56334f2e6efad7e241bf5e7459c40ed105c5478676f41c1244bd96752 14.21MB / 14.21MB                                                                                       1.1s
 => => sha256:404f02044bac0432ca522cbb9f254b1c91fcea6806bfeef0be0b243b2f31bab7 235B / 235B                                                                                             1.0s
 => => sha256:c4f42be2be53b900ebffc040c1df13de538434ccc5f5d954a56848a6169a3a3f 2.21MB / 2.21MB                                                                                         1.1s
 => => extracting sha256:9b829c73b52b92b97d5c07a54fb0f3e921995a296c714b53a32ae67d19231fcd                                                                                              0.4s
 => => extracting sha256:cb5b7ae361722f070eca53f35823ed21baa85d61d5d95cd5a95ab53d740cdd56                                                                                              0.5s
 => => extracting sha256:6494e4811622b31c027ccac322ca463937fd805f569a93e6f15c01aade718793                                                                                              4.3s
 => => extracting sha256:6f9f74896dfa93fe0172f594faba85e0b4e8a0481a0fefd9112efc7e4d3c78f7                                                                                             12.3s
 => => extracting sha256:5e3b1213efc56598e78bd602983945c164de2a37205e06a62dada823124dc743                                                                                              0.5s
 => => extracting sha256:9fddfdc56334f2e6efad7e241bf5e7459c40ed105c5478676f41c1244bd96752                                                                                              0.9s
 => => extracting sha256:404f02044bac0432ca522cbb9f254b1c91fcea6806bfeef0be0b243b2f31bab7                                                                                              0.0s
 => => extracting sha256:c4f42be2be53b900ebffc040c1df13de538434ccc5f5d954a56848a6169a3a3f                                                                                              0.4s
 => [internal] load build context                                                                                                                                                      0.4s
 => => transferring context: 21.68MB                                                                                                                                                   0.4s
 => [2/7] WORKDIR /app/cloud/                                                                                                                                                          6.5s
 => [3/7] COPY . /app/cloud/                                                                                                                                                           0.2s
 => [4/7] COPY config.yml /app/config_cloud/config.yml                                                                                                                                 0.0s
 => [5/7] COPY privkey.pem /etc/letsencrypt/live/dev2.virnao.com/privkey.pem                                                                                                           0.1s
 => [6/7] COPY cert.pem /etc/letsencrypt/live/dev2.virnao.com/cert.pem                                                                                                                 0.0s
 => [7/7] RUN pip install --no-cache-dir -r requirements.txt                                                                                                                           5.4s
 => exporting to image                                                                                                                                                                 0.2s
 => => exporting layers                                                                                                                                                                0.2s
 => => writing image sha256:ed09fce9bdc4ab94a9036b62fae93cf10ec8444e423d481f11c627482a93cc08                                                                                           0.0s
 => => naming to docker.io/library/mainloop                                                                                                                                            0.0s
[root@ip-172-31-72-189 cloud]# docker images
REPOSITORY                   TAG       IMAGE ID       CREATED          SIZE
mainloop                     latest    ed09fce9bdc4   11 seconds ago   934MB
nginx                        <none>    2a36393edaf1   2 weeks ago      187MB
grafana/grafana-enterprise   <none>    e52ba16e8c2e   2 weeks ago      416MB
ricoberger/script_exporter   v2.16.0   e04dc11e3586   4 weeks ago      27.8MB
grafana/grafana-enterprise   <none>    a12118839714   6 weeks ago      416MB
nginx                        latest    d453dd892d93   2 months ago     187MB
prom/pushgateway             latest    a30191ce4597   3 months ago     20.8MB
ricoberger/script_exporter   v2.5.2    16a9b2fc9798   20 months ago    22.3MB
prom/prometheus              v2.2.1    cc866859f8df   5 years ago      113MB

Dockerfile


FROM python:3.6
# Set working directory
WORKDIR /app/cloud/

# Copy current directory
COPY . /app/cloud/

# Copy config_cloud directory
COPY config.yml /app/config_cloud/config.yml

COPY config_flow /app/config_flow

COPY privkey.pem /etc/letsencrypt/live/dev2.virnao.com/privkey.pem

COPY cert.pem /etc/letsencrypt/live/dev2.virnao.com/cert.pem

COPY .sense-o-auth.yaml /root/.sense-o-auth.yaml

RUN pip install --no-cache-dir -r requirements.txt

ENTRYPOINT ["python3", "main.py"]

requirements.txt

ansible-core==2.11.12
attrs==22.2.0
bcrypt==4.0.1
certifi==2022.5.18.1
cffi==1.15.1
charset-normalizer==2.0.12
cryptography==40.0.1
docutils==0.18.1
future==0.18.3
idna==3.3
importlib-metadata==4.8.3
iniconfig==1.1.1
isodate==0.6.1
Jinja2==3.0.3
lockfile==0.12.2
MarkupSafe==2.0.1
netifaces==0.11.0
packaging==21.3
paho-mqtt==1.6.1
paramiko==3.1.0
pexpect==4.8.0
pluggy==1.0.0
prometheus-client==0.14.1
psutil==5.9.4
ptyprocess==0.7.0
py==1.11.0
pyaml==21.10.1
pycparser==2.21
pycurl==7.45.2
PyNaCl==1.5.0
pyOpenSSL==23.1.1
pyparsing==3.0.9
pytest==7.0.1
python-daemon==2.3.2
python-dateutil==2.8.2
PyYAML==6.0
rdflib==5.0.0
repoze.lru==0.7
requests==2.27.1
resolvelib==0.5.4
Routes==2.5.1
semantic-version==2.10.0
sense-o-api==1.26
setuptools-rust==1.1.2
simplejson==3.18.4
six==1.16.0
tomli==1.2.3
typing_extensions==4.1.1
urllib3==1.26.9
zipp==3.6.0
sunami09 commented 8 months ago

Moved to #81