search

bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform
https://birdhouse-deploy.readthedocs.io/en/latest/
Apache License 2.0
4 stars 6 forks source link

Add the `promtail` and `loki` optional components #474

Closed mishaschwartz closed 1 week ago

mishaschwartz commented 2 weeks ago

Overview

Promtail and loki are components of the Grafana stack (see the monitoring component at components/monitoring) that parses and ships logs to Grafana through loki.

It can also generate prometheus metrics from the logs that it parses that can be ingested by the monitoring Prometheus instance (the one created by the components/monitoring component).

Promtail can be deployed separately from loki if you just want to generate prometheus metrics.

For instructions on how to configure these components and start generating metrics see the documentation in optional-components/README.md.

Update the prometheus version to the current latest v2.53.3. This is required to support loading multiple prometheus scrape configuration files with the scrape_config_files configuration option.

Changes

Non-breaking changes

Breaking changes

None

Related Issue / Discussion

Additional Information

Links to other issues or sources.

CI Operations

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

crim-jenkins-bot commented 2 weeks ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2873/
Result :white_check_mark: SUCCESS

BIRDHOUSE_DEPLOY_BRANCH : prometheus-log-parser-loki
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1736/

NOTEBOOK TEST RESULTS 
    
[2024-11-07T16:16:35.624Z] ============================= test session starts ==============================
[2024-11-07T16:16:35.624Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-11-07T16:16:35.624Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-11-07T16:16:35.625Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-11-07T16:16:35.625Z] collected 301 items
[2024-11-07T16:16:35.625Z] 
[2024-11-07T16:16:46.805Z] notebooks-auth/geoserver.ipynb ..................                        [  5%]
[2024-11-07T16:18:04.533Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  9%]
[2024-11-07T16:18:04.533Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-11-07T16:18:48.081Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-11-07T16:18:56.954Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 17%]
[2024-11-07T16:19:05.672Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 19%]
[2024-11-07T16:26:44.187Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 23%]
[2024-11-07T16:26:44.187Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 23%]
[2024-11-07T16:26:44.187Z] ...............                                                          [ 28%]
[2024-11-07T16:26:51.888Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 30%]
[2024-11-07T16:26:59.966Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .......           [ 32%]
[2024-11-07T16:27:15.921Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 34%]
[2024-11-07T16:27:21.537Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 36%]
[2024-11-07T16:27:26.298Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 38%]
[2024-11-07T16:30:49.590Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 43%]
[2024-11-07T16:31:54.583Z] .............                                                            [ 47%]
[2024-11-07T16:31:58.792Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2024-11-07T16:32:00.319Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 51%]
[2024-11-07T16:32:17.706Z] .................                                                        [ 57%]
[2024-11-07T16:32:25.297Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 59%]
[2024-11-07T16:32:26.239Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 59%]
[2024-11-07T16:32:43.831Z] .........                                                                [ 62%]
[2024-11-07T16:32:53.358Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 66%]
[2024-11-07T16:33:02.745Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 68%]
[2024-11-07T16:33:04.132Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 68%]
[2024-11-07T16:33:07.520Z] .....                                                                    [ 70%]
[2024-11-07T16:33:25.644Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 70%]
[2024-11-07T16:33:42.388Z] ............                                                             [ 74%]
[2024-11-07T16:34:00.527Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-11-07T16:34:26.586Z] .....s.                                                                  [ 77%]
[2024-11-07T16:34:33.162Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 77%]
[2024-11-07T16:34:50.634Z] ..                                                                       [ 78%]
[2024-11-07T16:34:57.245Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 78%]
[2024-11-07T16:35:58.707Z] .........                                                                [ 81%]
[2024-11-07T16:36:08.715Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-11-07T16:41:15.047Z] ....                                                                     [ 83%]
[2024-11-07T16:41:15.047Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 83%]
[2024-11-07T16:41:15.047Z] ....                                                                     [ 85%]
[2024-11-07T16:41:15.047Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-11-07T16:41:20.006Z] .....                                                                    [ 87%]
[2024-11-07T16:41:24.209Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-11-07T16:41:38.024Z] .............                                                            [ 91%]
[2024-11-07T16:41:42.226Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-11-07T16:41:43.641Z] ......                                                                   [ 94%]
[2024-11-07T16:41:46.135Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-11-07T16:44:25.790Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-11-07T16:44:25.790Z] 
[2024-11-07T16:44:25.790Z] =============================== warnings summary ===============================
crim-jenkins-bot commented 2 weeks ago

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2874/
Result :white_check_mark: SUCCESS

BIRDHOUSE_DEPLOY_BRANCH : prometheus-log-parser-loki
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1737/

NOTEBOOK TEST RESULTS 
    
[2024-11-07T18:05:44.849Z] ============================= test session starts ==============================
[2024-11-07T18:05:44.849Z] platform linux -- Python 3.11.6, pytest-8.2.0, pluggy-1.5.0
[2024-11-07T18:05:44.849Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-11-07T18:05:44.849Z] plugins: anyio-4.3.0, dash-2.17.0, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0
[2024-11-07T18:05:44.849Z] collected 301 items
[2024-11-07T18:05:44.849Z] 
[2024-11-07T18:05:55.611Z] notebooks-auth/geoserver.ipynb ..................                        [  5%]
[2024-11-07T18:07:00.299Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  9%]
[2024-11-07T18:07:00.829Z] notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
[2024-11-07T18:07:47.627Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
[2024-11-07T18:07:56.925Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 17%]
[2024-11-07T18:08:05.585Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 19%]
[2024-11-07T18:15:36.359Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 23%]
[2024-11-07T18:15:36.359Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 23%]
[2024-11-07T18:15:36.885Z] ...............                                                          [ 28%]
[2024-11-07T18:15:44.621Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 30%]
[2024-11-07T18:15:51.319Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .......           [ 32%]
[2024-11-07T18:16:07.072Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 34%]
[2024-11-07T18:16:19.308Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 36%]
[2024-11-07T18:16:24.278Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 38%]
[2024-11-07T18:19:52.080Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 43%]
[2024-11-07T18:21:06.241Z] .............                                                            [ 47%]
[2024-11-07T18:21:08.205Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2024-11-07T18:21:10.116Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 51%]
[2024-11-07T18:21:26.266Z] .................                                                        [ 57%]
[2024-11-07T18:21:34.633Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 59%]
[2024-11-07T18:21:36.029Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 59%]
[2024-11-07T18:21:53.620Z] .........                                                                [ 62%]
[2024-11-07T18:22:02.694Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 66%]
[2024-11-07T18:22:11.824Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 68%]
[2024-11-07T18:22:13.216Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 68%]
[2024-11-07T18:22:16.875Z] .....                                                                    [ 70%]
[2024-11-07T18:22:38.869Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 70%]
[2024-11-07T18:22:54.501Z] ............                                                             [ 74%]
[2024-11-07T18:23:12.657Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
[2024-11-07T18:23:39.351Z] .....s.                                                                  [ 77%]
[2024-11-07T18:23:47.503Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 77%]
[2024-11-07T18:24:03.818Z] ..                                                                       [ 78%]
[2024-11-07T18:24:11.998Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 78%]
[2024-11-07T18:25:14.607Z] .........                                                                [ 81%]
[2024-11-07T18:25:24.613Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
[2024-11-07T18:30:00.594Z] ....                                                                     [ 83%]
[2024-11-07T18:30:01.632Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 83%]
[2024-11-07T18:30:07.358Z] ....                                                                     [ 85%]
[2024-11-07T18:30:11.747Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
[2024-11-07T18:30:17.782Z] .....                                                                    [ 87%]
[2024-11-07T18:30:22.000Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
[2024-11-07T18:30:34.680Z] .............                                                            [ 91%]
[2024-11-07T18:30:38.905Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
[2024-11-07T18:30:40.397Z] ......                                                                   [ 94%]
[2024-11-07T18:30:42.890Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2024-11-07T18:33:17.038Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-11-07T18:33:17.038Z] 
[2024-11-07T18:33:17.038Z] =============================== warnings summary ===============================
mishaschwartz commented 1 week ago

Closed in favour of #473 (as discussed in #444)