search

jupyterhub / mybinder.org-deploy

Deployment config files for mybinder.org
https://mybinder-sre.readthedocs.io/en/latest/index.html
BSD 3-Clause "New" or "Revised" License
76 stars 74 forks source link

Non-running build pods accumulating on GESIS node #2981

Closed rgaiacs closed 4 months ago

rgaiacs commented 4 months ago

This is a following up to https://github.com/jupyterhub/mybinder.org-deploy/issues/2979

The GESIS node is accumulating non-running build pods:

Screenshot 2024-05-14 at 09-06-29 Node Activity - Dashboards - Grafana

kubectl -n gesis get pods | grep build- outputs

build-aaronwatters-2dh5gizmos-994e37-bfe-ff                       0/1     Completed   0                14h
build-aazuspan-2deerepr-f2c0dc-c6c-59                             0/1     Completed   0                8h
build-acpennlab-2dstatistics-2dresampling-2donlin-9a8cd8-237-8b   0/1     Completed   0                8h
build-acpennlab-2dstatistics-2dresampling-2donlin-9a8cd8-85c-4c   0/1     Error       0                8h
build-adkchappy-2dmidwest-5frainfall-7c60cf-37b-f5                0/1     Error       0                9h
build-alan-2dturing-2dinstitute-2dsktime-486400-ebd-c6            0/1     Error       0                8h
build-allendowney-2delementsofdatascience-869ce1-286-8b           1/1     Running     0                15m
build-anaegel-2dpubchem-2dinterface-789081-592-3e                 0/1     Completed   0                12h
build-anitagraser-2dmovingpandas-54a8f6-311-c7                    0/1     Completed   0                14h
build-apache-2dspark-be99c5-28c-46                                0/1     OOMKilled   0                3h22m
build-apache-2dspark-be99c5-b14-dd                                0/1     OOMKilled   0                13h
build-apache-2dspark-be99c5-c09-ac                                0/1     OOMKilled   0                11m
build-apache-2dspark-be99c5-e6c-52                                0/1     OOMKilled   0                48m
build-automating-2dgis-2dprocesses-2dsite-526dbd-2a2-f8           0/1     OOMKilled   0                8h
build-blockworks-2dfoundation-2dmango-2dv4-310332-34d-67          0/1     Error       0                3h28m
build-bloomberg-2dipydatagrid-1cbd5b-23b-e4                       0/1     Error       0                35m
build-c7859-2dneurocomputing-5f-2dlab-2dinferior--e57ae2-488-ef   0/1     Completed   0                16h
build-carlgogo-2dvip-5fextras-d884f6-a7e-28                       0/1     Error       0                14h
build-computational-2dsediment-2dhyd-2driver-2dfl-f501b2-df2-d2   0/1     Completed   0                11h
build-curricularanalytics-2dca-2dnotebooks-b56ff7-462-80          0/1     Error       0                4h25m
build-dalpsychneuro-2dnesc-5f3505-5ftextbook-3f1085-1fe-7f        0/1     OOMKilled   0                8h
build-dask-2ddask-2dexamples-49f972-0f9-3d                        0/1     Error       0                74m
build-data-2dgymnasia-2djulia-2dbinder-c28e16-df2-91              0/1     Error       0                15h
build-dataverse-2ddvn-2d2fh8sfd2-ab2a90-342-2c                    0/1     OOMKilled   0                14h
build-dataverse-2ddvn-2d2fvq09a0-0a6edd-809-c4                    0/1     OOMKilled   0                12h
build-dcos-2dlabs-2ddata-2dscience-74f55c-891-a1                  0/1     Error       0                9h
build-deltares-2dhydromt-e49598-39c-4a                            0/1     Completed   0                11h
build-felipelema-2dmne-2dbinder-9aed64-d70-80                     0/1     Error       0                9h
build-franceme-2dcryptoguard-696b69-dfb-30                        0/1     Error       0                12m
build-fslaborg-2dxplot-4309fe-1e4-14                              0/1     Error       0                11h
build-fteicht-2dscikit-2ddecide-660aee-6ee-28                     0/1     Completed   0                11h
build-glaciohack-2dxdem-883c05-2c9-db                             0/1     Completed   0                4h54m
build-hainm-2dnglview-2dnotebooks-4a7f71-e88-76                   0/1     Error       0                14h
build-healthonrails-2dannolid-400d98-936-25                       0/1     OOMKilled   0                116m
build-hessehskempten-2d23s-5fstatbat-d914e4-aa2-e6                0/1     Completed   0                12h
build-hughshanahan-2dgraphalgos-65d15f-705-a1                     0/1     Error       0                13h
build-indian-2dinstitute-2dof-2dscience-2dqc-2dqi-9e335d-f32-4e   0/1     Error       0                7h8m
build-iubh-2dbsta01-2d01-b247eb-22e-0b                            0/1     Error       0                13h
build-jarnorfb-2dincense-a96517-bc7-c4                            0/1     Error       0                100m
build-joaquinamatrodrigo-2dskforecast-6ef0a3-6f3-c7               0/1     Completed   0                5h7m
build-jorgensd-2ddolfinx-2dtutorial-bf3b5f-ccb-da                 0/1     Completed   0                10h
build-jtpio-2dnotebook-8ff807-30f-c5                              0/1     Error       0                6h43m
build-jupyterlab-2djupyterlab-2ddemo-b2f8d0-496-88                0/1     Error       0                16h
build-kinoshitadaisuke-2dncu-5fmoon-5f202209-4d6817-544-c2        0/1     Completed   0                10h
build-landscapegeoinformatics-2dcsv-5fboxplots-5f-3bd02d-972-dd   0/1     Completed   0                22m
build-landscapegeoinformatics-2dlc-5fmetrics-5fre-636ab6-e58-a3   0/1     Error       0                11h
build-liam-2ddeacon-2dphaseshifts-951ba5-346-90                   0/1     Completed   0                10h
build-luizgh-2dsigver-29768b-6f8-da                               0/1     Error       0                14h
build-mahendrapaipuri-2djupyter-2dfs-06cea8-078-5b                0/1     Completed   0                14h
build-mdtanker-2dpolartoolkit-ab3134-bc1-02                       0/1     OOMKilled   0                169m
build-microsoft-2dhi-2dml-378b89-e31-2d                           0/1     Completed   0                107m
build-networkx-2dnx-2dguides-3a76db-e97-8e                        0/1     Error       0                158m
build-neuroinflab-2dpymice-b8610f-0fc-b5                          0/1     Error       0                5h34m
build-neurotechx-2deeg-2dexpy-639d7d-da1-1c                       0/1     Error       0                15h
build-pandas-2dprofiling-2dpandas-2dprofiling-32e002-0ec-a0       0/1     Completed   0                9h
build-pantelis-2dartificial-2dintelligence-97f8a4-a5a-87          0/1     OOMKilled   0                127m
build-piquil-2dqucumber-31a291-be2-89                             0/1     Error       0                12h
build-poloclub-2dwizmap-cdf230-80c-3d                             0/1     Completed   0                36m
build-probml-2ddynamax-0bd40d-46f-36                              0/1     Completed   0                10h
build-pyqtgraph-2dpyqtgraph-1e5d3e-9cf-21                         0/1     Completed   0                7h44m
build-rochefort-2dlab-2dfissa-3c23c7-8cf-e5                       0/1     Error       0                5h53m
build-rochefort-2dlab-2dfissa-3c23c7-cde-6b                       0/1     Error       0                8h
build-rte-2dfrance-2dgrid2op-41f702-31d-32                        0/1     Error       0                14h
build-scikit-2dimage-2dscikit-2dimage-60f762-16f-79               0/1     Error       0                9h
build-scikit-2dimage-2dscikit-2dimage-60f762-b76-8a               0/1     Error       0                38m
build-scikit-2dimage-2dscikit-2dimage-60f762-bf0-85               0/1     Error       0                13h
build-scikit-2dimage-2dscikit-2dimage-60f762-f5f-83               0/1     Error       0                16h
build-scikit-2dlearn-2dscikit-2dlearn-4c9324-402-67               0/1     Error       0                3h43m
build-scikit-2dlearn-2dscikit-2dlearn-4c9324-63f-db               0/1     Error       0                16h
build-scikit-2dlearn-2dscikit-2dlearn-4c9324-68b-2a               1/1     Running     0                50s
build-spatialaudio-2dnbsphinx-72289e-535-f1                       0/1     Completed   0                12h
build-tatsath-2dfin-2dml-3c8c97-bc1-11                            0/1     Error       0                13h
build-tboch-2dnotebooks-2dplayground-1d7fa6-0ee-00                0/1     Error       0                10h
build-tempobeatdownbeat-2dtutorial-c3fcab-afb-7c                  0/1     Error       0                5h13m
build-tpaviot-2dpythonocc-2dbinderhub-0118ce-0cd-30               1/1     Running     0                37m
build-uc-2dpython-2dintro-2dpython-2ddatasci-1d525b-7e2-2c        0/1     Completed   0                16h
build-ucla-2dbiostat-2d216-2d2022fall-2e8e2b-0de-57               0/1     Error       0                3h34m
build-ucla-2dbiostat216-2d2021fall-2ducla-2dbiost-570028-524-0c   0/1     Error       0                8h
build-valdanchev-2dreproducible-2ddata-2dscience--7afdd4-1fa-bb   0/1     Error       0                13h
build-victoromondi1997-2dblog-a5a175-d7b-00                       0/1     Error       0                8h
build-wikfeldt-2djupyterlab-2ddemo-940dbd-464-f8                  0/1     Error       0                6h52m
build-wolfv-2dmathics-0d3c60-cd8-45                               0/1     Error       0                72m
build-zarr-2ddevelopers-2dzarr-2dpython-3cf6c9-897-9b             0/1     Completed   0                3h53m

When looking at Graphana, this start after https://github.com/jupyterhub/mybinder.org-deploy/pull/2975

Screenshot 2024-05-14 at 09-13-47 Node Activity - Dashboards - Grafana

@manics do you know of any configuration that changed on binderhub? Thanks!

cc @arnim

manics commented 4 months ago

I can't think of anything. The only significant change in https://github.com/jupyterhub/mybinder.org-deploy/pull/2975 was to bump the JupyterHub chart.

rgaiacs commented 4 months ago

@manics Thanks for the reply.

https://github.com/jupyterhub/mybinder.org-deploy/pull/2978 or https://github.com/jupyterhub/mybinder.org-deploy/pull/2982 might have resolved this issue.

Screenshot 2024-05-15 at 08-41-43 Node Activity - Dashboards - Grafana