owid / owid-grapher

A platform for creating interactive data visualizations
https://ourworldindata.org
MIT License
1.35k stars 227 forks source link

Baking on staging servers sometimes freezes and times out #3699

Open Marigold opened 2 weeks ago

Marigold commented 2 weeks ago

Description

Grapher baking on our staging servers sometimes freezes when it is baking charts. Then it gets killed by Buildkite after 30 minutes. The output doesn't contain anything useful. Here's an example

  | 2024-06-10 10:52:37 CEST | /home/owid/live-data/bakedSite/grapher/social-expenditure-as-percentage-of-gdp.html
  | 2024-06-10 10:52:37 CEST | bake grapher page [] 3996/4582 113.7s 35/s 16.7s slug social-expenditure-as-percentage-of-gdp
  | 2024-06-10 10:52:37 CEST | /home/owid/live-data/bakedSite/grapher/so-emissions-by-world-region-in-million-tonnes.html
  | 2024-06-10 10:52:37 CEST | bake grapher page [] 3997/4582 113.8s 35/s 16.7s slug so-emissions-by-world-region-in-million-tonnes
  | 2024-06-10 10:52:37 CEST | /home/owid/live-data/bakedSite/grapher/soil-lifespans.html
  | 2024-06-10 10:52:37 CEST | bake grapher page [] 3998/4582 113.8s 35/s 16.6s slug soil-lifespans
  | 2024-06-10 11:18:04 CEST | # Received cancellation signal, interrupting
  | 2024-06-10 11:18:04 CEST |  
  | 2024-06-10 11:18:04 CEST | Session terminated, killing shell... ...killed.

Expected behaviour

It shouldn't freeze or it should at least print an informative error.

Steps to reproduce

It happens randomly, unfortunately. I'll be collecting logs from future failures in this issue.

Additional context

image
Marigold commented 2 weeks ago

Happened again in this build

2024-06-12 17:47:55 CEST | bake grapher page [] 4122/4585 97.4s 42/s 10.9s slug tax-revenues-as-a-share-of-gdp-unsdg
-- | --
  | 2024-06-12 17:47:55 CEST | /home/owid/live-data/bakedSite/grapher/tax-revenue-share-gdp-oecd-grsd.html
  | 2024-06-12 17:47:55 CEST | bake grapher page [] 4123/4585 97.4s 42/s 10.9s slug tax-revenue-share-gdp-oecd-grsd
  | 2024-06-12 17:47:55 CEST | /home/owid/live-data/bakedSite/grapher/tax-revenues-vs-income-inequality.html
  | 2024-06-12 17:47:55 CEST | bake grapher page [] 4124/4585 97.4s 42/s 10.9s slug tax-revenues-vs-income-inequality
  | 2024-06-12 17:47:55 CEST | /home/owid/live-data/bakedSite/grapher/tax-revenues-as-a-share-of-gdp-unu-wider.html
  | 2024-06-12 17:47:55 CEST | bake grapher page [] 4125/4585 97.5s 42/s 10.9s slug tax-revenues-as-a-share-of-gdp-unu-wider
  | 2024-06-12 17:47:55 CEST | /home/owid/live-data/bakedSite/grapher/taxes-on-income-vs-taxes-on-goods-and-services.html
  | 2024-06-12 17:47:55 CEST | bake grapher page [] 4126/4585 97.5s 42/s 10.8s slug taxes-on-income-vs-taxes-on-goods-and-services
  | 2024-06-12 17:47:55 CEST | /home/owid/live-data/bakedSite/grapher/taxes-as-share-of-cigarette-price.html
  | 2024-06-12 17:47:55 CEST | bake grapher page [] 4127/4585 97.5s 42/s 10.8s slug taxes-as-share-of-cigarette-price
  | 2024-06-12 17:47:55 CEST | /home/owid/live-data/bakedSite/grapher/taxes-on-goods-and-services-gdp.html
  | 2024-06-12 17:47:55 CEST | bake grapher page [] 4128/4585 97.6s 42/s 10.8s slug taxes-on-goods-and-services-gdp
  | 2024-06-12 17:47:55 CEST | /home/owid/live-data/bakedSite/grapher/taxes-on-incomes-of-individuals-and-corporations-gdp.html
  | 2024-06-12 17:47:55 CEST | bake grapher page [] 4129/4585 97.6s 42/s 10.8s slug taxes-on-incomes-of-individuals-and-corporations-gdp
  | 2024-06-12 17:47:55 CEST | /home/owid/live-data/bakedSite/grapher/tb-patients-living-with-hiv-receiving-art.html
  | 2024-06-12 18:12:08 CEST | # Received cancellation signal, interrupting
  | 2024-06-12 18:12:08 CEST |  
  | 2024-06-12 18:12:08 CEST | Session terminated, killing shell... ...killed.
Marigold commented 1 week ago

Another here

2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/co2/country/north-korea.html
-- | --
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/energy/country/north-korea.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/coronavirus/country/north-korea.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/co2/country/portugal.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/energy/country/portugal.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/coronavirus/country/portugal.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/co2/country/paraguay.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/energy/country/paraguay.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/coronavirus/country/paraguay.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/co2/country/palestine.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/energy/country/palestine.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/coronavirus/country/palestine.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/co2/country/french-polynesia.html
  | 2024-06-18 12:56:31 CEST | /home/owid/live-data/bakedSite/energy/country/french-polynesia.html
  | 2024-06-18 12:56:32 CEST | /home/owid/live-data/bakedSite/coronavirus/country/french-polynesia.html
  | 2024-06-18 12:56:32 CEST | /home/owid/live-data/bakedSite/co2/country/qatar.html
  | 2024-06-18 12:56:32 CEST | /home/owid/live-data/bakedSite/energy/country/qatar.html
  | 2024-06-18 13:25:01 CEST | # Received cancellation signal, interrupting
  | 2024-06-18 13:25:01 CEST |  
  | 2024-06-18 13:25:01 CEST | Session terminated, killing shell... ...killed.
  | 2024-06-18 13:25:03 CEST | Terminated
  | 2024-06-18 13:25:03 CEST | 🚨 Error: The command was interrupted by a signal