Poland data missing

[zbraniecki]

Not sure if I should start reporting new issues for Li yet, but https://liproduction-reportsbucket-bhk8fnhv1s76.s3-us-west-1.amazonaws.com/beta/latest/timeseries-byLocation.json from today misses country Poland.

Originally posted by @zbraniecki in https://github.com/covidatlas/li/issues/284#issuecomment-657259227

[jzohrab]

Thanks @zbraniecki , checking.

[jzohrab]

I just checked the report from the above and there appear to be some Polish states, @zbraniecki:

script:

let j = require('./timeseries-byLocation.json')

let pl = j.
    filter(n => n.locationID.toLowerCase().match(/iso1:pl/)).
    map(n => { return {
      locationID: n.locationID,
      name: n.name,
      'ts-2020-07-12': n.timeseries['2020-07-12'],
      'sources-2020-07-12': JSON.stringify(n.timeseriesSources)
    }})

console.log(JSON.stringify(pl, null, 2))

Results:

[
  {
    "locationID": "iso1:pl#iso2:pl-04",
    "name": "Kuyavian-Pomeranian Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 696,
      "deaths": 49,
      "growthFactor": 1
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-06",
    "name": "Lublin Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 814,
      "deaths": 20,
      "growthFactor": 1.05
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-08",
    "name": "Lubusz Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 234,
      "deaths": 0,
      "growthFactor": 1.26
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-10",
    "name": "Łódź Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 3420,
      "deaths": 187,
      "growthFactor": 1.01
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-14",
    "name": "Masovian Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 5377,
      "deaths": 352,
      "growthFactor": 1
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-16",
    "name": "Opole Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 1009,
      "deaths": 53,
      "growthFactor": 1.01
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-18",
    "name": "Subcarpathian Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 896,
      "deaths": 55,
      "growthFactor": 1.05
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-20",
    "name": "Podlaskie Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 902,
      "deaths": 16,
      "growthFactor": 1
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-22",
    "name": "Pomeranian Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 711,
      "deaths": 38,
      "growthFactor": 1.01
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-24",
    "name": "Silesian Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 13805,
      "deaths": 356,
      "growthFactor": 1
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-30",
    "name": "Greater Poland Voivodeship, PL",
    "ts-2020-07-12": {
      "cases": 3230,
      "deaths": 185,
      "growthFactor": 1.03
    },
    "sources-2020-07-12": "{\"2020-05-25..2020-07-12\":\"pl\"}"
  }
]

[jzohrab]

@zbraniecki - Was this data not present at the time you looked, or were you looking for something different?

[zbraniecki]

There are Voivodeships, but not country Poland.

[jzohrab]

Got it, thanks. Not sure why it dropped off. I just ran a crawl and scrape locally and it checked out - iso1:pl did show up, but it's not in the production report. Per status monitoring it did scrape successfully in prod at 2020-07-12 18:47:35 UTC. Looking into it.

[jzohrab]

iso1:pl is in LiProduction-LocationsTable dynamodb. Checking scrape logs, if they haven't expired..

[jzohrab]

Scrape logs there, nothing looks off.

[jzohrab]

Basic mistake on my part. The reports do a table scan of "locations" to get the locations, but the records are paginated. The reports were currently only including the first page of records, which was 2903 locations, and was missing the second and final page, 299 locations. "iso1:pl" happened to be in that second page.

Thanks very much for raising this issue, my mistake. I'll do a fix and will close this ticket when it's launched.

[jzohrab]

The report has been updated in staging, Poland (country) is now showing up in s3://listaging-reportsbucket-1bjqfmfwopcdd/beta/latest/timeseries-byLocation.json.

I'll promote this to prod and it should update soon. I'll leave this open until the report is updated in prod.

[jzohrab]

Hi @zbraniecki , confirmed that PL exists in prod now.

s3://liproduction-reportsbucket-bhk8fnhv1s76/beta/latest/timeseries-byLocation.json extract:

[
  {
    "locationID": "iso1:pl",
    "name": "PL",
    "ts-2020-07-12": "{\"cases\":37891,\"deaths\":1571,\"recovered\":27148,\"growthFactor\":1.01}",
    "sources-2020-07-12": "{\"2020-01-23..2020-05-24\":\"jhu\",\"2020-05-25..2020-07-13\":{\"jhu\":[\"recovered\"],\"pl\":[\"cases\",\"deaths\"]},\"2020-07-14\":\"jhu\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-02",
    "name": "Lower Silesian Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":2974,\"deaths\":150,\"growthFactor\":1}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-04",
    "name": "Kuyavian-Pomeranian Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":696,\"deaths\":49,\"growthFactor\":1}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-06",
    "name": "Lublin Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":814,\"deaths\":20,\"growthFactor\":1.05}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-08",
    "name": "Lubusz Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":234,\"deaths\":0,\"growthFactor\":1.26}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-10",
    "name": "Łódź Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":3420,\"deaths\":187,\"growthFactor\":1.01}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-12",
    "name": "Lesser Poland Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":2048,\"deaths\":45,\"growthFactor\":1.01}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-14",
    "name": "Masovian Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":5377,\"deaths\":352,\"growthFactor\":1}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-16",
    "name": "Opole Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":1009,\"deaths\":53,\"growthFactor\":1.01}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-18",
    "name": "Subcarpathian Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":896,\"deaths\":55,\"growthFactor\":1.05}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-20",
    "name": "Podlaskie Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":902,\"deaths\":16,\"growthFactor\":1}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-22",
    "name": "Pomeranian Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":711,\"deaths\":38,\"growthFactor\":1.01}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-24",
    "name": "Silesian Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":13805,\"deaths\":356,\"growthFactor\":1}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-26",
    "name": "Świętokrzyskie Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":858,\"deaths\":41,\"growthFactor\":1}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-28",
    "name": "Warmian-Masurian Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":282,\"deaths\":1,\"growthFactor\":1}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-30",
    "name": "Greater Poland Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":3230,\"deaths\":185,\"growthFactor\":1.03}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  },
  {
    "locationID": "iso1:pl#iso2:pl-32",
    "name": "West Pomeranian Voivodeship, PL",
    "ts-2020-07-12": "{\"cases\":635,\"deaths\":23,\"growthFactor\":1}",
    "sources-2020-07-12": "{\"2020-05-25..2020-07-13\":\"pl\"}"
  }
]