some deletions are missing country information

Hagellach37 commented 3 months ago

Some contributions of the type deletion are missing the country information. This leads to inconsistent statistics when comparing overall numbers and per country numbers.

Country information is missing for all OSM elements which have been deleted in the sense of OSM:

e.g. the OSM way has been deleted
https://www.openstreetmap.org/way/1260734826/history
v2: the way has been deleted in OSM
this contribution is missing when querying data only Chad

Country information is only available for deletions in the sense of ohsome:

e.g. some tags of an OSM way have been removed, but the way still exists in OSM
https://www.openstreetmap.org/way/512264174/history
v2 holds the tag place=hamlet --> v3 is still visible, only the place tag has been removed

Solution:

need to store country_iso_a3_before information in ClickHouse DB
need to use this country information when querying country stats

This is how the data looks in the stream:

{
  "replicationSequence": 6002312,
  "id": "way/1260734826",
  "majorChange": true,
  "current": {
    "info": {
      "version": 2,
      "changeset": 148598030,
      "timestamp": "2024-03-13T14:55:20Z",
      "user": 378610,
      "visible": false
    },
    "tags": {},
    "geometry": "LINESTRING EMPTY",
    "area": 0,
    "length": 0,
    "centroid": [],
    "countries": [],
    "mapFeature": false
  },
  "before": {
    "info": {
      "version": 1,
      "changeset": 148522680,
      "timestamp": "2024-03-11T20:54:25Z",
      "user": 17722275,
      "visible": true
    },
    "tags": {
      "place": "village"
    },
    "geometry": "POLYGON ((22.0044831 13.4230818, 22.0081705 13.4231201, 22.0082149 13.4269849, 22.0067212 13.4306146, 22.004774 13.4304708, 22.0016535 13.4283946, 22.0013035 13.4246881, 22.0044831 13.4230818))",
    "area": 495855,
    "length": 0,
    "centroid": [
      22.004992407803755,
      13.426554962886161
    ],
    "countries": [
      {
        "iso": "TCD",
        "centroid": true
      }
    ],
    "mapFeature": true
  },
  "mapFeatureEditType": -1,
  "deltaArea": -495855,
  "deltaLength": 0,
  "contributionTypes": [
    "DELETION"
  ],
  "changeset": {
    "replicationSequence": 5920609,
    "id": 148598030,
    "createdAt": "2024-03-13T14:55:20Z",
    "closedAt": "2024-03-13T14:55:21Z",
    "closed": true,
    "userId": 378610,
    "user": "Jorieke V",
    "tags": {
      "comment": "#hotosm-project-16297 Mapping of residential land use in #Djokoro, Chad for #missingmaps #msf",
      "source": "Bing",
      "hashtags": "#hotosm-project-16297;#Djokoro;#missingmaps;#msf",
      "created_by": "JOSM/1.5 (18907 en_GB)"
    },
    "hashtags": [
      "Djokoro",
      "hotosm-project-16297",
      "missingmaps",
      "msf"
    ],
    "editor": "JOSM/1.5 (18907 en_GB)"
  }
}

ElJocho commented 3 months ago

We have now fixed the bug for all data that is contributed from now on - history data will be fixed at the start of the next week if we do not encounter any unexpected issues. @Jorieke

Jorieke commented 3 weeks ago

Thanks for fixing, works excellent now!

GIScience / ohsome-now-stats-service

some deletions are missing country information #60