teambtcmap / btcmap-data

Location data for BTC Map pulled from OpenStreetMap
https://api.btcmap.org/v2/elements
GNU Affero General Public License v3.0
41 stars 14 forks source link

Nonsence nodes in the dataset #4526

Closed iBobik closed 1 year ago

iBobik commented 1 year ago

There are nodes like this. What it means? Should I ignore it?

Maybe deleted places?

{        
  id: 'node:10069082977',
  osm_json: { id: 10069082977, lat: 37.2539903, lon: -8.3479493, type: 'node' },
  tags: {
    category: 'other',
    'category:plural': 'other',
    'icon:android': 'local_florist'
  },
  created_at: '2022-10-02T17:54:09Z',
  updated_at: '2023-03-03T15:13:28.019Z',
  deleted_at: '2023-03-03T15:13:28Z'
}
{
  id: 'node:6817017145',
  osm_json: {
    changeset: 133376967,
    id: 6817017145,
    lat: 45.7609635,
    lon: 7.2995256,
    timestamp: '2023-03-06T19:29:32Z',
    type: 'node',
    uid: 1950277,
    user: 'Mannivu',
    version: 3
  },
  tags: {
    category: 'other',
    'category:plural': 'other',
    'icon:android': 'hotel'
  },
  created_at: '2022-09-25T08:45:08Z',
  updated_at: '2023-03-06T19:42:19.866Z',
  deleted_at: '2023-03-06T19:42:19Z'
}
{ 
  id: 'node:3152251297',
  osm_json: { id: 3152251297, lat: 48.1378711, lon: 11.5594377, type: 'node' },
  tags: {
    category: 'other',
    'category:plural': 'other',
    'icon:android': 'computer'
  },
  created_at: '2022-09-25T08:45:08Z',
  updated_at: '2023-03-07T22:21:57.501Z',
  deleted_at: '2023-03-07T22:21:57Z'
}
{
  id: 'node:2576618095',
  osm_json: { id: 2576618095, lat: 40.4022323, lon: -3.6077747, type: 'node' },
  tags: {
    category: 'other',
    'category:plural': 'other',
    'icon:android': 'imagesearch_roller'
  },
  created_at: '2022-09-25T08:45:08Z',
  updated_at: '2023-03-29T10:40:49.790Z',
  deleted_at: '2023-03-29T10:40:49Z'
}
{
  id: 'node:2186657844',
  osm_json: { id: 2186657844, lat: 52.30844, lon: 21.15293, type: 'node' },
  tags: { category: 'other', 'icon:android': 'question_mark' },
  created_at: '2022-09-25T08:45:08Z',
  updated_at: '2023-05-04T10:05:34.226Z',
  deleted_at: '2022-10-16T11:42:53Z'
}
{
  id: 'node:2778763970',
  osm_json: { id: 2778763970, lat: 35.7784362, lon: 139.6355146, type: 'node' },
  tags: { category: 'other', 'icon:android': 'question_mark' },
  created_at: '2022-09-25T08:45:08Z',
  updated_at: '2023-05-04T10:06:58.275Z',
  deleted_at: '2022-10-15T17:03:46Z'
}
bubelov commented 1 year ago

Yea, those are deleted places, everything with deleted_at != '' should not be displayed. Our client apps rely on caching and they need to get notified of places which stopped accepting bitcoins, that's why we also return the deleted places. Our endpoints are append-only, nothing really gets deleted.

It's not strictly necessary for the initial sync, when the caches aren't yet populated, so we may add a filter flag in the future, but there is a risk that this flag will be copied to the follow-up sync code, messing up the caches, so there are no perfect ways to avoid those footguns.