[date:...] same relation version returned twice

[mmd-osm]

Testing on lz4.overpass-api.de and z.overpass-api.de with test query:

[date:"2022-03-01T00:00:00Z"];
rel(3181794);
out meta;

Result:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.57 93a4d346">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2022-04-13T09:28:57Z"/>

  <relation id="3181794" version="172" timestamp="2022-01-31T21:09:56Z" changeset="116840305" uid="553736" user="A67-A67">
    <tag k="e-road:class" v="A-intermediate"/>
    <tag k="name" v="E 16 Norway west"/>
    <tag k="name:nl" v="E16 Noorwegen west"/>
    <tag k="network" v="e-road"/>
    <tag k="ref" v="E 16"/>
    <tag k="route" v="road"/>
    <tag k="type" v="route"/>
  </relation>
  <relation id="3181794" version="172" timestamp="2022-01-31T21:09:56Z" changeset="116840305" uid="553736" user="A67-A67">
    <member type="way" ref="771148880" role=""/>
    <member type="way" ref="778654104" role=""/>
[... a few hundred members omitted here ...]
    <member type="way" ref="41549024" role="forward"/>
    <member type="way" ref="509419292" role="forward"/>
    <tag k="e-road:class" v="A-intermediate"/>
    <tag k="name" v="E 16 Norway west"/>
    <tag k="name:nl" v="E16 Noorwegen west"/>
    <tag k="network" v="e-road"/>
    <tag k="ref" v="E 16"/>
    <tag k="route" v="road"/>
    <tag k="type" v="route"/>
  </relation>

</osm>

Issue: version 172 is printed twice, once without any members, and another time as complete data with tags, and all members.

The incomplete first entry seems to be copied over from "current" in https://github.com/drolbr/Overpass-API/blob/way_based_areas/src/overpass_api/data/collect_members.h#L446, and added to the list of attic results.

I don't know what the rationale behind this code is, and I also don't know, if this is an issue in filter_attic_elements, or a side effect of some incomplete/buggy data in the undeleted files.

This issue might be related to #661 as well, as it's a basically the same location in the code.

[drolbr]

The source of the problem here is a different one.

The semantics are as follows: at a given point in time t an object exists if

• it is attic and its timestamp is bigger than t or it is current
• and there is a meta for the same index with timestamp smaller or equal t
• and there is no entry in Undeleted with timestamp bigger than t and smaller than the object's timestamp

The semantics are such that there are as few entries in Undeleted as possible. A created and never changed node e.g. has not even an entry in attic.

According to instrumentation, the entry in Undeleted for Index 0xfe for the relation in question is missing although there should have been one for 2022-03-05T10:09:22Z. Thus, there is a bug that corrupted the data, but the cause is unclear so far.

[mmd-osm]

Thanks for looking into this tricky issue. I found some more examples:

All of those examples were found by the following query - assumption is that all members have type "relation", hence the relation itself doesn't have a geometry on its own. On the other hand, not all of the relations returned by the query were affected by the issue.

[maxsize:3G];
rel[!name](if:count_members(relations)==count_members() && version() > 5);
out ids;

and

[maxsize:3G];
rel[name](if:count_members(relations)==count_members() && version() > 5);
out ids;

Query used for analysis:

timeline(relation,11571957);
for (t["created"])
{
  retro (_.val)
  {
    rel(11571957);
    if (count(relations) > 1) {
      out meta;
    }
  }
}

=> Total of 18287 relations analyzed, => 2849 relations have an issue : https://gist.github.com/mmd-osm/8df91b32001edb53523d03a382e83257

Relation 12916419 may be good to reproduce the issue, it has 6 versions and about 5 relation members only.

[drolbr]

Fixed in 48842a1bdd38fb1b804730aa4331527d5af61060 Unrelated to #661. No retrospective repair planned so far, waiting instead for the next database rebuild.

[mmd-osm]

Thanks! It seems this check was added back in https://github.com/drolbr/Overpass-API/commit/2ab85a3c1089c6b630c3b2d807044d220d1e2e5e for both relations and ways, although the commit message only states "some in fact deleted ways bogusly present in attic data.". It's not clear though, why relations are included in the check. I'm not sure if there's some unit test around covering this issue.

Most likely, this issue has been around for quite a while: affected relation 445838 was last updated in 2013. By the time of the last database rebuild back in 2014-2016, the issue was already present.