Remove duplicates

[jibe-b]

When several icals are used as source and some events are duplicate between the two, all events are displayed.

When there is more than one ical as source, there could be a check of identical/similar events to keep only one.

[niccokunzmann]

@jibe-b Thanks for opening the issue. Can you provide examples? have you seen this behavior?

[jibe-b]

Thanks @niccokunzmann for answering fast!

We reference fb events in several fb groups and through a fb account we get ical links.

Some events are referenced in several calendars.

Here is how we display them: https://costrasbourg.frama.io/calendar/#tous-les-%C3%A9v%C3%A9nements-r%C3%A9f%C3%A9renc%C3%A9s-dans-les-agendas

The first calendar displays several icals, the ones following display the icals one by one.

[niccokunzmann]

Thanks for the details! I was not sure how equal they are.

I had a look and it seems they even have the same UID, see the two examples:

{
  "description": "La dynamique est lancée pour certains calendriers-groupes,\n\nRetrouvons-nous pour discuter de la dynamique des autres calendriers ! (qu'est-ce qu'on décide de référencer dans ces calendriers, à qui on les diffuse, à qui est-ce qu'on propose de se les approprier)\n\nLes calendriers dont on parlera :\n\"Réflexions collectives sur des sujets de société\" : https://www.facebook.com/groups/2285014951735781/events\n\n\"Partages de savoirs et savoir-faire\"\nhttps://www.facebook.com/groups/1734231396683491/events\n\nEt ceux que vous souhaiterez dans la liste https://www.facebook.com/permalink.php?story_fbid=2055133834535175&id=2055111037870788\n\nCeci est la suite de \"Lancement de calendriers-groupes\" (toujours entre gens motivés !) https://www.facebook.com/events/2322318631425985/\n\nhttps://www.facebook.com/events/824946671175130/",
  "end_date": "2019-04-09T18:30:00.000Z",
  "end_date_iso": "2019-04-09T18:30:00+00:00",
  "end_date_iso_0": "2019-04-09T18:30:00+00:00",
  "geo": null,
  "ical": "BEGIN:VEVENT\r\nSUMMARY:Boostons les calendriers-groupes - Strasbourg et alentours\r\nDTSTART:20190409T163000Z\r\nDTEND:20190409T183000Z\r\nDTSTAMP:20190404T161434Z\r\nUID:e824946671175130@facebook.com\r\nSEQUENCE:806607\r\nCLASS:PUBLIC\r\nCREATED:20190326T081107Z\r\nDESCRIPTION:La dynamique est lancée pour certains calendriers-groupes\\,\\n\r\n \\nRetrouvons-nous pour discuter de la dynamique des autres calendriers ! (\r\n qu'est-ce qu'on décide de référencer dans ces calendriers\\, à qui on l\r\n es diffuse\\, à qui est-ce qu'on propose de se les approprier)\\n\\nLes cale\r\n ndriers dont on parlera :\\n\"Réflexions collectives sur des sujets de soci\r\n été\" : https://www.facebook.com/groups/2285014951735781/events\\n\\n\"Parta\r\n ges de savoirs et savoir-faire\"\\nhttps://www.facebook.com/groups/173423139\r\n 6683491/events\\n\\nEt ceux que vous souhaiterez dans la liste https://www.f\r\n acebook.com/permalink.php?story_fbid=2055133834535175&id=2055111037870788\\\r\n n\\nCeci est la suite de \"Lancement de calendriers-groupes\" (toujours entre\r\n  gens motivés !) https://www.facebook.com/events/2322318631425985/\\n\\nhtt\r\n ps://www.facebook.com/events/824946671175130/\r\nLAST-MODIFIED:20190404T161434Z\r\nLOCATION:L'Artichaut Grand Rue\r\nORGANIZER;CN=\"Calendriers à mettre entre toutes les mains - Strasbourg et\r\n  alentours\":MAILTO:noreply@facebookmail.com\r\nPARTSTAT:TENTATIVE\r\nSTATUS:CONFIRMED\r\nURL:https://www.facebook.com/events/824946671175130/\r\nEND:VEVENT\r\n",
  "location": "L'Artichaut Grand Rue",
  "recurrence": null,
  "sequence": "806607",
  "start_date": "2019-04-09T16:30:00.000Z",
  "start_date_iso": "2019-04-09T16:30:00+00:00",
  "start_date_iso_0": "2019-04-09T16:30:00+00:00",
  "text": "Boostons les calendriers-groupes - Strasbourg et alentours",
  "uid": "e824946671175130@facebook.com",
  "url": "https://www.facebook.com/events/824946671175130/",
  "id": 1556106366089,
  "_timed": true,
  "_sday": 1,
  "_eday": 2,
  "_length": 1,
  "_sweek": 1,
  "_sorder": 0,
  "_first_chunk": true,
  "_last_chunk": true
}

{
  "description": "La dynamique est lancée pour certains calendriers-groupes,\n\nRetrouvons-nous pour discuter de la dynamique des autres calendriers ! (qu'est-ce qu'on décide de référencer dans ces calendriers, à qui on les diffuse, à qui est-ce qu'on propose de se les approprier)\n\nLes calendriers dont on parlera :\n\"Réflexions collectives sur des sujets de société\" : https://www.facebook.com/groups/2285014951735781/events\n\n\"Partages de savoirs et savoir-faire\"\nhttps://www.facebook.com/groups/1734231396683491/events\n\nEt ceux que vous souhaiterez dans la liste https://www.facebook.com/permalink.php?story_fbid=2055133834535175&id=2055111037870788\n\nCeci est la suite de \"Lancement de calendriers-groupes\" (toujours entre gens motivés !) https://www.facebook.com/events/2322318631425985/\n\nhttps://www.facebook.com/events/824946671175130/",
  "end_date": "2019-04-09T18:30:00.000Z",
  "end_date_iso": "2019-04-09T18:30:00+00:00",
  "end_date_iso_0": "2019-04-09T18:30:00+00:00",
  "geo": null,
  "ical": "BEGIN:VEVENT\r\nSUMMARY:Boostons les calendriers-groupes - Strasbourg et alentours\r\nDTSTART:20190409T163000Z\r\nDTEND:20190409T183000Z\r\nDTSTAMP:20190404T161434Z\r\nUID:e824946671175130@facebook.com\r\nSEQUENCE:806607\r\nCLASS:PUBLIC\r\nCREATED:20190326T081107Z\r\nDESCRIPTION:La dynamique est lancée pour certains calendriers-groupes\\,\\n\r\n \\nRetrouvons-nous pour discuter de la dynamique des autres calendriers ! (\r\n qu'est-ce qu'on décide de référencer dans ces calendriers\\, à qui on l\r\n es diffuse\\, à qui est-ce qu'on propose de se les approprier)\\n\\nLes cale\r\n ndriers dont on parlera :\\n\"Réflexions collectives sur des sujets de soci\r\n été\" : https://www.facebook.com/groups/2285014951735781/events\\n\\n\"Parta\r\n ges de savoirs et savoir-faire\"\\nhttps://www.facebook.com/groups/173423139\r\n 6683491/events\\n\\nEt ceux que vous souhaiterez dans la liste https://www.f\r\n acebook.com/permalink.php?story_fbid=2055133834535175&id=2055111037870788\\\r\n n\\nCeci est la suite de \"Lancement de calendriers-groupes\" (toujours entre\r\n  gens motivés !) https://www.facebook.com/events/2322318631425985/\\n\\nhtt\r\n ps://www.facebook.com/events/824946671175130/\r\nLAST-MODIFIED:20190404T161434Z\r\nLOCATION:L'Artichaut Grand Rue\r\nORGANIZER;CN=\"Calendriers à mettre entre toutes les mains - Strasbourg et\r\n  alentours\":MAILTO:noreply@facebookmail.com\r\nPARTSTAT:TENTATIVE\r\nSTATUS:CONFIRMED\r\nURL:https://www.facebook.com/events/824946671175130/\r\nEND:VEVENT\r\n",
  "location": "L'Artichaut Grand Rue",
  "recurrence": null,
  "sequence": "806607",
  "start_date": "2019-04-09T16:30:00.000Z",
  "start_date_iso": "2019-04-09T16:30:00+00:00",
  "start_date_iso_0": "2019-04-09T16:30:00+00:00",
  "text": "Boostons les calendriers-groupes - Strasbourg et alentours",
  "uid": "e824946671175130@facebook.com",
  "url": "https://www.facebook.com/events/824946671175130/",
  "id": 1556106366137,
  "_timed": true,
  "_sday": 1,
  "_eday": 2,
  "_length": 1,
  "_sweek": 1,
  "_sorder": 2,
  "_first_chunk": true,
  "_last_chunk": true
}

I think, we can say this: If two events have the same UID, only one of them should be displayed.

[jibe-b]

Yes, in our case they are exacty the same so matching UID is fine.

And if two elements are very similar but have different UIDs, displaying them may enable humans to identify duplicates and lead them to correct the duplication by hand.

[niccokunzmann]

@jibe-b Please have a look at the new deployment. I pondered a while on this problem and finally decided how to solve it best. Usually, I do not want to solve other people's issues. In this case, I liked it very much.

Safe assumption from my point of view: If UID and DTSTART are the same, then the events are the same.

[jibe-b]

Dear @niccokunzmann, thanks a lot for implementing this filter!

You've started to code this project 2 weeks before we started to crowdsource the referencing of our calendars and started using it 3 weeks ago so we enjoy alot you adding this feature so quickly :)

[niccokunzmann]

@jibe-b I hear you are happy about the improvement. I guess, this issue can be closed now. If you think otherwise, you can comment below. See you ;)