search

pipes-digital / pipes

Repository for Pipes
https://pipes.digital
GNU Affero General Public License v3.0
254 stars 21 forks source link

item GUID is not unique #68

Closed sato942 closed 4 years ago

sato942 commented 4 years ago

hello, i created a feed here https://www.pipes.digital/pipe/PN681Yq6

i noticed that my result return different items with same GUID

for example if you see this output

<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
<channel>
<title>
Warlocks
  </title>
<link>
https://www.pipes.digital/feed/temp
  </link>
<description>

  </description>
<pubDate>
Mon, 22 Jun 2020 17:05:54 +0200
  </pubDate>
<item>
<title>
Jojomojomon
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Jojomojomon
  </link>
<pubDate>
Sun, 21 Jun 2020 22:28:00 +0200
  </pubDate>
<guid isPermaLink="false">
49d763630c43bb3607f4c79e22b60a87
  </guid>
<content:encoded>
472.94
  </content:encoded>
<dc:date>
2020-06-21T22:28:00+02:00
  </dc:date>

  </item>
<item>
<title>
Darkecho
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Darkecho
  </link>
<pubDate>
Sun, 21 Jun 2020 12:01:00 +0200
  </pubDate>
<guid isPermaLink="false">
1f593bb4949b05cf3dc4fc0ae2bf090c
  </guid>
<content:encoded>
473.63
  </content:encoded>
<dc:date>
2020-06-21T12:01:00+02:00
  </dc:date>

  </item>
<item>
<title>
Quadpower
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Quadpower
  </link>
<pubDate>
Fri, 19 Jun 2020 13:58:00 +0200
  </pubDate>
<guid isPermaLink="false">
475288208f4bf5aced1be88db09b3269
  </guid>
<content:encoded>
475.63
  </content:encoded>
<dc:date>
2020-06-19T13:58:00+02:00
  </dc:date>

  </item>
<item>
<title>
Omnistrax
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Omnistrax
  </link>
<pubDate>
Fri, 19 Jun 2020 00:04:00 +0200
  </pubDate>
<guid isPermaLink="false">
406cf785a981a1334ad5cf32be9c7603
  </guid>
<content:encoded>
467.94
  </content:encoded>
<dc:date>
2020-06-19T00:04:00+02:00
  </dc:date>

  </item>
<item>
<title>
Stevé
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Stev%C3%A9
  </link>
<pubDate>
Thu, 18 Jun 2020 15:35:00 +0200
  </pubDate>
<guid isPermaLink="false">
b43d7fbbac9be8b791cbe6e1d665b601
  </guid>
<content:encoded>
472.25
  </content:encoded>
<dc:date>
2020-06-18T15:35:00+02:00
  </dc:date>

  </item>
<item>
<title>
Leilanáyiff
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Leilan%C3%A1yiff
  </link>
<pubDate>
Thu, 18 Jun 2020 15:25:00 +0200
  </pubDate>
<guid isPermaLink="false">
a8f5d7f8db462b97b753af86687f3efd
  </guid>
<content:encoded>
476.69
  </content:encoded>
<dc:date>
2020-06-18T15:25:00+02:00
  </dc:date>

  </item>
<item>
<title>
Shalaeandra
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Shalaeandra
  </link>
<pubDate>
Thu, 18 Jun 2020 02:04:00 +0200
  </pubDate>
<guid isPermaLink="false">
a7bbde7d94bedef9621de495e611df7d
  </guid>
<content:encoded>
478.63
  </content:encoded>
<dc:date>
2020-06-18T02:04:00+02:00
  </dc:date>

  </item>
<item>
<title>
Swybe
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Swybe
  </link>
<pubDate>
Thu, 11 Jun 2020 21:02:00 +0200
  </pubDate>
<guid isPermaLink="false">
462e0247c1d39f3de61dc05c497b1905
  </guid>
<content:encoded>
459.88
  </content:encoded>
<dc:date>
2020-06-11T21:02:00+02:00
  </dc:date>

  </item>
<item>
<title>
Dualippa
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Dualippa
  </link>
<pubDate>
Thu, 11 Jun 2020 15:52:00 +0200
  </pubDate>
<guid isPermaLink="false">
10790e255fed7e236783a4be0a314d63
  </guid>
<content:encoded>
470.06
  </content:encoded>
<dc:date>
2020-06-11T15:52:00+02:00
  </dc:date>

  </item>
<item>
<title>
Haji
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Haji
  </link>
<pubDate>
Tue, 09 Jun 2020 19:58:00 +0200
  </pubDate>
<guid isPermaLink="false">
b35fc4199b58b51760d004495bc561c8
  </guid>
<content:encoded>
476.94
  </content:encoded>
<dc:date>
2020-06-09T19:58:00+02:00
  </dc:date>

  </item>
<item>
<title>
Hairyback
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Hairyback
  </link>
<pubDate>
Sat, 06 Jun 2020 12:28:00 +0200
  </pubDate>
<guid isPermaLink="false">
ccee61e7b598746c8f4a7fd559942fd0
  </guid>
<content:encoded>
476.13
  </content:encoded>
<dc:date>
2020-06-06T12:28:00+02:00
  </dc:date>

  </item>
<item>
<title>
Pedbag
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Pedbag
  </link>
<pubDate>
Thu, 04 Jun 2020 15:55:00 +0200
  </pubDate>
<guid isPermaLink="false">
3c5908f481157f9ab7d354d7c5d1b02d
  </guid>
<content:encoded>
473.31
  </content:encoded>
<dc:date>
2020-06-04T15:55:00+02:00
  </dc:date>

  </item>
<item>
<title>
Szolmer
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Szolmer
  </link>
<pubDate>
Fri, 29 May 2020 15:37:00 +0200
  </pubDate>
<guid isPermaLink="false">
fc183337c6c5ef3018327d78d02cbb60
  </guid>
<content:encoded>
473.13
  </content:encoded>
<dc:date>
2020-05-29T15:37:00+02:00
  </dc:date>

  </item>
<item>
<title>
Grugnir
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Grugnir
  </link>
<pubDate>
Thu, 28 May 2020 20:10:00 +0200
  </pubDate>
<guid isPermaLink="false">
49f5176f76bd0826b24f391d87ed41cd
  </guid>
<content:encoded>
464.38
  </content:encoded>
<dc:date>
2020-05-28T20:10:00+02:00
  </dc:date>

  </item>
<item>
<title>
Jying
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Jying
  </link>
<pubDate>
Mon, 25 May 2020 23:22:00 +0200
  </pubDate>
<guid isPermaLink="false">
d78b7d30fa301590aea00f94c59c9775
  </guid>
<content:encoded>
478.88
  </content:encoded>
<dc:date>
2020-05-25T23:22:00+02:00
  </dc:date>

  </item>
<item>
<title>
Sôt
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/S%C3%B4t
  </link>
<pubDate>
Mon, 25 May 2020 22:17:00 +0200
  </pubDate>
<guid isPermaLink="false">
e1da4d702ac43f825b2717ecfff7e4b0
  </guid>
<content:encoded>
475.94
  </content:encoded>
<dc:date>
2020-05-25T22:17:00+02:00
  </dc:date>

  </item>
<item>
<title>
Infernalize
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Infernalize
  </link>
<pubDate>
Sun, 17 May 2020 11:25:00 +0200
  </pubDate>
<guid isPermaLink="false">
ccd3eae6d0e9e3f1762995f03dafa9ac
  </guid>
<content:encoded>
459.75
  </content:encoded>
<dc:date>
2020-05-17T11:25:00+02:00
  </dc:date>

  </item>
<item>
<title>
Alataló
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Alatal%C3%B3
  </link>
<pubDate>
Sun, 10 May 2020 06:27:00 +0200
  </pubDate>
<guid isPermaLink="false">
475288208f4bf5aced1be88db09b3269
  </guid>
<content:encoded>
475.63
  </content:encoded>
<dc:date>
2020-05-10T06:27:00+02:00
  </dc:date>

  </item>
<item>
<title>
Goonw
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Goonw
  </link>
<pubDate>
Thu, 30 Apr 2020 17:42:00 +0200
  </pubDate>
<guid isPermaLink="false">
892a0e64b4db83902da196f6e589ae7e
  </guid>
<content:encoded>
477.13
  </content:encoded>
<dc:date>
2020-04-30T17:42:00+02:00
  </dc:date>

  </item>
<item>
<title>
Eruunlock
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Eruunlock
  </link>
<pubDate>
Sun, 19 Apr 2020 20:23:00 +0200
  </pubDate>
<guid isPermaLink="false">
b79625fb62bdd8d8e6ac94b91f0aae50
  </guid>
<content:encoded>
477.63
  </content:encoded>
<dc:date>
2020-04-19T20:23:00+02:00
  </dc:date>

  </item>
<item>
<title>
Drloosen
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Drloosen
  </link>
<pubDate>
Sun, 19 Apr 2020 20:03:00 +0200
  </pubDate>
<guid isPermaLink="false">
4571e8bbc5958a7b12a6903be6f1ced0
  </guid>
<content:encoded>
472.75
  </content:encoded>
<dc:date>
2020-04-19T20:03:00+02:00
  </dc:date>

  </item>
<item>
<title>
Dítch
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/D%C3%ADtch
  </link>
<pubDate>
Mon, 13 Apr 2020 19:43:00 +0200
  </pubDate>
<guid isPermaLink="false">
0c8fb82205e546e460fbc996a9663217
  </guid>
<content:encoded>
469.56
  </content:encoded>
<dc:date>
2020-04-13T19:43:00+02:00
  </dc:date>

  </item>
<item>
<title>
Hashsolo
  </title>
<link>
https://www.wowprogress.com/character/eu/draenor/Hashsolo
  </link>
<pubDate>
Sun, 12 Apr 2020 00:33:00 +0200
  </pubDate>
<guid isPermaLink="false">
475288208f4bf5aced1be88db09b3269
  </guid>
<content:encoded>
475.63
  </content:encoded>
<dc:date>
2020-04-12T00:33:00+02:00
  </dc:date>

  </item>
<dc:date>
2020-06-22T17:05:54.087729+02:00
  </dc:date>

  </channel>

  </rss>

you will find three items with the same GUID 475288208f4bf5aced1be88db09b3269

is it possible to make the items more unique?

onli commented 4 years ago

The root source of this issue is probably at https://github.com/pipes-digital/pipes/blob/bddfa3fae7e6e650425da74a3a199955439cfd10/blocks/builderblock.rb#L46

The builder block takes the hash of the content as GUID. In this case, the content is the same, thus the GUID is the same. To make them unique we'd need something that differs at each item. But also stays stable each time the feed is created, otherwise feed readers would get confused.

In this case the link would help and fit, right? If a link is set also use that to create the guid hash. I think that can be added to the block.

onli commented 4 years ago

It's even a bit more complicated. The builder block used the guid from before, the extract block in this case, and that one has no link or title it could use.

I added a new checkbox to the builder block, recalculate GUID. When activated the block will use a hash of the title, link and content for the guid. Would be great if you could report back whether that solves this issue for you?

sato942 commented 4 years ago

one thing i did before your change is adding Truncate block to limit the amount of reported content, in my case i am not expecting more than one or two new entries everyday.

i am removing the limit and enabling the new change, will report back if i see any issues.

sato942 commented 4 years ago

i have had no issues so far.

onli commented 4 years ago

Great, thank you for reporting back. I will close here for now then.