Repeated items in Feedly, other readers?

[scripting]

I've had a report by one user of repeated items from Scripting News in Feedly. Are other people seeing this? In other feed readers? I've seen no problems in River5.

[michaelkamleitner]

That user would be me ;) Here's a screenshot of my Feedly:

It doesn't seem to affect ALL posts. The one from yesterday, starting with "I like to say I like the 280 char ..." seems to be affected a lot (6+ duplicates). Others ("the perfect length of a podcast...") not at all. I'd assume, it depends on how often Dave updates an item/post?

I think this behaviour is going on for a few weeks, but hard to say, since as I wasn't reading a lot in the past month. Thanks Dave for bringing this up here, anyway!

[scripting]

What's the URL of the feed you're subscribed to?

Also, re it depending on how often I update a post, there's no reason to think there's a correlation. Unless the guid on the item changes, and I'm pretty sure it doesn't, or the item would repeat in River5 as well, probably.

Until we figure out what the problem is we won't know what caused it. ;-)

[michaelkamleitner]

http://www.scripting.com/rss.xml - same since 2000-something ;)

[benzado]

I've seen a few in Newsblur: http://www.newsblur.com/site/6720249/scripting-news

Same feed URL: http://scripting.com/rss.xml

Not the same posts as @michaelkamleitner, if you scroll down to November 2-5, some posts are doubled.

[laymusic]

This is what I get in gwene.org as read in gnus in gnuemacs:

R [ 12: unknown ] The open web needs a friend R < 12: unknown > O [ 8: unknown ] Why Google really wants you to use HTTPS O < 8: unknown > O [ 6: unknown ] Phoenix bike share O < 6: unknown >

A bunch of untitled posts have also been duplicated, but I can't demonstrate that from the Summary buffer.

[scripting]

Interesting about Nov 2-5, that's when I was at the Newsgeist conference, and I was posting from my laptop, not my desktop. I'm using a new CMS, and it's possible I didn't copy over everything I needed, and it saw some posts as new that weren't.

[JPhilpin]

I have seen same issue - will watch carefully for which and why - and report with specifics - I just assumed that maybe you were doing some updates on a post that caused the repost

[JPhilpin]

Screen grab of feedly right now ...

[JPhilpin]

this one form mac feedly 1 minute ago

[JPhilpin]

this one from iphone 2 days ago (apologies for the delay)

[scripting]

Not that this has anything to do with the problem, why are they putting "It's even worse than it appears" in front of each post that doesn't have a title? Is that what they're doing? Is this some kind of joke?

On Sun, Nov 12, 2017 at 5:14 PM, JPhilpin notifications@github.com wrote:

this one from iphone 2 days ago (apologies for the delay)

[image: img_6081] https://user-images.githubusercontent.com/18621554/32704032-fe55d8a2-c7a2-11e7-8fb4-1226165e22b1.png

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/42#issuecomment-343772517, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm9O-RWfnKdUoUwut6kF_Fxt2c4Q9kFks5s122-gaJpZM4QaG9k .

[scripting]

Is that the iPhone version of Feedly or some other app?

[scripting]

I've set up a script that takes a snapshot of the feed once a minute. I'm going to leave it running, so when I get the next report, I will have snapshots to go back and review to see if the guids change. That would be what would trigger an update if the bug is on my end.

[scripting]

BTW, this is what I see in Feedly when I subscribe to my feed.

[scripting]

I have a theory about what happened, unfortunately it's hard to reproduce.

My CMS runs on the local machine, not on a server. So when the local machine moves into a different time zone, the GUIDs and permalinks are changed, because they are computed in the current time zone. That's a bug waiting to happen.

So it would seem that the permalinks to all items in the feed would change when I move to a different time zone, and change back again when I return home.

There must be a good solution to this, and I have to figure it out before I take my next trip.

Update

I think I found and implemented the fix. It was easy, I hope. ;-)

[jsavin]

Hi Dave,

This reminds me a little of some of the issues we ran into when moving databases back and forth between Windows and Mac servers. Native strings strings mapped some characters differently on the two platforms and we didn’t have Unicode, so we (Brent and I?) decided to store data in one or the other character-set (ISO-8859-1 I think) and convert on input and output. That allowed Manila site databases to be portable between Mac and Windows.

I think we eventually decided that we should handle date/time storage similarly, using UTC for storage and converting to server or user-time on I/O.

If the time zone change turns out to be the culprit then off the top of my head, I can think of a few solutions:

1) Save the permalink as an attribute in the outline when it’s initially generated, and if the attribute exists when the feed is published then use it. This makes the value sticky so a time change won’t affect it.

2) Convert local time to GMT/UTC for storage when the node is created, so the generated permalink won’t change when the local time zone changes. This probably implies converting some other UI also, but it has the advantage that the saved date/time is “canonical”.

3) Store the time zone along with the date/time and convert that time zone to GMT/UTC when the feed is published. Basically the same as #2, but means that either the storage format will change or an additional attribute needs to be added.

2 seems the most robust fix IMHO since the stored data will always be in the same time zone, and you can convert it as needed for UI or publishing. The downside is that depending on how it’s used, you might also need to go back through your history and converting the stored date/times to GMT (assuming that the “real” time matters, as opposed to simple uniqueness).

HTH, -Jake

On Nov 12, 2017, at 8:13 PM, Dave Winer notifications@github.com wrote:

I have a theory about what happened, unfortunately it's hard to reproduce.

My CMS runs on the local machine, not on a server. So when the local machine moves into a different time zone, the GUIDs and permalinks are changed, because they are computed in the current time zone. That's a bug waiting to happen.

So it would seem that the permalinks to all items in the feed would change when I move to a different time zone, and change back again when I return home.

There must be a good solution to this, and I have to figure it out before I take my next trip.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/42#issuecomment-343806486, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvqgoV2svWlIRkw583zJIu7EZiYuL1Gks5s18H2gaJpZM4QaG9k.

[scripting]

Thanks Jake for the thorough analysis. It really helps. I went with #2. 😎

BTW, I didn't have to go back through history, because I grandfathered-in the old broken GUID formation code. And this isn't the first bug I've fixed in this area. Rule #1 still applies. ;-)

Here's the code for permalink-generation, so you can see..

function getPermalinkString (when) { //7/9/17 by DW
    var pattern = "HHMMss", flUseDateFormat = false;
    if (new Date (when) < new Date ("Sun Jul 09 2017 17:53:55 GMT")) {
        pattern = "hhmmss";
        flUseDateFormat = true;
        }
    else {
        if (new Date (when) < new Date ("Mon, 13 Nov 2017 03:56:28 GMT")) {
            pattern = "hhMMss";
            flUseDateFormat = true;
            }
        }
    if (flUseDateFormat) {
        return ("a" + dateFormat (when, pattern));
        }
    else {
        function pad (num) {
            return (utils.padWithZeros (num, 2));
            }
        var d = new Date (when);
        return ("a" + pad (d.getUTCHours ()) + pad (d.getUTCMinutes ()) + pad (d.getUTCSeconds ()));
        }
    }

PS: To everyone else, Jake and I worked together on Manila and Radio UserLand, many many years ago. These problems aren't new, as you can see. ;-)

[scripting]

Here's the writeup of the fix, implemented late last night..

http://scripting.com/2017/11/13.html#a145425

Dave

[michaelkamleitner]

Thanks for working on this problem, fingers crossed that this is the solution (I'll keep my eyes open) :)

[JPhilpin]

Many thanks Dave - I am looking at Feedly on Mac, iPhone and iPad ... all is as should be. All older dupes also gone. And for completeness, the white out of black image above was Feedly on iPhone. Thankyou.

[jsavin]

You're welcome. Glad I could help. 🙇

-Jake

On Nov 13, 2017, at 3:11 AM, Dave Winer notifications@github.com wrote:

Thanks Jake for the thorough analysis. It really helps. I went with #2. 😎

On Monday, November 13, 2017, Jake Savin notifications@github.com wrote:

Hi Dave,

This reminds me a little of some of the issues we ran into when moving databases back and forth between Windows and Mac servers. Native strings strings mapped some characters differently on the two platforms and we didn’t have Unicode, so we (Brent and I?) decided to store data in one or the other character-set (ISO-8859-1 I think) and convert on input and output. That allowed Manila site databases to be portable between Mac and Windows.

I think we eventually decided that we should handle date/time storage similarly, using UTC for storage and converting to server or user-time on I/O.

If the time zone change turns out to be the culprit then off the top of my head, I can think of a few solutions:

1) Save the permalink as an attribute in the outline when it’s initially generated, and if the attribute exists when the feed is published then use it. This makes the value sticky so a time change won’t affect it.

2) Convert local time to GMT/UTC for storage when the node is created, so the generated permalink won’t change when the local time zone changes. This probably implies converting some other UI also, but it has the advantage that the saved date/time is “canonical”.

3) Store the time zone along with the date/time and convert that time zone to GMT/UTC when the feed is published. Basically the same as #2, but means that either the storage format will change or an additional attribute needs to be added.

2 seems the most robust fix IMHO since the stored data will always be in

the same time zone, and you can convert it as needed for UI or publishing. The downside is that depending on how it’s used, you might also need to go back through your history and converting the stored date/times to GMT (assuming that the “real” time matters, as opposed to simple uniqueness).

HTH, -Jake

On Nov 12, 2017, at 8:13 PM, Dave Winer <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

I have a theory about what happened, unfortunately it's hard to reproduce.

My CMS runs on the local machine, not on a server. So when the local machine moves into a different time zone, the GUIDs and permalinks are changed, because they are computed in the current time zone. That's a bug waiting to happen.

So it would seem that the permalinks to all items in the feed would change when I move to a different time zone, and change back again when I return home.

There must be a good solution to this, and I have to figure it out before I take my next trip.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < https://github.com/scripting/Scripting-News/issues/42# issuecomment-343806486>, or mute the thread https://github.com/ notifications/unsubscribe-auth/ABvqgoV2svWlIRkw583zJIu7EZiYuL 1Gks5s18H2gaJpZM4QaG9k.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/42#issuecomment-343833004, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm9O75TpFLOihHnSv6HFB84pNvHVEPbks5s1-xggaJpZM4QaG9k .

-- Typed on an iPad with fat fingers. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[jsavin]

On more thing - a bit of wisdom I suppose, learned in the early era of bogging CMSen to share with others who may not have similar experience...

Nowadays whenever I run across a feature that deals with dates and times or, learned a little later, phone numbers (from work on contact record synchronization in Messenger at Microsoft), or learned a few years earlier, floating-point numbers... When I encounter any of these my Spidey Sense tingles and I pay extra attention. I suppose all that work we did on SOAP interop plays in too.

The underlying principle is that the data had to survive a "round trip", whatever that means in context. As long as information can start in one place, go somewhere else, and return to it's point of origin intact, you have a pretty good chance of coming up with a working design. The details are important but vary by scenario. The principle does not.

Funny that in this case there was a literal "trip" involved! 😉

Cheers, -Jake

PS. Thanks for introducing me to Postel's Law, Dave.

PPS. General advice for software developers: If you can help it, never do your own time-zone conversions. Chances are that the owners of the date/time libraries for your platform know more about this problem than you do. 😉

[michaelkamleitner]

So sorry to report, that this issue still seems to be on... :(

[JPhilpin]

yes - that is what I am seeing aswell

/JOHN

On Nov 16, 2017, at 2:12 PM, Michael Kamleitner notifications@github.com wrote:

So sorry to report, that this issue still seems to be on... :(

https://user-images.githubusercontent.com/341737/32922779-4591d998-cb34-11e7-9989-d24581efa9bb.png — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/42#issuecomment-345104743, or mute the thread https://github.com/notifications/unsubscribe-auth/ARwkcnb8vrT0TYjQsbF5sLrFyGp81iWhks5s3M9cgaJpZM4QaG9k.

[scripting]

This is what I'm seeing in my Feedly account. What's the difference in our setup? I have no experience with Feedly.

[JPhilpin]

interesting - my dupes are now gone … that said the difference between the views is that you are viewing in magazine view - and I am viewing in title only

you can toggle between those views (and two others) by clicking on the pie symbol - highlighted graphic

bottom line - currently no dupes showing …

/JOHN

On Nov 16, 2017, at 4:05 PM, Dave Winer notifications@github.com wrote:

This is what I'm seeing in my Feedly account. What's the difference in our setup? I have no experience with Feedly.

https://user-images.githubusercontent.com/1686843/32925809-e7770ada-cb11-11e7-9954-96bf47b6770e.png — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/42#issuecomment-345123522, or mute the thread https://github.com/notifications/unsubscribe-auth/ARwkcqg4qeEcXtbO-wHf0M_TZbfFd8Zlks5s3On5gaJpZM4QaG9k.

[JPhilpin]

I am seeing dupes this morning in Feedly on the mac - see screenshot. BUT - the dupes are not on Feedly on the iPAD ... I wonder if it is something that they are doing ?

[michaelkamleitner]

Sorry for late reply, didnt read regularly recently. Problem seems to persist :(

[JPhilpin]

same for me - attached is from this morning

[scripting]

Maybe someone from Feedly can give us an idea what this problem might indicate about the feed.

Otherwise I don't have any advice. The feed is valid as far as I know.

[scripting]

One thing I'd love to know -- where are they getting "It's even worse than it appears" -- It's not in the feed.

http://xmlviewer.scripting.com/?url=http://scripting.com/rss.xml

Dave

[scripting]

As before I'm not seeing any problems, or any "it's even worse than it appears" messages.

[michaelkamleitner]

Super strange! Well, I have enough of Feedly's shenanigans, I just removed your feed and re-subscribed http://scripting.com/rss.xml - didnt help :( I'm going to ping Feedly on Twitter, maybe someones listening...

[scotthansonde]

Feedly is my main feed reader, and I'm not seeing any problems.

[michaelkamleitner]

That just makes it even stranger :( Well, I'm paying Feedly hundreds $/year for our team subscription, so maybe they will listen :)

RT if you feel like: https://twitter.com/_subnet/status/941713060368191488

[JPhilpin]

I find edwin very responsive at Feedly :

edwin@feedly.com mailto:edwin@feedly.com

one thought - I am a pro member if michael also is and scott isn’t … would that be a clue ?

On Dec 15, 2017, at 8:57 AM, Scott Hanson notifications@github.com wrote:

Feedly is my main feed reader, and I'm not seeing any problems.

https://user-images.githubusercontent.com/8681117/34052124-5241911c-e1c1-11e7-8e81-9794f426bda0.png — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/scripting/Scripting-News/issues/42#issuecomment-352056535, or mute the thread https://github.com/notifications/unsubscribe-auth/ARwkchS4_kgQKMMaR7OIIbTDHpQtJxAfks5tAqT6gaJpZM4QaG9k.

[scotthansonde]

I'm a pro member as well. Nice try, though. :-)