search

oppianmatt / lsc2cal

Reddit LSC feed to icalendar format
0 stars 0 forks source link

Event not turning up in calendar #3

Open AlanApter opened 11 years ago

AlanApter commented 11 years ago

xksa's event isn't in the icalendar - http://www.reddit.com/r/LondonSocialClub/comments/1johf2/131013_jesus_christ_superstar_the_musical_the_o2/ (nice one, xksa).

I looked through your source code and saw that it pulls the data out of http://api.reddit.com/r/londonsocialclub/hot/?sort=new&limit=100, it's not there. It goes back to an event posted 2013-08-19, 44 days ago. xksa's event was posted 2013-08-04 so it missed the limit.

I guess this is low priority because it's rare for an event to be posted two months in advance (nice one xksa). There are other examples - e.g. http://www.redd.it./1lbqmr/. But then again, it could happen, I'm using the icalendar as my primary source for lsc events.

And there is a scaling issue - according to my estimation, in the past 44 days, there were an average of 1.93023256 events per day in the ical (I subtracted the 17 future events in my ical, and divided 83 by 44). This does not count all the events that happened 40ish days ago but weren't included in the ical because they were posted before the event on 2013-08-19. Currently the window is events posted 6 weeks ago. If more events got posted, the window would narrow even further.

However, barc0de's calendar includes it (giving rise to the possibility of scraping off barc0de). I'm not sure if that's because:

a) barc0de scrapes the data using a different method that pulls everything from way back when

b) barc0de stores the data while this is just a scraper (if he does that, I'm unsure if barc0de's calendar deletes an event after it gets deleted, should ask him).

c) barc0de is a dark wizard and used a dark ritual involving black magic the sacrifice of innocents in order to gain control over...

Hi,

I'm not sure what this is, I found this guy slumped over a keyboard in a public library, I started to read what he wrote and I thought it might be important in the investigation of the case. I'm afraid to get involved so I won't be leaving any name or contact details - good luck.

Yours sincerely,

Anonymous

oppianmatt commented 11 years ago

Yeah it's because mine is just a scraper and didn't store the previous events (it's stateless). That's cause it was just a quick and dirty thing. Plan is to put the events into a db (and periodically check them, removed deleted events etc). And more features like perhaps even a reddit bot that posts a link to the calendar event.

I did investigate try out new vs hot etc but wasn't much different so settled on hot (thinking that old events that are popular enough will get upvoted and still be in the feed).

oppianmatt commented 11 years ago

I believe the one that keeps the event around scrape it from my feed initially but then just keeps them in the feed.

barcod3 commented 11 years ago

Its option C:

Actually my script compares matts feed with the existing sidebar calendar html - if xksa has added his event manually then my script will pick it up even if its not in the ical feed. I dont read the events off reddit myself

I did setup data feeds(http://lsccalendar.azurewebsites.net/About.aspx) out of my calender script that could be incorporated into the ical feed but that seems like a snake eating its own tail

AlanApter commented 11 years ago

Relevant xkcd