Server errors - is this one of you?

[mialondon]

From the inbox - not sure if it's related to anything you've been doing @harryjmoss @jmiguelv ?

Please, review the background jobs of your server. This is the trace error

---

Traceback (most recent call last): File "/var/www/pybossa/env/local/lib/python2.7/site-packages/rq/worker.py", line 479, in perform_job rv = job.perform() File "/var/www/pybossa/env/local/lib/python2.7/site-packages/rq/job.py", line 466, in perform self._result = self.func(*self.args, **self.kwargs) File "/var/www/pybossa/pybossa/jobs.py", line 722, in news or (tmp[0]['updated'] != d.entries[0]['updated'])): IndexError: list index out of range

[harryjmoss]

It looks like these errors started around the time the servers were restarted to pick up the kernel update. I restarted the rq scheduler & worker services this morning as well as the pybossa service, so hopefully this will fix things.

[mialondon]

Thanks! I'll keep an eye out for updates in the inbox.

[harryjmoss]

Ok, just had another email reporting this error, so consider this not fixed for now. It's being triggered by an attempt to call news()

def news():
    """Get news from different ATOM RSS feeds."""
    import feedparser
    from pybossa.core import sentinel
    from pybossa.news import get_news, notify_news_admins, FEED_KEY
    try:
        import cPickle as pickle
    except ImportError:  # pragma: no cover
        import pickle
    urls = ['https://github.com/Scifabric/pybossa/releases.atom',
            'http://scifabric.com/blog/all.atom.xml']
    score = 0
    notify = False
    if current_app.config.get('NEWS_URL'):
        urls += current_app.config.get('NEWS_URL')
    for url in urls:
        d = feedparser.parse(url)
        tmp = get_news(score)
        if (d.entries and (len(tmp) == 0)
           or (tmp[0]['updated'] != d.entries[0]['updated'])):
            sentinel.master.zadd(FEED_KEY, float(score),
                                 pickle.dumps(d.entries[0]))
            notify = True
        score += 1
    if notify:
        notify_news_admins()

I'm not sure what's changed to cause this since the kernel update, but I'll start taking a look.

[harryjmoss]

I did a little digging yesterday and found we're seeing essentially the same issue as this. There's supposed to be some protection against no entries in news jobs as of commit 38050a9, which makes me wonder what this is. I checked our settings_local.py in /var/www/pybossa/ and saw there was an empty string being passed to NEWS_URL on l235 (now removed).

The flurry of error messages getting emailed out at 17:13 on 23/05 was due to me restarting the nginx server while pybossa (+other services) were still running, so shouldn't be seen again! I'm hoping restarting the server + all running services in the correct order will sort this, otherwise the cause remains a mystery. I'll carry on having a look into this to try and spot anything else that might be causing it.

For my own reference, the kernel update was performed by following the steps here labelled 'Upgrade kernel only' to avoid any software version conflicts.

[mialondon]

@harryjmoss thanks for the update! The inbox seems free of error reports so it looks like the mystery is solved. If nothing comes in over the weekend then we can close this.

[harryjmoss]

Between restarting everything on Thursday and clearing the failed job queue this morning, I don't see the usual 16:48-17:00 errors popping up so it looks like this has been resolved.