search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

RSS feed for ZMQ alerts #1712

Closed zdenekmaxa closed 12 years ago

zdenekmaxa commented 13 years ago

Alternative sinks for alert messages

zdenekmaxa commented 13 years ago

maxa: Regarding RSSFeedSink:

8e09c9da98331a5fcdbf0b1e9b52e7f2ba11a7cc http://wiki.python.org/moin/RssLibraries a882b088aca6f50faac4bb3055943e85fbf7a218 http://www.feedparser.org 4df8c9216ee1b3870126b78906b5b4113eea6823 http://www.dalkescientific.com/Python/PyRSS2Gen.html 25c96c71ed4036bca7ee049c01534fb8ba04701b http://lxml.de/

vkuznet commented 13 years ago

valya: Replying to [comment:2 maxa]:

Regarding RSSFeedSink:

  • a few RSS manipulating libs evaluated 8e09c9da98331a5fcdbf0b1e9b52e7f2ba11a7cc:
    • a882b088aca6f50faac4bb3055943e85fbf7a218 is very advanced one, understands myriad feed format derivations, versions, etc, yet doesn't provide any mechanism to build and output XML for RSS feed. Since we only need read back our own feed files it's not necessary to be very versatile and a882b088aca6f50faac4bb3055943e85fbf7a218 would provide only one-way overkill.
    • 4df8c9216ee1b3870126b78906b5b4113eea6823 Conversion from Python types to RSS XML, again one way thing, moreover not updated for a while.
  • Didn't like adding two different dependencies for this task
  • 25c96c71ed4036bca7ee049c01534fb8ba04701b is a very good, general purpose XML library, faster than standard Python XML libs and its addition has already been suggested while ago during TFC work. Provides easy parsing from XML into Python data structure and back, RSSFeedSink depends on it. I guess other areas may benefit from it too.

8e09c9da98331a5fcdbf0b1e9b52e7f2ba11a7cc http://wiki.python.org/moin/RssLibraries a882b088aca6f50faac4bb3055943e85fbf7a218 http://www.feedparser.org 4df8c9216ee1b3870126b78906b5b4113eea6823 http://www.dalkescientific.com/Python/PyRSS2Gen.html 25c96c71ed4036bca7ee049c01534fb8ba04701b http://lxml.de/

I experiment in DAS with 25c96c71ed4036bca7ee049c01534fb8ba04701b, so if people will vote for it I'll be happy to use it. Valentin

zdenekmaxa commented 13 years ago

maxa: Regarding !TwitterSink:

(1) http://dev.twitter.com/pages/libraries (2) http://pypi.python.org/pypi/python-twitter/ (3) http://pypi.python.org/pypi/oauth2/ For reference: http://github.com/ryanmcgrath/twython http://code.google.com/p/tweepy

zdenekmaxa commented 13 years ago

maxa: patch-series-maxa-alertfw-special-sinks.tgz has both !RSSFeedSink and !TwitterSink implemented. Dave, please review.

drsm79 commented 13 years ago

metson: I'm dubious of twitter sink, not least because it's a lot of dependencies for limited utility.

zdenekmaxa commented 13 years ago

maxa: Replying to [comment:7 metson]:

I'm dubious of twitter sink, not least because it's a lot of dependencies for limited utility.

First, I thought we depend on simplejson and httplib2 already, no? Though even two extra deps (python-twitter and oauth2) may be a bit too much for what it buys, I agree. I noticed that some older version of twitter lib was present in the codebase already (I suggested to remove it from Services) and Dave said there were some issues like blocking etc long ago with twitter. So I wanted to explore the idea and managed to arrive at a functional solution. If the opinion is that !TwitterSink is an overkill in the Alert sinks offer, I am happy to put it on hold and remove it from the patch.

evansde77 commented 13 years ago

evansde: I would say lets leave the twitter sink out for now. We probably should look at what is being sent in the Alert messages before we enable essentially public broadcasting of them as well.

drsm79 commented 13 years ago

metson: Simplejson is (or should be) deprecated (since we've moved to python 2.6). Oauth is enough of a hog to be a blocker IMHO. We should remove the twitter code from wmcore, too (could you open a ticket for that?)

zdenekmaxa commented 13 years ago

maxa: Which sinks will be used will always be a matter of configuration, so !TwitterSink may not be enabled until we are confident with the system overall.

Anyway, I will send a new patch without twitter stuff (mine) and it will also remove the old twitter lib.

If we need this broadcasting in the future, the functionality is basically ready and I will check if std python json can work instead of simplejson and what issues oauth brings.

zdenekmaxa commented 13 years ago

maxa: patch-series-maxa-alertfw-special-sinks-rss.tgz

Dave, please review.

zdenekmaxa commented 13 years ago

maxa: For record, TwitterSink.tgz, python-twitter-0.8.2: {{{ try:

Python >= 2.6

import json as simplejson except ImportError: try:

Python < 2.6

import simplejson

}}}

evansde77 commented 13 years ago

evansde: Patch with RSS Feed committed.

Can you open a ticket for Stu/Diego to make sure that lxml gets added as an external?

zdenekmaxa commented 13 years ago

maxa: lxml dependency - #1825 closing this ticket.

zdenekmaxa commented 13 years ago

maxa: Reimplement !RSSFeedSink via couch to spare lxml dependency here.

from #1825 discussion:

We already have a logger handler that writes to couch, so we could log each article as a couch document and then use a view or show to format the last N items as an RSS feed. There is apparently an RSS feed function in the Couch Book....

List function to make the RSS/ATOM feed: http://wiki.apache.org/couchdb/Formatting_with_Show_and_List http://guide.couchdb.org/draft/transforming.html

The nice thing about this is that you can also pull out stats from the alerts with other views in the couchapp. The CouchDB log handler is in https://svnweb.cern.ch/trac/CMSDMWM/browser/WMCore/trunk/src/python/WMCore/WMLogging.py

zdenekmaxa commented 13 years ago

maxa: Simon,

with reference to ticket:1712#comment:17 above:

I don't think CouchDB as such has a feature of easily exporting documents in RSS and/or Atom feed formats. Or I wasn't able to find it. The transforming.html you forwarded points to a Sofa blog example application which is capable of doing Atom feed export via its own couchapp: https://github.com/jchris/sofa/blob/master/vendor/couchapp/lib/atom.js This seems to depend on some extension javascript library, though it's not complicated. So, in a way we'd be replacing one dependency (lxml) by another.

Regarding your last comment referring to WMLogging.py: actually, there already is !CouchSink for alerts doing the same. If the final conclusion really is to generate the Atom feed via CouchDB, I guess that future !AtomSink will just make sure that !CouchSink is enabled and configured.

I just wanted to summarize, it may be easier to discuss further on IM.

zdenekmaxa commented 13 years ago

maxa: Replying to [comment:20 maxa]:

Simon,

with reference to ticket:1712#comment:17 above:

I don't think CouchDB as such has a feature of easily exporting documents in RSS and/or Atom feed formats. Or I wasn't able to find it. The transforming.html you forwarded points to a Sofa blog example application which is capable of doing Atom feed export via its own couchapp: https://github.com/jchris/sofa/blob/master/vendor/couchapp/lib/atom.js This seems to depend on some extension javascript library, though it's not complicated. So, in a way we'd be replacing one dependency (lxml) by another.

Regarding your last comment referring to WMLogging.py: actually, there already is !CouchSink for alerts doing the same. If the final conclusion really is to generate the Atom feed via CouchDB, I guess that future !AtomSink will just make sure that !CouchSink is enabled and configured.

I just wanted to summarize, it may be easier to discuss further on IM.

as discussed on IM with Simon:

zdenekmaxa commented 13 years ago

maxa: patch-series-maxa-alerts-fw-rss-lxml-removal-01 patch removes rss feed lxml implementation.

Dave, please review.

ps: some further feedback may still be put on this ticket from atom+couch stuff.

zdenekmaxa commented 13 years ago

maxa: * RSS, resp. Atom feed implementation for Alerts (visualisation) done under #1639