search

cpan-testers / cpantesters-project

A meta-project for tracking CPAN Testers project goals
6 stars 1 forks source link

Metabase update feeds are not working / not useful #13

Closed preaction closed 6 years ago

preaction commented 7 years ago

It is a nice feature of CPANTesters that it has a set of useful feeds for each part of its process, starting with data incoming to the Metabase.

There is a text-based log of the latest reports to the Metabase available here at http://metabase.cpantesters.org/tail/log.txt, but it is not ideal for getting a feed of incoming test reports. There is an ATOM feed at http://metabase.cpantesters.org/tail/recent.xml, but it has not been updated since 2011.

We need to fix the ATOM feed to be working again, and we may want to think about other kinds of data feeds.

xdg commented 7 years ago

The Atom feed cronjob is commented out. That may have been done to save money as every request incurs SimpleDB query costs. What we really need is a way to do one query and generate multiple outputs from it.

xdg commented 7 years ago

I should probably remove the old XML file so it 404's rather than just serves up stale data.

preaction commented 7 years ago

Okay, it seems like I should be able to modify the tail-recent-reports.pl to also update the Atom feed, and we may be able to cut out the querying altogether if we add the logging to the web app that writes the data (and if the web app is the only way to write the data, we can do other fun things like WebSockets pretty easily, if we really want to encourage this sort of thing).

preaction commented 7 years ago

As we discussed in e-mail, we could also push to the log directly from the Metabase::Web application. But considering barbie's response to #14, it might be better to have Metabase::Web write to a websocket feed. Locally, the log could be generated from that websocket feed (if we want it to be complicated like that), but importantly the CPANTesters backend can have multiple workers subscribe to the feed and improve processing speed

preaction commented 6 years ago

The backend processing is now powered by Minion, and the Metabase shim API directly writes new jobs to the Minion queue. This isn't 100% perfect, it'd be nicer if they were not so tightly coupled, but it's enough for now.