make an AdminTasks API and corresponding model with v2.0

[bepetersn]

The problem this would solve is the monitoring of our server, #67. By creating an API, to which our administrative tasks can POST their results, we will be setting ourselves up to create a dashboard interface, if desired. Listing all admin tasks, as opposed to keeping a running status of the project's status, allows us to focus on the events of our server as they happen -- which is typically only once a day, anyway.

Equally as important as setting up a hub for pulling the results of admin tasks, we can use handlers underneath the hood of the POST functionality to perform tasks such as pushing email notifications via our Google Group, or even things like #246, #271, #272, and #273.

This new API should be backed with Flask, SqlAlchemy, and our first PostgreSQL database table for v2.0. I would also like to convert the Version model into a SQL-based model, so that we can reference it with each administrative task.

Here's my preliminary spec for the fields that should be on the "AdminTasks" model:

• started_at
• elapsed_time
• name: e.g. "scrape", "audit", "request summary"
• description: e.g. "FAILURE: couldn't reach sheriff's site"
• version: ref. to current version of codebase

[bepetersn]

What do you think, @nwinklareth? I know you're gonna be away the coming meet ... respond when you have the time.

[nwinklareth]

The word task in field names is redundant. datetime_started => started_at summary => description -- because failure messages are not summaries

I am unsure of the action_required field. I see this database as a a data warehouse as opposed to a task to be done table. What about others?

[bepetersn]

Thanks for the corrections on fieldnames.

I also see what you're saying with regard to data vs. action-based information. This brings to light one of the last conflicts I had about this way of handling the dashboard, which was, briefly: how can we bring out the event-driven side of this design problem?

Because it is about events, and so this kind of "handling" code really belongs where we're working with events. In light of this, I went back to an area of software systems I have been neglecting to learn about, which is message queues, for Python this means software like Celery, RabbitMQ, and a bunch of others.

I recommend reading this article, it did a decent job comparing different solutions: http://www.darkcoding.net/software/choosing-a-message-queue-for-python-on-ubuntu-on-a-vps/

Their conclusion is that using Redis is the simplest. I also found a library called RQ for streamlining Redis queue usage, so I think I'm going to look into how to integrate this into our current system. One advantage to this approach over using a SQL database to solve server monitoring is that it runs in memory, instead of being stored indefinitely in a table.

I CAN think of a few reasons we might want to look back at a history of admin tasks, but they are mostly secondary concerns. I think that our primary problem is an event / messaging / subscription problem and not a persistence one.

[bepetersn]

I'm still trying to work out exactly what this implies about my whole "adminTasks" API, or the more important part, the ability to track the current status of all our administrative components.

Here's some brief thoughts, as I feel this issue is now very fragmented and needs direction:

• We've talked for awhile about having some part of the scraper always running on the server. I think this is the answer to the need for that "always-on" component.
• We still need an API that shows the status of our project. When I conceptualized this before, I couldn't think how to keep this up-to-date constantly (as opposed to just after our once-a-day programs run), and thinking that we wouldn't need anything other than this 80-90% of the time, I ignored the issue. But if we have a program constantly running on the server, IT can update the status API.
• Our status API then might just be a shallow representation that gets updated by this Monitor/Controller program.
• We could still keep a history of AdminTasks if desired, which the Monitor could pull from to keep the status API up to date. This might not be an essential component, however, and we could try to do without it during a first iteration.