Service 503

[aaaronic]

This isn't quite the correct place to report this issue, I supposed, but the http://courtbot.herokuapp.com/ site is down as a 503 right now:

<h1>Application Error</h1>
<div class="article"> 
    <p>An error occurred in the application and your page could not be served.  Please try again in a few moments.</p>
    <p>If you are the application owner, check your logs for details.</p>
</div>

[aaaronic]

Note, you get the same response for http://courtbot.herokuapp.com/cases?q=valid_case_number as well.

[samharden]

Usually Heroku does that when an app is turned off or exceeds the number of hours allotted to it per month if it's running on a free dyno. Since the web address for the app isn't a custom domain I'm assuming that you're looking at a free version of the app.

[aaaronic]

That's the version being pointed to by the atlantaga.gov domain right now.

[aaaronic]

It's hard-coded as the action for the form here: https://github.com/codeforamerica/court.atlantaga.gov/blob/79b2b824e33161589427f63152b67140efa11672/site.js

[samharden]

Oh I know - It's not my app, and unless I had a way to get into the Heroku account for that one I couldn't tell you what's really going on. My best guess is that it was created as a "free dyno" on Heroku and has run out of free hours on the server.

[aaaronic]

@samharden ah, I'm sorry. Your having responded made me think you were one of the developers working on/with it.

[s2t2]

@kuanb any ideas?

[loumoore]

I work for CfA and am taking a look. We had run out of free hours as suggested above, so I upgraded to a hobby dyno but now we're getting an application error that keeps bringing it back down. I'm not familiar with the codebase so bear with me, but we're investigating.

[kuanb]

Hi all I'm just on phone right now but I think this app was shut down as it is not in use in production. Correct me if I'm wrong...

On Fri, Oct 28, 2016 at 6:36 PM, Lou Moorenotifications@github.com wrote:
I work for CfA and am taking a look. We had run out of free hours as suggested above, so I upgraded to a hobby dyno but now we're getting an application error that keeps bringing it back down. I'm not familiar with the codebase so bear with me, but we're investigating.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub, or mute the thread.

[aaaronic]

It's actively used to look up cases on the municipal court of Atlanta's website. So the 503s prevent any case lookups.

Aaron Hansen

On Oct 28, 2016 22:32, "Kuan Butts" notifications@github.com wrote:

Hi all I'm just on phone right now but I think this app was shut down as it is not in use in production. Correct me if I'm wrong...

On Fri, Oct 28, 2016 at 6:36 PM, Lou Moorenotifications@github.com wrote: I work for CfA and am taking a look. We had run out of free hours as suggested above, so I upgraded to a hobby dyno but now we're getting an application error that keeps bringing it back down. I'm not familiar with the codebase so bear with me, but we're investigating.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/codeforamerica/courtbot/issues/11#issuecomment-257065887, or mute the thread https://github.com/notifications/unsubscribe-auth/AB9UQiMrsZkb0q485Rpq2ENlCQ7Dro9fks5q4rBJgaJpZM4Kjb3M .

[loumoore]

We're still actively investigating -- we're treating this as a production outage and will resolve. We'll keep this thread updated. Thanks for your patience.

On Friday, October 28, 2016, Aaron Hansen notifications@github.com wrote:

It's actively used to look up cases on the municipal court of Atlanta's website. So the 503s prevent any case lookups.

Aaron Hansen

On Oct 28, 2016 22:32, "Kuan Butts" <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Hi all I'm just on phone right now but I think this app was shut down as it is not in use in production. Correct me if I'm wrong...

On Fri, Oct 28, 2016 at 6:36 PM, Lou Moore<notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote: I work for CfA and am taking a look. We had run out of free hours as suggested above, so I upgraded to a hobby dyno but now we're getting an application error that keeps bringing it back down. I'm not familiar with the codebase so bear with me, but we're investigating.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/codeforamerica/courtbot/issues/11#issuecomment- 257065887, or mute the thread https://github.com/notifications/unsubscribe-auth/ AB9UQiMrsZkb0q485Rpq2ENlCQ7Dro9fks5q4rBJgaJpZM4Kjb3M .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/codeforamerica/courtbot/issues/11#issuecomment-257067076, or mute the thread https://github.com/notifications/unsubscribe-auth/AAlOZYuUB79YXb0yDkR5DVtddv3Sa6Quks5q4rc5gaJpZM4Kjb3M .

[loumoore]

update: service has been restored as of 1:25am PDT.

details: we spent most of the evening attempting to troubleshoot the application and understand some major differences in the format of the data in the production courtbot database vs the courtbot code. the problem turned out to be that the database itself was a rather old version with old data, and needed to be rebuilt with the latest schema and data from the atlanta.gov csv file. doing that allowed us to restore service. we have also made sure the daily jobs to re-populate the database with fresh data are scheduled going forward, and we'll take a look at adding better monitoring as well next week.

let us know if there are any further issues.

[s2t2]

Thanks. What was the difference in data format between new and old? And perhaps most importantly, are the data posters aware of the need to post data in a consistent format?

There are other apps that also rely on this data, which might need updating as well.

On Oct 29, 2016 4:36 AM, "Lou Moore" notifications@github.com wrote:

update: service has been restored as of 1:25am PDT.

details: we spent most of the evening attempting to troubleshoot the application and understand some major differences in the format of the data in the production courtbot database vs the courtbot code. the problem turned out to be that the database itself was a rather old version with old data, and needed to be rebuilt with the latest schema and data from the atlanta.gov csv file. doing that allowed us to restore service. we have also made sure the daily jobs to re-populate the database with fresh data are scheduled going forward, and we'll take a look at adding better monitoring as well next week.

let us know if there are any further issues.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/codeforamerica/courtbot/issues/11#issuecomment-257079364, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRGp0qs7es4hibI7Mw8bLSWy4_JgBZOks5q4wWQgaJpZM4Kjb3M .

[loumoore]

data format may have been a confusing choice of words. there were no issues with the data being provided to us in csv format.

the problem was that our courtbot database schema was wrong. it's not obvious how it got in that state, but it's as if it had been created with an old development version. for example, the cases table had a varchar 'citation' column rather a json 'citations' column, and a varchar 'date' column rather than a date 'date' column. the data also appeared to be old and likely invalid -- the table only had 855 rows, with dates like 'March 18th' and unstructured citation data like '123456'.

we realized eventually that the app is designed to have its database blown away and re-created from atlanta.gov csv files daily. doing that resulted in the database being re-created with the proper schema which is consistent with both the courtbot application code and the atlanta.gov api consumer's expectations.

the new data was pulled in from the most recent http://courtview.atlantaga.gov/courtcalendars/court_online_calendar/codeamerica..csv file and resulted in 24862 rows of valid-looking case data. (we did back up the original database before doing this of course).

so, again, the data being provided to us was not a problem. ingesting the atlanta.gov csv data worked fine once it was set up to do that again.

if there are other services that rely on this we'd be happy to validate that they are still operating as expected if you can point me in their direction. but unless there are other data sources we are supposed to be importing, it looks like no further action is required to fully restore service.

there will be further action on our part to improve operationally so we can avoid getting into this situation again. documentation, monitoring, and better change visibility all would have been helpful. we'll conduct a post-mortem to determine the actions we'll take.

[aaaronic]

I can confirm that the case I was originally looking for now shows up as expected. Thanks for resolving the issue.