Robustness of OLAC site

[GoogleCodeExporter]

To ensure availability of key OLAC web pages, we should not be generating them 
dynamically 
from a database.  Instead, we could cache these pages and serve cached versions 
if the database 
is not reachable.  The PHP script could test if the database is reachable, and 
serve a cached 
version of the content if it is not.  This still depends on PHP working, and 
another solution is to 
have a buildbot that periodically checks out the svn repository, builds new 
content, and commits 
the results back into the repository from where it is served to the web.  This 
way, if the build 
process breaks we still have content that can be viewed (it just gets stale).

Some pages that could be treated in this way:
http://www.language-archives.org/archives.php
http://www.language-archives.org/services.php4
http://www.language-archives.org/register/archive_list.php4
http://www.language-archives.org/metrics/*
the individual archive pages (olac-archive metadata)

Original issue reported on code.google.com by StevenBird1 on 6 Jan 2009 at 10:55

[GoogleCodeExporter]

I'm afraid this will increase complexity of our software infrastructure. Also, 
I'm
not even sure if this is feasible at all for web 2.0 apps like our metrics page.

I think the instability comes from the complexity of our infrastructure where
everything (NFS, database server, web server, etc.) is intermingled and shared 
by
everyone. One person makes a mistake, and many other people are affected. 
Failure in
one component leads to a failure in a larger system.

So, instead, it maybe possible for us to obtain an independent server with 
LAMP. We
did this already for dspace server. (With VM technology, it has become quite 
easy
thing to do.) This doesn't have to mean that we have to change anything in the 
DNS
record. We can use apache proxy to tunnel the l-a.org contents from the virtual
server to the current web server.

Original comment by haepal on 26 Jan 2009 at 7:20

• Changed state: Accepted

[GoogleCodeExporter]

Agree that we want to consolidate our infrastructure in a replicable 
environment. 
Then it is a later step to set up the automatic building process.

Original comment by StevenBird1 on 26 Jan 2009 at 10:16

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Changing status to "Started."
Still waiting a response from Systems.

Original comment by haepal on 6 Feb 2009 at 4:43

• Changed state: Started

[GoogleCodeExporter]

we've secured a server. I'll start working on setting up a copy of l-a.org 
there.

Original comment by haepal on 19 Feb 2009 at 9:30

[GoogleCodeExporter]

Status update: Not much progress for the last couple of month since there were 
enough
tasks with higher priority. The basic infrastructure components are there. I 
still
need to fill in the details. Especially, I'm trying to implement a central site
configuration database to make it easy for different components of the OLAC web 
site
to share information and collaborate.

Original comment by haepal on 27 Jul 2009 at 3:24

• Removed labels: Milestone-2009-01

[GoogleCodeExporter]

I've pointed the DNS records for language-archives.org to the new server.

Original comment by StevenBird1 on 2 Oct 2009 at 5:15

[GoogleCodeExporter]

Unfortunately, it just gives the default Apache server page "It works!".  I'm
changing the address back, but setting new.language-archives.org to point to
nltk.ldc.upenn.edu.

Original comment by StevenBird1 on 2 Oct 2009 at 8:20

[GoogleCodeExporter]

Considering that (1) http://nltk.ldc.upenn.edu/ returns "It works!", and (2) 
you've
set new.language-archives.org to point to NLTK.ldc, I suspect you originally 
set it
to nltk.ldc instead of olac.ldc.

Original comment by haepal on 2 Oct 2009 at 2:23

[GoogleCodeExporter]

Fixed now.

Original comment by StevenBird1 on 2 Oct 2009 at 9:19

[GoogleCodeExporter]

Made a backup copy of current web (revision 1248) under the
"web_old_server_before_oct_09" tag.

I will start updating the trunk with changes that I made for the new server.

Original comment by haepal on 2 Oct 2009 at 10:15

• Added labels: Milestone-2009-11

[GoogleCodeExporter]

The new server has been running for more than a month without big trouble.

Original comment by haepal on 14 Dec 2009 at 6:13

[GoogleCodeExporter]

The remaining issue are:

(1) latency
    Andrea thinks this is a network issue coming in and out of penn
    Consider doing a stress test on the virtual server
(2) migration to the production server
(3) implementation of the email alarm

Original comment by haepal on 24 Mar 2010 at 2:00

[GoogleCodeExporter]

Original comment by haepal on 24 Mar 2010 at 2:01

• Added labels: Milestone-2010-09
• Removed labels: Milestone-2009-11

[GoogleCodeExporter]

Re: comment #13

It seems that (1) and (2) have been resolved.

I've asked systems about (3).

Original comment by haepal on 23 Sep 2010 at 8:50

[GoogleCodeExporter]

I believe all the issues have been resolved.

Original comment by haepal on 18 Nov 2010 at 8:23

• Changed state: Fixed

[GoogleCodeExporter]

Do we have any recent measures of server uptime and latency?  I have the 
impression we are not getting better than 99% uptime.

Original comment by StevenBird1 on 10 Feb 2011 at 8:30

• Changed state: Accepted

[GoogleCodeExporter]

The http server was available for 99.3% of the time for last 62 days.

I see that there were bad outages on 1/7, 1/14, 2/1 and 2/3.

     Status: UP (Reachable since Feb 05, 03:21:55)
      Probe: HTTP (port 80)
    Up Time: n/a
  Availability:           99.3 % (of 61 days, 23 hours, 29 minutes)
  TCP Failures:           0.01 % (of 177272 total attempts)
  Short-term Packet Loss:  0.0 % (of 100 last attempts) 
  Recent Failure: 2 attempts at Feb 05, 03:21:01
  Response time:  61 msec
Recent Outages:
  02/05 03:21:01: DOWN for 54 seconds
  02/03 10:54:08: DOWN for 39 minutes, 6 seconds
  02/01 10:42:53: DOWN for 1 hour, 5 minutes
  01/29 03:20:50: DOWN for 1 minute, 6 seconds
  01/27 03:25:34: DOWN for 18 seconds
  01/14 16:05:14: DOWN for 2 hours, 53 minutes
  01/09 02:26:16: DOWN for 18 seconds
  01/07 04:12:34: DOWN for 4 hours, 21 minutes
  01/04 03:42:45: DOWN for 18 seconds
  12/24 03:18:25: DOWN for 25 seconds
Last updated Feb 10, 15:51:31; interval: 30 seconds

Original comment by haepal on 10 Feb 2011 at 8:59