Parses Terminating Server Side.

[GoogleCodeExporter]

Hello,

We've successfully setup this on our site and are able to SSH in to run the
parses locally on our host. The problem, we're on shared hosting and the
script is terminating after 4 parses. My thought is that the host doesn't
like use using that many CPU cycles continuously.

Is there a way to pause the script between attempts so that we can do this
all server side?

We're able to run the script on our local machines just fine, but we're
attempting to write an upload script which runs the parse transparently to
the people we give access to uploads to.

Original issue reported on code.google.com by RMagn...@gmail.com on 2 Apr 2009 at 6:55

[GoogleCodeExporter]

I'd suggest using the -split command to split the combined log into individual 
files, 
then parse those one by one.

Overall, that is slower, but the individual parsing runs take less time.

Original comment by gergely....@gmail.com on 2 Apr 2009 at 7:18

[GoogleCodeExporter]

You can force it to slow itself down by adding something like "-throttle 1" or 
"-throttle 2". The higher the 
number the slower it will be.

Original comment by gianmerlino@gmail.com on 2 Apr 2009 at 7:45

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Would the two of you mind providing a link or explaining what those things do 
exactly?

The -Split command, how is it doing the splits? Based on encounter or date? 
We're
terminating by the 4th encounter. 

for the -throttle 1,2,etc. is there a percentage as to what 1/2/3/etc throttles 
the
CPU usage to. 

Either of these sound like good solutions, and speed isn't a problem. I'd just 
like
something that is shared hosting friendly. 

Original comment by RMagn...@gmail.com on 2 Apr 2009 at 8:28

[GoogleCodeExporter]

-split splits by encounter (and if you have -trash, it will put any non-boss 
stuff
into a separate file).

As far as I understand the code, -throttle N makes stasis wait/sleep N 
milliseconds
between each line read.

In your case, I'd go with -split, it should be more reliable against the 
problem you
described. I believe so because most scenarios where a program is stopped after 
a
while is usually done so after it's been running for a set amount of time. 
Throttle
will not help against that, split will.

However, split isn't exactly a speed demon, either, so if your logs are huge, it
might still run too long.

Original comment by gergely....@gmail.com on 3 Apr 2009 at 1:14

[GoogleCodeExporter]

-throttle 1 is working fine. Yes, it's slower, but it's doing the job. 

Original comment by RMagn...@gmail.com on 6 Apr 2009 at 2:17

[GoogleCodeExporter]

Good to know.

Just so you're aware, -throttle is really simple: it works by sleeping for a 
number of milliseconds in between 
processing each line. A 100,000 line log would take 100 seconds longer to parse 
with "-throttle 1", 200 seconds 
longer with "-throttle 2", and so on.

Original comment by gianmerlino@gmail.com on 6 Apr 2009 at 2:36

• Changed state: Fixed

[GoogleCodeExporter]

What is the correct syntax for the -split command? I cannot seem to get it to 
work
correctly. Sorry to necro this ticket.

Thanks

Original comment by cogentcr...@gmail.com on 19 Apr 2009 at 9:36

[GoogleCodeExporter]

Think it may be an outdated option, I haven't been able to get it working either

Original comment by wilshire...@gmail.com on 19 Apr 2009 at 9:52

[GoogleCodeExporter]

Well, The first post is less then 30 days old so I would not think it is a out 
of
date option.

Original comment by cogentcr...@gmail.com on 19 Apr 2009 at 11:56