bgreenlee / pygtail

Pygtail reads log file lines that have not been read. It will even handle log files that have been rotated. Based on logcheck's logtail2 (http://logcheck.org)
GNU General Public License v2.0
251 stars 79 forks source link

High CPU usage #55

Closed joseaguardia closed 2 years ago

joseaguardia commented 4 years ago

Hello!

When I run the following python script, my server use too much CPU (load of 2.5 with 2 cores) and 90% is from that script:


#!/usr/bin/python
# -*- coding: utf-8 -*-

from pygtail import Pygtail
import os
import socket
import subprocess
import time

hostname = socket.gethostname()

while True:
    for line in Pygtail("/var/log/httpd/access.log"):
        if " 403 " in line:   
            #delete empty lines    
            line=line.rstrip()
            alert = subprocess.check_output("echo '%s' | awk '{print \"*%s* - IP: \" $1 \" \\n `\" $6  \" \" $7 \"`\"}' | tr -d '\"'" % (line, hostname), shell=True)
            print(alert)

I've tried to put a time.sleep(1) at the end of while loop, and at the end of for line..., however any line is processed (and get low CPU usage).

The website is on pre-production, so the usage is very low.

Can you help me, please?

Thanks!

ghost commented 4 years ago

Use the python re (regex) library instead of subprocess:

import re
...
re.search(r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', line).group()
...