Log Size Limit Problem - Githubissues

Hello,

I have increased my log_size_limit value (/etc/elsa_node.conf) at 3TB but my 
disk usage is still stock at 37%. 

Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vg_data-lv_nsm  3.6T  1.3T  2.2T  37% /nsm

I don't understand why the log size does not reach 3TB.

Here is my elsa_node.conf

{
        # Local mysql database connection info
        "database" : {
                "db": "syslog,port=50000",
                "data_db": "syslog_data",
                "dsn" : "dbi:mysql:database=syslog",
                "username" : "******",
                "password" : "******"
        },
        # Trivial directories where the system will coordinate locks.  Default should always be fine unless you change from "/usr/local/elsa"
        "lockfile_dir": "/opt/elsa/node/tmp/locks",
        # Tuning for number of indexes the system will handle.  There are temporary indexes which are (by default) 60 second batch-loaded indexes which get consolidated when the total number of logs in temporary indexes is over the setting for "perm_index_size."  By default, ELSA will consolidate its temporary indexes when it hits 10 million or temporary indexes are more than 40% of 200 (80).  If 200 permanent indexes are used before it hits the log size capacity, then it will prematurely overwrite permanent indexes.  Therefore, if you do not collect 10 million logs in less than 200 minutes, you will lose logs because they are overwritten prematurely.  So, if your logging rate is less than 833.3 logs per second, you will want to increase the number of indexes.
        "num_indexes": 200,
        # If you want to archive logs in addition to indexing them, leave this and/or edit the numbers.  Set percentage to 0 if you do not want to archive.
        "archive": {
                # Uncomment to establish a retention period in days for archive logs
                #"days": 90,
                #"percentage": 33,
                "percentage": 66,
                "table_size": 10000000
        },
        # Size limit for logs + index size.  Set this to be 90-95% of your total data disk space.
        "log_size_limit" : 3000000000000,
        "sphinx" : {
                # Uncomment to establish a retention period in days for indexed logs
                #"days": 14,
                # location for sphinx indexer, should not need to change unless you installed to a non-standard location
                "indexer": "/usr/bin/indexer",
                # emergency consolidation will be done if too many temp indexes pile up
                "allowed_temp_percent" : 40,
                # Check to be sure that temporary indexes (which use a lot of RAM) never go above this amount of RAM, or they will be prematurely consolidated.
                "allowed_mem_percent": 25,
                # This should not need to be changed unless you change the ELSA root dir
                "config_template_file" : "/opt/elsa/node/conf/sphinx.conf.template",
                # Host/port sphinx runs on.  Default should be fine.
                "host" : "127.0.0.1",
                "port" : 9312,
        "mysql_port" : 9306,
                # The actual config file sphinx will use, should not need to change unless you are installing to non-standard locations.
                "config_file" : "/etc/sphinxsearch/sphinx.conf",
                # Data directory for storing indexes.  Place on your biggest storage.
                "index_path" : "/nsm/elsa/data/sphinx",
                # This is the size of the temporary index batches.  You should probably leave as the default, but you can increase for fewer temp indexes, especially if you have low log volume.  The higher the value here, the longer it will take for new logs to be available to queries.
                "index_interval" : 60,
                # Setting for how many temp logs should be allowed before being consolidated.  This value should be fine for systems with more than 833 logs/second.
                "perm_index_size" : 10000000,
                # Where the optional stopwords file is
                "stopwords": {
                        "file": "/etc/sphinxsearch/sphinx_stopwords.txt",
                        "top_n": 0,
                        "interval": 0,
                        "whitelist": []
                },
                # Where the optional stopwords file is
                "pid_file": "/var/run/sphinxsearch/searchd.pid"
        },
        # Where to place the ELSA management logs (not the logs we are receiving)
        "logdir" : "/nsm/elsa/data/elsa/log",
    "mysql_dir": "/nsm/elsa/data/elsa/mysql",
        # How many concurrent log readers can run. 1 should be fine for up to 50k logs/sec on the same node.
        "num_log_readers" : 1,
        # Logging level management logs (does not affect logs received).
        "debug_level" : "TRACE",
        # Where to store the incoming logs before being loaded and processed.  You can set to /dev/shm on Linux for faster loading, but shouldn't be necessary.
        "buffer_dir" : "/nsm/elsa/data/elsa/tmp/buffers/",
        # Log any parsing errors?  This can be very verbose initially, but needed to tell if there are some subtle parsing problems.
        "log_parse_errors": 1,
        # Stats retention settings
        "stats" : {
                "retention_days": 365
        },
        # Minimum number of hosts we expect to get logs from, currently only used to record a warning message in the log if we don't get this many
        "min_expected_hosts": 2
}

Does someone have an idea ?

Thanks for your time
Original issue reported on code.google.com by Steve.Bo...@gmail.com on 11 Dec 2014 at 1:54
SafeAF / enterprise-log-search-and-archive

Log Size Limit Problem #230
