soprasteria / cybersecurity-miw

Cybersecurity Mobile Investigation Workstation, a C++ Map-Reduce for Log Analysis
Other
32 stars 12 forks source link

Miw is freezing on a simple case #5

Open BestPig opened 7 years ago

BestPig commented 7 years ago

My goal is to count per user, the average of the second value

Here is my exemple of log file I used sample.log

user 1000
user 500
{
    "format_name":"test",
    "delims":" \"",
    "quotechar":"",
    "fields":[
        {
            "name":"user",
            "pos":0,
            "type":"string",
            "key": true
        },
        {
            "name":"rbytes",
            "pos":1,
            "type":"int",
            "aggregated":true,
            "aggregation":"mean"
        }
    ]
}

The command I run

miw -fnames sample.log -ofname test.csv -format_name ../formats/test -output_format csv
files=./data/one.log
I0411 16:30:39.882609  4253 job.cc:122] files size=1
I0411 16:30:39.882654  4253 job.cc:127] Processing file=sample.log

miw seems frozen, it should be a quick computing

beniz commented 7 years ago

Final mean value is now reported instead of numerator and denominator separated. As for the freezing, with fewer lines than cores, you need to use -map_tasks 1, otherwise the leftover cores do freeze... Definitely a bug, but not critical since having more logs than cores is more common than the opposite!