search

syslog-ng / syslog-ng

syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
https://www.syslog-ng.com
Other
2.14k stars 476 forks source link

Using grouping-by() on a large file is very memory intensive #4457

Open faxm0dem opened 1 year ago

faxm0dem commented 1 year ago

syslog-ng

Version of syslog-ng

4.1.1

Platform

RHEL8

Issue

Failure

We're trying to use grouping-by() for a new use-case, that is to simply aggregate a certain number of logfiles. Whe running, syslog-ng eats up all ram and gets oom-killed by linux.

Steps to reproduce

rm -f /tmp/{c,R,p}; syslog-ng -c /tmp/c -R /tmp/R -p /tmp/p -f syslog-ng-dcache-webdav-curated-for-github.conf --no-caps -dv

Configuration

#
@version: 4.0
@include scl.conf

source s_dcache_webdav {
  file(
    "dcache-webdav.log",
    flags(no-parse),
  );
};

parser p_dcache_webdav {
  kv-parser(
    template("$MESSAGE"),
    value-separator("="),
    prefix("dcache."),
  );
};

parser p_date_dcache_webdav {
  date-parser(
    format("%Y-%m-%dT%T.%f%Z"),
    template("${dcache.ts}"),
  );
};

parser p_aggregate_dcache_webdav {
  grouping-by(
    where(
      "${dcache.event}" == "org.dcache.webdav.request"
    ),
    timeout(30),
    key("${dcache.request.method}-${dcache.response.code}"),
    scope(global),
    aggregate(
      inherit-mode(none),
      value('MESSAGE', "Aggregated dcache webdav ${context-id}"),
      value('dcache.aggregation.min', "$(min ${dcache.duration})"),
      value('dcache.aggregation.max', "$(max ${dcache.duration})"),
      value('dcache.aggregation.average', "$(average ${dcache.duration})"),
      value('dcache.aggregation.sum', "$(sum ${dcache.duration})"),
      value('dcache.aggregation.count', "$(- $(context-length) 1)"),
      value('dcache.request.method', "${dcache.request.method}@1"),
      value('dcache.response.code', "${dcache.response.code}@1"),
      value('dcache.event', 'org.dcache.webdav.request.aggregated')
    ),
  );
};

destination d_dcache_parsed_webdav {
  file(
    "/tmp/dcache_parsed_webdav.json",
    template("$(format-json -s nv-pairs)\n"),
  );
};

log {
  source(s_dcache_webdav);
  parser(p_dcache_webdav);
  parser(p_date_dcache_webdav);
  parser(p_aggregate_dcache_webdav);
  filter {
    "${dcache.event}" == "org.dcache.webdav.request.aggregated";
  };
  destination {
    file("/dev/stdout", template("$(format-json -s nv-pairs @timestamp=$S_ISODATE)\n"));
  };
};

Input file

level=WARN ts=2023-04-28T00:00:39.035+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/calexpBackground/20230202/y/y_sim_1.4/282353/calexpBackground_LSSTCam-imSim_y_y_sim_1_4_282353_R30_S10_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.fits response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53686 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=11
level=WARN ts=2023-04-28T00:00:39.054+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/calibrate_metadata/20221130/r/r_sim_1.4/242464/calibrate_metadata_LSSTCam-imSim_r_r_sim_1_4_242464_R02_S00_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.json response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53686 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=12
level=WARN ts=2023-04-28T00:00:39.061+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/calibrate_metadata/20221120/u/u_sim_1.4/235069/calibrate_metadata_LSSTCam-imSim_u_u_sim_1_4_235069_R14_S02_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.json response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53696 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=9
level=WARN ts=2023-04-28T00:00:39.062+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/isr_log/20230719/400464/isr_log_LSSTCam-imSim_g_sim_1_4_400464_R24_S22_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.json response.code=202 response.reason="OK" socket.remote=1.2.3.4:53694 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=11
level=WARN ts=2023-04-28T00:00:39.064+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/srcMatch/20230809/r/r_sim_1.4/414866/srcMatch_LSSTCam-imSim_r_r_sim_1_4_414866_R33_S10_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.fits response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53692 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=14
level=WARN ts=2023-04-28T00:00:39.067+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/calexpBackground/20221026/g/g_sim_1.4/221616/calexpBackground_LSSTCam-imSim_g_g_sim_1_4_221616_R34_S02_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.fits response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53700 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=8
level=WARN ts=2023-04-28T00:00:39.070+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/writeSourceTable_log/20221023/g/g_sim_1.4/219211/writeSourceTable_log_LSSTCam-imSim_g_g_sim_1_4_219211_R34_S20_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.json response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53686 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=12
level=WARN ts=2023-04-28T00:00:39.071+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/calexp/20221102/z/z_sim_1.4/226981/calexp_LSSTCam-imSim_z_z_sim_1_4_226981_R13_S10_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.fits response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53698 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=12
level=WARN ts=2023-04-28T00:00:39.074+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/calexpBackground/20230716/i/i_sim_1.4/397331/calexpBackground_LSSTCam-imSim_i_i_sim_1_4_397331_R02_S01_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.fits response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53696 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=10
level=WARN ts=2023-04-28T00:00:39.079+0200 event=org.dcache.webdav.request request.method=HEAD request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_2/20230427T115632Z/characterizeImage_metadata/20230803/z/z_sim_1.4/409077/characterizeImage_metadata_LSSTCam-imSim_z_z_sim_1_4_409077_R14_S02_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_2_20230427T115632Z.json response.code=404 response.reason="Not Found" socket.remote=1.2.3.4:53692 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=9
level=INFO ts=2023-04-28T00:09:43.628+0200 event=org.dcache.webdav.request request.method=OPTIONS request.url=https://w.x.y.z:2880/ response.code=200 response.reason=OK socket.remote=1.2.3.4:43132 user-agent=python-requests/2.28.1 user.mapped=nobody duration=17
level=INFO ts=2023-04-28T00:09:43.718+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=31
level=INFO ts=2023-04-28T00:09:43.765+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/icSrc_schema/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=45
level=INFO ts=2023-04-28T00:09:44.164+0200 event=org.dcache.webdav.request request.method=PUT request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/icSrc_schema/icSrc_schema_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.fits response.code=307 location=https://nodeacli396.example.com:30000/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/icSrc_schema/icSrc_schema_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.fits?dcache-http-uuid=3a1fc964-a559-44e3-94d1-7d1dbd8a6fa1&dcache-http-ref=https%3A%2F%2Fccdavrubinint.example.com%3A2880 socket.remote=1.2.3.4:43182 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] transaction=door:WebDAV-nodeacli404@webdav-nodeacli404Domain:AAX6WJ0kMLA:1682633384037000 duration=150
level=INFO ts=2023-04-28T00:09:44.966+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/src_schema/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=45
level=INFO ts=2023-04-28T00:09:45.157+0200 event=org.dcache.webdav.request request.method=PUT request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/src_schema/src_schema_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.fits response.code=307 location=https://nodeacli395.example.com:30000/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/src_schema/src_schema_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.fits?dcache-http-uuid=506c85a3-a99e-422a-aa57-baf4fed81002&dcache-http-ref=https%3A%2F%2Fccdavrubinint.example.com%3A2880 socket.remote=1.2.3.4:43306 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] transaction=door:WebDAV-nodeacli404@webdav-nodeacli404Domain:AAX6WJ0zjkg:1682633385045001 duration=136
level=INFO ts=2023-04-28T00:09:45.835+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/calibrate_config/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=253
level=INFO ts=2023-04-28T00:09:46.389+0200 event=org.dcache.webdav.request request.method=PUT request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/calibrate_config/calibrate_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py response.code=307 location=https://nodeacli450.example.com:30000/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/calibrate_config/calibrate_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py?dcache-http-uuid=c45cae32-96a5-4c36-8baa-55e318d08244&dcache-http-ref=https%3A%2F%2Fccdavrubinint.example.com%3A2880 socket.remote=1.2.3.4:43434 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] transaction=door:WebDAV-nodeacli404@webdav-nodeacli404Domain:AAX6WJ1Aljg:1682633385896000 duration=513
level=INFO ts=2023-04-28T00:09:46.808+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/isr_config/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=38
level=INFO ts=2023-04-28T00:09:47.293+0200 event=org.dcache.webdav.request request.method=PUT request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/isr_config/isr_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py response.code=307 location=https://nodeacli398.example.com:30000/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/isr_config/isr_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py?dcache-http-uuid=d7bf466c-d17e-4886-baa2-ac1b3b654fb9&dcache-http-ref=https%3A%2F%2Fccdavrubinint.example.com%3A2880 socket.remote=1.2.3.4:43528 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] transaction=door:WebDAV-nodeacli404@webdav-nodeacli404Domain:AAX6WJ1PHPg:1682633386857000 duration=465
level=INFO ts=2023-04-28T00:09:47.785+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/characterizeImage_config/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=38
level=INFO ts=2023-04-28T00:09:47.925+0200 event=org.dcache.webdav.request request.method=PUT request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/characterizeImage_config/characterizeImage_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py response.code=307 location=https://nodeacli401.example.com:30000/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/characterizeImage_config/characterizeImage_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py?dcache-http-uuid=00cc4281-fbb6-4066-bd47-b312c03975bb&dcache-http-ref=https%3A%2F%2Fccdavrubinint.example.com%3A2880 socket.remote=1.2.3.4:43580 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] transaction=door:WebDAV-nodeacli404@webdav-nodeacli404Domain:AAX6WJ1eMFg:1682633387844001 duration=109
level=INFO ts=2023-04-28T00:09:48.050+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/writeSourceTable_config/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=32
level=INFO ts=2023-04-28T00:09:48.256+0200 event=org.dcache.webdav.request request.method=PUT request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/writeSourceTable_config/writeSourceTable_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py response.code=307 location=https://nodeacli448.example.com:30000/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/writeSourceTable_config/writeSourceTable_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py?dcache-http-uuid=3de71a30-dead-4dc1-9e63-8b5be99dac60&dcache-http-ref=https%3A%2F%2Fccdavrubinint.example.com%3A2880 socket.remote=1.2.3.4:43604 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] transaction=door:WebDAV-nodeacli404@webdav-nodeacli404Domain:AAX6WJ1iCLg:1682633388092000 duration=189
level=INFO ts=2023-04-28T00:09:48.543+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/transformSourceTable_config/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=42
level=INFO ts=2023-04-28T00:09:48.633+0200 event=org.dcache.webdav.request request.method=PUT request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/transformSourceTable_config/transformSourceTable_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py response.code=307 location=https://nodeacli404.example.com:30000/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/transformSourceTable_config/transformSourceTable_config_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.py?dcache-http-uuid=60b86d28-1e52-47a6-b426-39ad0331f592&dcache-http-ref=https%3A%2F%2Fccdavrubinint.example.com%3A2880 socket.remote=1.2.3.4:43656 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] transaction=door:WebDAV-nodeacli404@webdav-nodeacli404Domain:AAX6WJ1pjoE:1682633388589001 duration=72
level=INFO ts=2023-04-28T00:09:54.446+0200 event=org.dcache.webdav.request request.method=MKCOL request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/packages/ response.code=201 socket.remote=1.2.3.4:40292 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] duration=32
level=INFO ts=2023-04-28T00:09:54.846+0200 event=org.dcache.webdav.request request.method=PUT request.url=https://w.x.y.z:2880/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/packages/packages_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.yaml response.code=307 location=https://nodeacli454.example.com:30000/pnfs/example.com/abcd/repos/dp02/2.2i/runs/FRDF/DP0.2/validation_dcache/step1_3/20230427T135920Z/packages/packages_2_2i_runs_FRDF_DP0_2_validation_dcache_step1_3_20230427T135920Z.yaml?dcache-http-uuid=78ef0bd6-63e8-4256-9466-aac475fb8399&dcache-http-ref=https%3A%2F%2Fccdavrubinint.example.com%3A2880 socket.remote=1.2.3.4:44386 user-agent=python-requests/2.28.1 user.mapped=43757:1021,1021[qKfOJZ/O:d1Uolx/P] transaction=door:WebDAV-nodeacli404@webdav-nodeacli404Domain:AAX6WJ3Ewig:1682633394568000 duration=309

output logs

{"dcache":{"response":{"code":"404"},"request":{"method":"HEAD"},"event":"org.dcache.webdav.request.aggregated","aggregation":{"sum":97,"min":8,"max":14,"count":9,"average":10}},"MESSAGE":"Aggregated dcache webdav ","@timestamp":"2023-04-28T00:00:39+02:00"}
{"dcache":{"response":{"code":"202"},"request":{"method":"HEAD"},"event":"org.dcache.webdav.request.aggregated","aggregation":{"sum":11,"min":11,"max":11,"count":1,"average":11}},"MESSAGE":"Aggregated dcache webdav ","@timestamp":"2023-04-28T00:00:39+02:00"}
{"dcache":{"response":{"code":"200"},"request":{"method":"OPTIONS"},"event":"org.dcache.webdav.request.aggregated","aggregation":{"sum":17,"min":17,"max":17,"count":1,"average":17}},"MESSAGE":"Aggregated dcache webdav ","@timestamp":"2023-04-28T00:09:43+02:00"}
{"dcache":{"response":{"code":"201"},"request":{"method":"MKCOL"},"event":"org.dcache.webdav.request.aggregated","aggregation":{"sum":556,"min":31,"max":253,"count":9,"average":61}},"MESSAGE":"Aggregated dcache webdav ","@timestamp":"2023-04-28T00:09:54+02:00"}
{"dcache":{"response":{"code":"307"},"request":{"method":"PUT"},"event":"org.dcache.webdav.request.aggregated","aggregation":{"sum":1943,"min":72,"max":513,"count":8,"average":242}},"MESSAGE":"Aggregated dcache webdav ","@timestamp":"2023-04-28T00:09:54+02:00"}
bazsi commented 1 year ago

30s worth of those logs might be huge. What happens in this case is that syslog-ng would store all messages in a context that relate to the same key()

You could probably get away with smaller timeouts.

bazsi commented 1 year ago

Some additional information: this might not be a very good fit for grouping-by() as it will form the "groups" in memory, meaning that all the messages that have the same key within 30s will be stored in memory until a trigger criteria is hit, which would never happen if you have a continuous stream of messages pushing out the timeout continuously.

grouping-by() was originally designed to process events that span multiple messages but are more limited in time, e.g. multiple messages from an SMTP transaction that we would like to turn into a single message.

What would be an easy solution here is to add a max-messages / max-time / max-context-size limit, which would trigger the aggregation and close the context, thereby freeing memory.

Would that help your use-case? How often do you want to emit these aggregations?

faxm0dem commented 1 year ago

The way we solve this in riemann for instance is to not store all the context, but only the result of the (partial) consolidation function min/max/count/moving average. I understand this would involve quite some rewrite. I think what you propose would suit us well in this use-case : a max-memory-size or max-context-size. The downside would be that we would get more output, but then I guess we can chain the output through another grouping-by() as this won't affect min/max/avg.