search

Concordium / Testnet3-Challenges

This repo is dedicated to Concordium Incentivized Testnet3.
https://developers.concordium.com/
Apache License 2.0
325 stars 471 forks source link

concordium-node-retrieve-logs output is truncating after 2gb #955

Closed EmreNOP closed 3 years ago

EmreNOP commented 3 years ago

Whenever, i try to get logs with concordium-node-retrieve-logs, created file is just 2gb. and noticed that my latest 3 days logs are not in concordium-testnet-node.log file

and than ive got my logs directly from docker with below command day by day.

184 docker container logs 49815d8caf87 --since 2020-10-14T00:00:00 --until 2020-15-00T00:00:00 > 14.log 185 docker container logs 49815d8caf87 --since 2020-10-14T00:00:00 --until 2020-10-15T00:00:00 > 14.log 186 docker container logs 49815d8caf87 --since 2020-10-15T00:00:00 --until 2020-10-16T00:00:00 > 15.log 187 docker container logs 49815d8caf87 --since 2020-10-16T00:00:00 --until 2020-10-17T00:00:00 > 16.log 188 docker container logs 49815d8caf87 --since 2020-10-17T00:00:00 --until 2020-10-18T00:00:00 > 17.log 189 docker container logs 49815d8caf87 --since 2020-10-18T00:00:00 --until 2020-10-19T00:00:00 > 18.log 190 docker container logs 49815d8caf87 --since 2020-10-19T00:00:00 --until 2020-10-20T00:00:00 > 19.log 191 docker container logs 49815d8caf87 --since 2020-10-20T00:00:00 --until 2020-10-21T00:00:00 > 20.log 192 docker container logs 49815d8caf87 --since 2020-10-21T00:00:00 --until 2020-10-22T00:00:00 > 21.log 193 docker container logs 49815d8caf87 --since 2020-10-22T00:00:00 --until 2020-10-23T00:00:00 > 22.log

xx@xx:~/Documents/concordium-software$ cat 14.log | grep "Consensus layer started" 2020-10-14T19:46:06.038473951Z: INFO: Consensus layer started xx@xx:~/Documents/concordium-software$ cat 15.log | grep "Consensus layer started" xx@xx:~/Documents/concordium-software$ cat 16.log | grep "Consensus layer started" 2020-10-16T08:45:01.627768980Z: INFO: Consensus layer started 2020-10-16T16:39:52.995671261Z: INFO: Consensus layer started xx@xx:~/Documents/concordium-software$ cat 17.log | grep "Consensus layer started" 2020-10-17T09:46:21.915731601Z: INFO: Consensus layer started 2020-10-17T16:06:42.717900907Z: INFO: Consensus layer started xx@xx:~/Documents/concordium-software$ cat 18.log | grep "Consensus layer started" 2020-10-18T16:44:12.223313810Z: INFO: Consensus layer started xx@xx:~/Documents/concordium-software$ cat 19.log | grep "Consensus layer started" 2020-10-19T09:43:11.622628250Z: INFO: Consensus layer started 2020-10-19T16:49:47.926726104Z: INFO: Consensus layer started xx@xx:~/Documents/concordium-software$ cat 20.log | grep "Consensus layer started" 2020-10-20T07:58:13.377478326Z: INFO: Consensus layer started 2020-10-20T19:25:55.419950605Z: INFO: Consensus layer started xx@xx:~/Documents/concordium-software$ cat 21.log | grep "Consensus layer started" 2020-10-21T09:38:57.821702249Z: INFO: Consensus layer started 2020-10-21T16:24:17.101517849Z: INFO: Consensus layer started xx@xx:~/Documents/concordium-software$ cat 22.log | grep "Consensus layer started" 2020-10-22T09:08:05.716988975Z: INFO: Consensus layer started 2020-10-22T17:11:49.271294889Z: INFO: Consensus layer started xx@xx:~/Documents/concordium-software$ cat concordium-testnet-node.log | grep "Consensus layer started" 2020-10-14T19:46:06.038473951Z: INFO: Consensus layer started 2020-10-16T08:45:01.627768980Z: INFO: Consensus layer started 2020-10-16T16:39:52.995671261Z: INFO: Consensus layer started 2020-10-17T09:46:21.915731601Z: INFO: Consensus layer started 2020-10-17T16:06:42.717900907Z: INFO: Consensus layer started 2020-10-18T16:44:12.223313810Z: INFO: Consensus layer started 2020-10-19T09:43:11.622628250Z: INFO: Consensus layer started 2020-10-19T16:49:47.926726104Z: INFO: Consensus layer started 2020-10-20T07:58:13.377478326Z: INFO: Consensus layer started

xx@xx:~/Documents/concordium-software$ cat /proc/sys/fs/file-max 9223372036854775807 xx@xx:~/Documents/concordium-software$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7781 max locked memory (kbytes, -l) 65536 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7781 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

xx@xx:~/Documents/concordium-software$ file concordium-node-retrieve-logs concordium-node-retrieve-logs: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=6fcd1a4b831b1cc7f0ff7b2c8223b5e69dad50c5, stripped xx@xx:~/Documents/concordium-software$ uname -a Linux burcu 5.4.0-51-generic #56-Ubuntu SMP Mon Oct 5 14:28:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

xx@xx:~/Documents/concordium-software$ dd if=/dev/zero of=./LargeFile bs=1024 count=3000000 3000000+0 records in 3000000+0 records out 3072000000 bytes (3.1 GB, 2.9 GiB) copied, 14.1962 s, 216 MB/s xx@xx:~/Documents/concordium-software$ du -csh ./LargeFile 2.9G ./LargeFile 2.9G total

My system is 64 bit and that binary has compiled 64 bit. so, 2bg file problem should not be 32bit limitation.

ab-concordium commented 3 years ago

The log retrieval tool is not well-optimized and uses excessive memory. There will be an updated tool available next week that will most likely resolve this issue for you as well.

concordium-cl commented 3 years ago

Log issues resolved; for more info, see https://github.com/Concordium/Testnet3-Challenges/blob/main/logs.md