Can't pipe "burp -a S" output

[davromaniak]

Hi.

I'm trying to send a daily status email using the output from "burp -a S" but I can't seem to be able to "pipe" it.

Here's the output of "burp -c /etc/burp/burp.conf -a S"

2020-09-01 16:33:31 +0000: burp[29811] cname from cert: deci

 burp status                                          2020-09-01 16:33:33 +0000

   atto                    idle  last backup: 0000245 2020-09-01 04:16:53 +0000
   centi                   idle  last backup: 0000025 2020-09-01 04:10:51 +0000
   deca                    idle  last backup: 0000357 2020-09-01 04:04:27 +0000
   deci                    idle  last backup: 0000366 2020-09-01 04:21:53 +0000
   micro                   idle  last backup: 0000025 2020-09-01 04:02:03 +0000
   milli                   idle  last backup: 0000025 2020-09-01 04:07:17 +0000
   nano                    idle  last backup: 0000363 2020-09-01 04:22:44 +0000
   peta                    idle  last backup: 0000084 2020-09-01 04:09:53 +0000
   pico                    idle  last backup: 0000365 2020-09-01 04:17:00 +0000
   yocto                   idle  last backup: 0000032 2020-09-01 04:09:06 +0000
   yotta                   idle  last backup: 0000073 2020-09-01 04:08:37 +0000
   zepto                   idle  last backup: 0000005 2020-09-01 04:05:56 +0000

When I try to pipe the output to any other tool (even cat), there's nothing, like :

root@deci:~# burp -c /etc/burp/burp.conf -a S | wc -l
0
root@deci:~# burp -c /etc/burp/burp.conf -a S | cat  
root@deci:~# burp -c /etc/burp/burp.conf -a S | grep 'status'
root@deci:~# burp -c /etc/burp/burp.conf -a S | mail -s "Burp status" -a "From: burp@xxx.xxx" xxx@xxx.xxx
Null message body; hope that's ok

I discovered this issue when using the summary_script as it sends me an empty email.

I'm using burp 2.2.18 on Debian Buster (using Ziirish package repository).

Don't hesitate to ask me for any information needed.

Thanks.

[ziirish]

hm, I haven't been able to reproduce the issue yet using the "same" setup:

$ burp -V                          
burp-2.2.18
$ burp -c /etc/burp/burp.conf -a S | cat

 burp status                                          2020-09-01 19:10:56 +0200

   adn                     idle  last backup: 0000448 2020-05-25 00:07:01 +0200
   zeal                    idle  last backup: 0000540 2020-09-01 01:00:02 +0200
   zeal-mail               idle  last backup: 0026389 2020-09-01 19:00:02 +0200
   zeal-offsite            idle  last backup: 0000000 never

Can you check that your burp.conf file contains stdout = 1 (although I tried setting this to 0 without being able to reproduce the problem either)

[davromaniak]

Hi.

I have stdout = 1 in burp.conf file.

Here's the whole burp.conf file I use (the same for barely all clients) :

mode = client
port = 4971
status_port = 4972
server = localhost
password = xxx
cname_fqdn = 0
pidfile = /var/run/burp.client.pid
syslog = 0
stdout = 1
progress_counter = 1
server_can_restore = 0
cross_filesystem=/home
cross_all_filesystems=0
ca_burp_ca = /usr/sbin/burp_ca
ca_csr_dir = /etc/burp/CA-client
ssl_cert_ca = /etc/burp/ssl_cert_ca.pem
ssl_cert = /etc/burp/ssl_cert-client.pem
ssl_key = /etc/burp/ssl_cert-client.key
ssl_peer_cn = burpserver
include = /etc
include = /home
include = /root
include = /var
exclude_fs = sysfs
exclude_fs = tmpfs
nobackup = .nobackup
exclude_comp=bz2
exclude_comp=gz

There's nothing extravagant in it.

Thanks.

[ziirish]

Is burp the only tool affected by this situation?

Can you try some dummy things like: ls | cat or find | cat? You may also try running burp through stdbuf in order to make sure this isn't something related to the terminal mode or something that may affect the way stdout would be buffered.

Example: stdbuf -oL -- burp -c /etc/burp/burp.conf -a S | cat (this will set stdout buffering mode to line which is supposed to be the default in a terminal)

[davromaniak]

So no issues with other pipe commands.

I tried the stdbuf -oL --... and it works.

It's running in a LXC container (using Proxmox) maybe the buffering mode is already setup differently in it, I don't know.

Thanks.

[ziirish]

I found this stating that there might indeed be some buffering on pty's passed to LXC's containers.

I'll check if we lack a fflush(stdout) at some point in order to flush the buffer prior exiting.

@grke any thought about this?

[grke]

Hello,

How can I reproduce the issue? I have never used lxc/proxmox before.

[davromaniak]

Hi.

In case it's needed, I can spawn a clone of my burp server instance and give you SSH access to it.

BTW, one of the clients is a Raspberry Pi 4 running Raspberry Pi OS (previously called Raspbian, based on Debian Buster) with burp 2.1.32 and I have the same issue as on LXC and the stdbuf magic also fixes the issue.

Thanks.

[ziirish]

I'll try to find a moment during the weekend to find a way to reproduce the issue but if I fail, then I'd be interested by your offer to provide us a container to connect to.

Of course if @grke prefers to have a look as he knows best the codebase then I'll leave it here.

My feeling is we just need to force flush stdout at some point prior returning from the -a S command. Since all the io operations are wrapped with asfd, I'd say burp likes buffered FD's. That would explain why some environments that don't enforce the buffering mode of stdout show the problem.

[grke]

Hello,

Burp uses fileno() to get stdout's file descriptor, then does the writes direct to the file descriptor. I think this means that fflush(stdout) would have no effect, as fflush operates on the buffers of the FILE * that wraps the actual file descriptor.

Maybe the containerisation somehow means that it needs an fsync instead?

Maybe try 'fsync(fd);' before the 'close(fd);' in src/fsops.c:close_fd()?

I will look to see if I can install lxc and try to reproduce.

[ziirish]

Thanks for the insights @grke I tried playing with the stdout buffering mode without being able to reproduce the issue yet.

I guess having access to your environment would speed things up if that's not a big deal for you @davromaniak I can send you a public SSH key and a from IP address on IRC if that's ok with you?

[davromaniak]

I can spin up a clone of the my burp server instance, send me your SSH key on IRC and I'll do it over the next few days.

Thanks.

[ziirish]

Hello,

I've been toying with fsync and fflush without any success for now. I don't know how to reproduce this either out of the container.

[wejdross]

some_command.sh |& wc -l this should solve Your issue with burp or any other software that reports to stderr anyway, on burp-2.5.4 pipes work just fine :)