Flush log file each minute

[rene0]

Once #4 is done, it should be possible to flush the log file each minute because the sample rate is independent from the CPU load.

[JsBergbau]

Anything new about it? Its a real problem for me that the logfile isn't written while the program is running. So I can't use the received data to work with in another program.

I've also added in line 726 setbuf(logfile,NULL); in input.c but has no effect. I'm not so deep into programming, can you explain why?

[JsBergbau]

Correction: Now at the first start after a reboot you can see "--new log--" in the logfile right after start. Program exited with STRG+C, logfile deleted and dcf77pi re-run there is no "--new log--" in the file anymore. This seems really strange for me.

Very curious to hear your explanation.

[rene0]

"So I can't use the received data to work with in another program." <-- I think you want to use the library (libdcf77.so) instead, dcf77pi is just a front-end. For analyzing existing logs, there is already dcf77pi-analyze.

Using setbuf(logfile, NULL) would cause each character to be flushed immediately, putting stress in SD-like cards (typically in a Raspberry Pi) or high NFS load. setbuf() has an option to flush the buffer per line, which is what we want here. (Normally a block of size BUFSIZ is allocated so the file is only updated about once every 30 minutes - 2 hours). The log file was flushed every minute by dcf77pi for a while but it turned out that this caused too much delay, confusing the reception of bit 0 (see commit b1af9e481bb354a899f4a7fe31675595c24033be). That bit is used to determine the length of a 0 pulse.

With STRG+C you mean Ctrl+C? You can just press Q to exit dcf77pi as shown, it will save the log file. If you change the log file when running by pressing L the old log file will be saved before the new one is opened. dcf77pi opens existing log files in append mode, so no existing content is ever deleted.

[JsBergbau]

<-- I think you want to use the library (libdcf77.so) instead I'll take a look. A simple text file from where I can read in is probably the easiest for me (I'm very new to Linux and mostly firm with bash scripts, but not with C-programs or shared librarys).

Using setbuf(logfile, NULL) would cause each character to be flushed immediately, putting stress in SD-like cards (typically in a Raspberry Pi) or high NFS load. Yeah I know, thats not good for the SD-Card. But strange thing is, even with this line in Code, it doesn't output immediately. I can't understand this why. And even this strange behaviour that it is writing immediately after the first start after boot "--new log--", but nothing more. The other lines appear very delayed, but also ok for me, as long as the appear within 1-2 hours.

The log file was flushed every minute by dcf77pi for a while but it turned out that this caused too much delay, confusing the reception of bit 0 (see commit b1af9e4). That bit is used to determine the length of a 0 pulse. Very resonable, because it blocks the program. What do you think about doing the writing in another thread, like this:

1. open the log create --new log --, close it
2. call "write_log(char *)" (or simmiliar), passing one line of received chars to write out 3.create a new thread in that function, that does the writing 3.1 open log file in append mode 3.2 write out the received line 3.3 close the file
3. now return to the caller. There is almost no delay in the main program, so minute bit will be detected correctly.

With STRG+C you mean Ctrl+C? You can just press Q to exit dcf77pi as shown, it will save the log file. Yeah, sorry you're right, I mean Ctrl+C. Yeah you're right with pressing Q and I want to controll the Programm from a script so I can't send Q key (or is there a solution? Btw: I've noticed sending "q" doesn't work, only "Q". Is there a special reason for case sensitivity?

[rene0]

you're right with pressing Q and I want to controll the Programm from a script so I can't send Q key (or is there a solution?

Perhaps you can try the readpin program (installed in the same place as dcf77pi)? dcf77pi is really designed to be a TUI program, not a UNIX pipe.

Btw: I've noticed sending "q" doesn't work, only "Q". Is there a special reason for case sensitivity?

Yes, to make it harder to accidentally exit the program (you have to press two keys simultaneously now, or use Caps Lock).

[rene0]

Very resonable, because it blocks the program. What do you think about doing the writing in another thread, like this:

That's indeed what I experienced. Flushing the log file each minute at the minute should be easier with #4 finished (I believe). For now I can also flush it at second 2 of each minute, given that bits 1-14 are only for licensed Meteotime users and therefore not very interesting (I asked them if I could add a Meteotime decoder to the program/library by using their decoder chip but they refused).

[JsBergbau]

Perhaps you can try the readpin program

After start, it outputs every second, but after a few seconds, it seems to hang and skips a lot of seconds. Don't know why.

For now I can also flush it at second 2 of each minute, given that bits 1-14 are only for licensed Meteotime users and therefore not very interesting

Please do not, I need these bits, because I want to decode the weather data.

(I asked them if I could add a Meteotime decoder to the program/library by using their decoder chip but they refused).

Perhaps they misunderstood you and thought your program will decode the data without chip. There is already a program that decodes the weather data with a connected chip. See https://dcf77logs.de/Software.aspx?id=000003 And even the decoded data is provided free for everyone. https://dcf77logs.de/live

[rene0]

And even the decoded data is provided free for everyone. https://dcf77logs.de/live

That was explicitly forbidden when I was on the phone with them, so dcf77logs is technically illegal.

[JsBergbau]

Perhaps the staff had a bad day, because when you have to buy their chip to decode, then more people would do and they get money. There are even ways to decode it without their chip at all. Its similiar to buying a weatherstation with their technology and streaming that data to the internet. No one can forbid that.

[JsBergbau]

Problem with changed code was not working is now solved, due to different install versions I was using different versions of dcf77pi which had not the changes, I've made. Sorry for that.

However even with flushing logfile after each character, dcf77pi has no problems in detecting the correct time and bits.

[rene0]

Doesn't that increase the load? You could also reapply the above commit to flush the log file each minute.

Regarding Meteotime, more people buying their chip seems plausible but back in 2014 they answered:

As discussed by phone yesterday, the Meteotime-service is something totally different from DCF77. It is a privately financed, not public and not free of charge, offered data service. The Meteotime-service is provided by HKW and it just uses the DCF77-langwave signal as carrier for the meteorological information. The use of Meteotime data is only allowed with reference to a valid license contract between customer and HKW. HKW has to verify, whether a license contract with the applying company can be signed or not. HKW has to release / adjust the license contract in accordance to the nature of application.

[JsBergbau]

Regarding Meteotime, more people buying their chip seems plausible but back in 2014 they answered:

Strange answer. Everyone can buy the HKW581 Chip and use the data http://arduino-projects4u.com/product/meteotime-hkw581/ But ok, they're Germans, they're sometimes very strange.

Doesn't that increase the load?

The CPU load is between 0-1 % read from a RDP connections, so this overhead is also there.

I didn' know how to revert back to the commit with flushing the logfile each minute (and perhaps that was the problem, flushing each minute is too much data at the same time, whereas flushing every char makes no problems.)

However I've created a safe way to flush the data every minute within a thread (so the main program can't block and miss a signal). Feel free to include that code into your project. Would be great if you can make the interval configurable with the json file, but it's also ok like the way now.

Everything is done in input.c (and input.h to declare function)

include

int append_logfile(const char * const logfilename) { logfile = fopen(logfilename, "a"); if (logfile == NULL) return errno; fprintf(logfile, "\n--new log--\n\n"); pthread_t flush_thread; pthread_create(&flush_thread,NULL,flush_logfile,NULL); return 0; }

`void *flush_logfile() { for(;;) { fflush(logfile); sleep(60); }

}`

[rene0]

Using a thread might be interesting too. I might have meant "flush like file at each minute marker", which might be different from each minute in case of reception errors

I did have an email conversation with the owner of ArduinoProjects4U, I'll ask how he thinks about decoding Meteotime.

[JsBergbau]

Tested the changed code now for 2 days (dcf77pi is running since then). It works perfectly. Not a single Bit was lost in that time. Before the log was flushed unpredictably and perhaps thats why I had some bit errors from time to time like 00001011001110000100101101100000001110100__11111000000110010a60821c2.0290 In the attachment are the changed sources.

input.zip

[rene0]

Before the log was flushed unpredictably and perhaps thats why I had some bit errors from time to time like 00001011001110000100101101100000001110100__11111000000110010a60821c2.0290

It could be because of flushing, but also because of bad radio reception. dcf77pi writes a dash to the log file (and colors the corresponding bit yellow in the live view) if the pulse is too long (more than 300ms). dcf77pi-analyze will substitute the value of the corresponding bit in the previous minute in those cases.

[rene0]

I finally sent the email today, let's see what happens.

[rene0]

According to him interpreting the plaintext is legal when using the HKW 581.

[rene0]

Code for flushing the log file committed, perhaps revisit this once #4 is done