Huge log files even when do nothing other then have the server up

[MarcioMonego]

Hello, First of all congrats of the huge efforts on making Craftbeerpi live!

I have used cbpi 3 for a long time and last week I spent some time updating Raspberry and Craftbeerpi to the last version. Cbpi3 also have the issue of keeping reading and writing to log files continuously until you cannot read charts or compare two or more mash runs because the garbage written. Then you have to delete all logs manually and loss last run information or brab part of files.

With Cbpi 3 I made some local changes in Onewire implementation to stop it keeping writing logs when we were not using the entire system. I have 3 kettles and a flow sensor, plus one fermenter.

The thing I made when reading sensor data and deciding to write or not to log file was care if, and only write the log of the sensor if:

A mash was in progress, then all sensors that belong to a kettle can write, sensors belonging to fermenters NOT A sensor belongs to a kettle that is in auto mode OR that the actor is ON A fermenter profile is in progress and the sensor belongs to the fermenter in progress If only a fermentation profile is running and none of kettles is with heater on or in auto mode, none of kettle sensors should write data to logs.

With this simple change I get rid of very large amount of data being stored in log files

If I care about the graphics results of 2 or more mash profile runs separated by medium/long periods of time I can, because the time lapse were not write a lot of garbage values to log files.

When I made this to Cbpi3(2 years +-) I found some problem that I guess to be related to Kettle still writing to log and I don't go further as the behavior was on some non extensible class of cbpi3.

Hope you can take a look on this and feel free ask something else about this.

[avollkopf]

Currently, no automated feature is planned for this. You can either disable csv logging for all sensors or leave it on for all sensors. No differentiation as some sensors could be used for different kettles / fermenters.

I recommend anyhow to use influxdb logging and disable csv logging at all.

[MarcioMonego]

Very disappointed with this, @avollkopf your recommendation to use influxdb will get us rid of huge garbage data? I know it will not. We will just still to inflate the database as with csv files, get garbage data independent if it is useful for our system. Thank you for the recommendation, it enforces the community effort to get better software and solutions. Congrat for close the issue without any effort to get engagement to build a better solution, like I already received from Manuel time ago that let me to just make solutions to myself and not for a community.

[lalo-uy]

This is not perfect but keep my CBPi3 logs steady for 4 years. Move all lines older than 3 days ( 72 h ) to a new folder.

Run by cron every night over each log file.

!/usr/bin/python

import sys import os import shutil import time tem_arch="/run/az" if os.getuid() > 0 : tem_arch="/run/user/1000/az"

limit = time.strftime("%Y-%m-%d",time.localtime(time.time()-72*3600)) origin=sys.argv[1] with open(origin,"r") as org: line=org.readline() with open("./respaldos/"+origin,"a") as dst: while limit > line[0:10] :

print line[0:10]

         dst.write(line);
         line=org.readline()

   with open(tem_arch,"w") as dst:
      shutil.copyfileobj(org,dst)

shutil.move(tem_arch,origin)

El El sáb, 7 de ene. de 2023 a la(s) 20:36, MarcioMonego < @.***> escribió:

Very disappointed with this, @avollkopf https://github.com/avollkopf your recommendation to use influxdb will get us rid of huge garbage data? I know it will not. We will just still to inflate the database as with csv files, get garbage data independent if it is useful for our system. Thank you for the recommendation, it enforces the community effort to get better software and solutions. Congrat for close the issue without any effort to get engagement to build a better solution, like I already received from Manuel time ago that let me to just make solutions to myself and not for a community.

— Reply to this email directly, view it on GitHub https://github.com/avollkopf/craftbeerpi4/issues/72#issuecomment-1374654091, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXBW4OXL7AVB57KHR3QAQDWRH4XZANCNFSM6AAAAAARBTC4OA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[avollkopf]

@MarcioMonego I don't know you, but thank you so much for your valuable feedback. That gives me some more insight into the community you are talking about. Typically, a community lives from contributions of each individual of the community. I have spent quite some amount of my free-time to get cbpi4 up and running as the initial version from Manuel was not production ready. My free-time was limited and is limited even more since January 1st. Cbpi4 is now working reliably and has quite some new features. It is now open source and it is free. I am currently focussing on a long-term stability which is the reason, why I am working on the compatibility with Python 3.10 and newer python packages on the server side and react 18 on the UI side which takes quite some time.

Everybody can contribute to the system, and many users do this. Lately, I have not seen any contribution from your side besides this issue and your answer (I 've learned quite some time ago, that I never should respond to something/somebody, when I am pissed... Just an idea).

Anyhow, as mentioned, it is an open-source community and everybody is invited to contribute something and the software is free of charge. And everybody is free to use the software or not.

To sum it up: Take it (use it as it is), Change it (contribute) or leave it (as you have written in your answer).