Logging need to be moved in a dedicated DB table, and only if enabled generate logfile

[vecna]

• Logs need to be collected in DB,
• Every logged event need to be part of one or more tag (usable by Admin to select event to checks)
• Need a dedicated /admin/log API
• Need a dedicated Admin panel
• Log policy need to be shared via public /node API

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[vecna]

This is part of the OTF security enhancement, this is my estimation on how to approach this:

• write a "Log" class.
• Log in memory,
• once a while, flush in database
• do a REST API for admin and for receiver,
  • the REST API check both in memory and in DB
• every log entry has specified:
  • a verbosity level
  • info: what is happening in high level
  • error: what is failed
  • debug: what is happening in low level
  • security: bruteforce, anomaly threshold (goes also in a file, other ticket)
• an owner (a receiver.id, internaltip.id or 'admin'), a log entry may have multiple owner, 'admin' is not implicit. operation inside internaltip are not display to admin.

I guess in two-three days I can change all the "log.*" call properly and provide the rest API, not the GLClient side able to filter by tag and stuff.

/cc @evilaliv3

[evilaliv3]

ok in general but also for this i think we are getting too deep into the code before a proper specification. For what concern logging and debugging we have till now done it quite randomly putting debug output here and there.

some comments for the specification related things you highlighted:

for what concern the owner you have written please use the user.id for both admin and recevier. this way we are already ready to have multiple admin. this should be kept in mind cause other fact that apart from the missing change admin name feature or add admin feature we already have in the DB the possibility to have multiple admins. you have rightly thought that the log should have a TAG assigned but it was not specified above. we should evaluate how to implement them so that the same log encrypt can have multiple tag assigned (this is doable with references and in the future should be studied also for context object and receivers object so that probably a TAG table should be needed with a format like (id, ref_object_type, label_internationalized)

p.s. the example of TAG fastly written here is an example to show you my need to go for a specification so that a feature is not some code added here and there but something that should be studied and specified in order to keep in mind the full system.

[fpietrosanti]

In any case, from implementation perspective please see which kind of logging framework do exists to do that kind of job, without implementing it from scratch. In Java there's log4j that can be configured/customized and used for that purpose (that include log-rotation, log-archival, log-cleanup that must be part of anything that "increase it's size during time" like logs).

It's likely that Python and/or Twisted have something similar.

[evilaliv3]

yep thats for sure fabio but it's more for the final output for the system to be outputted on the filesystem.

i did it in past for tor2web by using https://twistedmatrix.com/documents/current/api/twisted.python.logfile.DailyLogFile.html more than on log rotation we should study also what are the standard formats for the serialization (or let's say the most common one on a unix system)

[fpietrosanti]

@evilaliv3 logging to db is much more sensibile/risky than logging to filesystem and require more care, including rotation, archival/automatic-deleting, visualizzati in, export and usually there are logging framework to do that

[vecna]

I can't take a look existing framework and figure out how integrate them in globaleaks design. not now at least, not in two days. I step out of the issue.

[vecna]

the default Logger class ( https://docs.python.org/2/library/logging.html ) supports rotation: http://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/

[fpietrosanti]

@vecna using Twisted it's maybe best, for file based logging, to use Twisted stuff https://twistedmatrix.com/documents/current/api/twisted.python.logfile.DailyLogFile.html like in tor2web? (but i don't know what's the specific topic)

[evilaliv3]

yep i confirm that using the DailyLogFile is "tha way" and i would like to provide some feedback on this topic as i managed to do that on Tor2web.

[fpietrosanti]

@NSkelsey @evilaliv3 shall we close this ticket considering too old / being replaced by the database logging / auditing framework to be done for the OpenWhistleblowing things?

[NSkelsey]

Sure, this means keeping OW and GL aligned with changes ported from one into the other and vice versa.

[evilaliv3]

I do not agree with this.

The log we discussed is the tip log and deals with building a datacentric where the center is the tip-id.

This ticket istead deals with moving the whole logfile used for debugging and errors etc to the db.

I may agree that we can close it anyhow if we agree that we prefer to keep on the db only relevant log and keep to the file the debugging log given the fact that files can be archived and gzipped easily and not impact in the database that will instead grow indefinitely

[vecna]

I gave a look exactly yesterday night on the status of log-refactor... I would not have time to complete it, but I can align it with the current devel and ... i don't know. in theory has also another OTF label on it.

[evilaliv3]

Closing in favour of https://github.com/globaleaks/GlobaLeaks/issues/2579