[TASK] add output raw-jsonl

lemoer commented 3 years ago

Description

This output takes the respondd response as sent by the node and includes it in a Line-Delimited JSON (JSONL) document. In this format each line can be interpreted as separate JSON element, which is useful for json streaming. The first line is json object holding the timestamp and version of the file. Then there follows one line for each node, each containing a json object.

An example output looks like this:

{"version":1,"updated_at":"2021-03-27T21:58:48+0100","format":"raw-nodes-jsonl"}
{"firstseen": ..., "lastseen": ..., "online":true, "statistics": {...}, "nodeinfo": {...}, "neighbours":null, "custom_fields":null}
{"firstseen": ..., "lastseen": ..., "online":true, "statistics": {...}, "nodeinfo": {...}, "neighbours":null, "custom_fields":null}
{"firstseen": ..., "lastseen": ..., "online":true, "statistics": {...}, "nodeinfo": {...}, "neighbours":null, "custom_fields":null}
{"firstseen": ..., "lastseen": ..., "online":true, "statistics": {...}, "nodeinfo": {...}, "neighbours":null, "custom_fields":null}
...

Signed-off-by: Leonardo Mörlein git@irrelefant.net

Motivation and Context

This is useful for json streaming.

Checklist:

[x] My code follows the code style of this project.
[x] I have added also tests for my new code.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.

genofire commented 3 years ago

I am a little bit confused about this PR - sorry, for the next question:

why we should use JSONL if there is already some "good" normal JSON outputs? what you like to do with this files afterwards?
if you like to handle streams / liveupdates - would there be a JSONL-database (into file or TCP or ...) more useful?

the PR looks really good:

you have to fix the tests
a comment above the RawNode-struct: Node -> RawNode

so i am willing to merge it

lemoer commented 3 years ago

Regarding your questions:

Streaming is meant in a sense, that you can process information as you download the file.
- Streaming is not meant in a sense that you will get live updates.
- With JSON, most parsers require that you get the full JSON before you can extract information.
- Therefore there is JSONL (and similar formats).
To explain my usecase a bit more in detail:
- I would like to process the information on gluon devices with comparably low memory.
- If I want to use nodes.json here, I would first have to download it completely, keep it in memory and then parse it. Keeping a nodes.json in memory would require ~3MB of ram.
- If, on the other hand, I use the raw.jsonl file, I can process the information line by line during the download. After a line has been processed, it can be discarded. Therefore I only need a buffer that can hold one line at a time. This only requires ~1-3KB of ram.

Thanks for the feedback. The patches broke during rebase. Things are fixed now. I also found that my last commit (from another PR) had unclean go-fmt. That's why I added a patch for it here first, so that the tests pass.

FreifunkBremen / yanic