Put repo and run metrics in log messages

HonkingGoose commented 11 months ago

Describe the proposed change(s).

Have a simple log at the end of each repository run that shows:

per run metrics (e.g. commits and PRs created/updated/merged/deleted/etc)
per repo metrics (PRs open, awaiting, merged, etc.)

Describe why we need/want these change(s).

Putting the metrics in the log makes it easier for Renovate users to create their own "Renovate Status Dashboard", like @onigoetz did for Swissquote. [^1]

There's also an issue asking for Prometheus support:

17661

Relevant discussions:

23105
23953
17456

[^1]: Renovate docs, Swissquote

lmilbaum commented 11 months ago

Is it possible to dump this info in a separate log file?

rarkins commented 11 months ago

Yes, using LOG_FILE (it dumps all logs).

I don't wish to add additional ways to output files as then we need to maintain them all

HonkingGoose commented 11 months ago

Yes, using LOG_FILE (it dumps all logs).

That should go in the docs somewhere then. I'm thinking of making an issue to track "Explain how to create your own Renovate metrics dashboard" with things like this:

Who can create a Dashboard (probably only self-hosted)
Using LOG_FILE to grab logs
Link to Swissquote example
Copy/paste relevant parts of @onigoetz's explanation from https://github.com/renovatebot/renovate/discussions/23105#discussioncomment-6366621 into the new docs (I'll ask for their permission first, of course)
Whatever else the maintainers think is important to mention

reitzig commented 11 months ago

I am in the process of building a custom "dashboard" (step 0: data in JSON format) on top of the log file as well (stuck to the Atlassian suite, yay). I haven't been able to find any specification of the several data formats and values, so it's been ... slow.

To mention just one example: how many different result values in branchesInformation are there, what do they mean, and how do they relate to the numbers in Renovate repository PR statistics? Or: What are all the ways in which repo errors can be represented? (Looking at you, skipReason!) Which repo errors are promoted to run errors, and why?

Today, I found myself wishing I could just get the data used to build the Dependency Dashboard issues dumped as JSON to file instead of Markdown to API. That said, a single stats message per repo might already be an improvement. Pulling them into single files is an easy post-processing task, after all.

Anyway, if "use the log file" is indeed the canonical answer, I'd ask for documentation on the stats message formats and some form of API stability assurances for them.

One thing to note: Dumping huge blobs of data in the JSONL log -- most notably, the HTTP stats messages -- can skrew up ingest to tools like OpenSearch, among other issues. In our case, the ops people told me they couldn't get fluentd to properly import Renovate (debug) logs; two things they mentioned were that individual lines are too long (no limit they tried in fluentd worked), and different messages had different top-level fields (which seems to mess with OpenSearch message processing). I'm not an expert in these tools, so take that with a grain of salt.

HonkingGoose commented 11 months ago

No specification of data formats and values

I haven't been able to find any specification of the several data formats and values, so it's been ... slow.

@reitzig did you see the Renovate docs, Troubleshooting Renovate page? It explains the log debug levels, and how to get JSON output when self-hosting Renovate.

Can you tell me what keywords you used to search? Or how you clicked around in the docs? Maybe I can improve the docs in some way, so others are more likely to find this information. 😄

Other relevant issue

You'll probably also want to subscribe to this issue:

24156

About the rest of your comment

I'll let the maintainers respond to the rest of your comment. 😉

reitzig commented 11 months ago

I am in the process of building a custom "dashboard"

FWIW, I hope I'll be allowed to open-source that work. It's for a client, so they get to decide. 🤞

@reitzig did you see the Renovate docs, Troubleshooting Renovate page? It explains the log debug levels, and how to get JSON output when self-hosting Renovate.

I did, thanks. Getting the JSONL log has not been my problem. My only confusion there was whether LOG_FORMAT and LOG_LEVEL change stdout and/or file log, but that was easily tested.

You'll probably also want to subscribe to #24156

I certainly did, thanks!

renovatebot / renovate