sosy-lab / benchexec

BenchExec: A Framework for Reliable Benchmarking and Resource Measurement
Apache License 2.0
228 stars 198 forks source link

Improvements to Documentation #999

Open incaseoftrouble opened 8 months ago

incaseoftrouble commented 8 months ago

I want to provide some improvement suggestions to the documentation after understanding more about benchexec.

In particular, add a clearer overview of what which tool is doing, how it can be used, etc. in README and a concise user guide.

This issue is mainly to track that I want to do this and don't forget it :)

PhilippWendler commented 5 months ago

For the record: Some improvements clarifying that runexec is usable standalone were made in the linked commit (resulting from discussion in #1031).

Current plan for further improvement is a quickstart tutorial that reviewers can link for "how to do better benchmarking easily" as discussed in https://github.com/sosy-lab/benchexec/pull/1031#issuecomment-2128663277 and the following comments.

PhilippWendler commented 4 months ago

Is there anything left that was not covered by #1044?

incaseoftrouble commented 4 months ago

I will double check the following:

incaseoftrouble commented 4 months ago

Making all links relative should definitely be done, but is trivial.

Content wise, the following would still be available:

1) Structuring readme

2) Moving the python API stuff to a separate file (and later on maybe add e.g. an example of how this can be integrated)

3) Restructuring benchexec

(modulo the formatting changes)

PhilippWendler commented 4 months ago

I unfortunately don't have a lot of time to pursue this further, and from the linked diff it is hard to see the exact changes and their benefits. So I won't actively work on this. But if you think this is worthwhile and would like to submit a PR, I will of course review it.

Please do keep our current style of formatting text, though, I think there is still a misunderstanding. We do not do hard line wraps at x characters per line. We format text more like code, just without indentation: One sentence (statement) per line, and if the sentence (statement) is too long for that because it is longer than ca. 72-80 characters, it is split across several lines according to its internal structure (AST), i.e., with line breaks between subclauses (subexpressions) such that the most closely related parts of a sentence (statement) stay together on a line (line breaks as far towards the root of the AST as possible). When rewriting a part of a long sentence, do not change the line wrappings of the complete sentence.

This has the effect of being both readable and minimizing diffs. We have very good experience with this for example also when writing papers.

incaseoftrouble commented 4 months ago

I unfortunately don't have a lot of time to pursue this further, and from the linked diff it is hard to see the exact changes and their benefits. So I won't actively work on this. But if you think this is worthwhile and would like to submit a PR, I will of course review it.

Yes, the question is mainly which of these points you consider interesting / relevant. I would then (eventually) write the points we settle on.

Please do keep our current style of formatting text, though, I think there is still a misunderstanding. We do not do hard line wraps at x characters per line.

Yes, its an old diff. I still don't fully understand the AST rule but that's for later.

PhilippWendler commented 4 months ago

I unfortunately don't have a lot of time to pursue this further, and from the linked diff it is hard to see the exact changes and their benefits. So I won't actively work on this. But if you think this is worthwhile and would like to submit a PR, I will of course review it.

Yes, the question is mainly which of these points you consider interesting / relevant. I would then (eventually) write the points we settle on.

In principle I am not opposed to any of them, but in the current state I also don't really see the benefit.

Separating the Python API becomes nice if it grows. With the current size of the file I guess we can just leave it.