Library Detail Backend Changes

[frankwiles]

• [x] Gather Github stats necessary for new graphs
• [x] Render library.adoc in the root of the repository to HTML and save it into the database in a TextField which will be rendered below the graphs and other header/link elements.
• [ ] Periodically pull and re-render these in a cron. Probably more frequently than we ware pulling maintainers and authors

[williln]

Related issues:

• 212

• 178

@frankwiles what should the refresh schedule be for the various library-adjacent objects? I can start breaking those out into smaller issues.

objects:

Casually broken out into separate API calls, and we'll just refresh whatever data is there when we grab it.

• Version repo
• Specific Version tags
• Library repo
• Specific Boost version tags of Libraries (LibraryVersions)
• Pull requests
• Issues
• Authors of Libraries
• Maintainers of LibraryVersions

[williln]

notes from post-meeting sync with Greg:

Fields to make available in template:

• [x] Name: library.name
• [x] documentation URL: ~Link to the Antora documentation. No https://, only the remaining URL. Not sure yet how we get this.~ Forced a link that is "good enough" but not valid for all libraries. will need to fix.
• [x] URL to GitHub issue for that library -- pretty sure we have this or can format it
• [x] URL to GitHub repo for that library -- we have this but I need to confirm the field name
• [x] First date of the library -- according to GitHub? Not in libraries.json.
• [x] C++ version -- we have this, and it's in the libraries.json as cxxstd. I need to confirm what we call it but I think it's the same
• [x] Description -- library.description
• [x] Categories: library.categories
• [ ] Commits Per Month graph -- see below
• [ ] Commits Per Year graph -- see below
• [x] Profile photos -- use user.image for authors and maintainers that have profile photos. Otherwise use a static image.
• [x] Introduction -- this is probably the asciidoc data but need to confirm
• [x] Make the boost versions available so they can be used in a dropdown
• [x] Make sure that filter can return a LibraryVersion details object

commits per month

• [ ] commits to main?
• [ ] Should we collect full historical data, or only for the last year, or some other time frame?
• [ ] how current do we want the data? (Example: On March, we want to see that we have 7 commits so far for March and not have to wait until the end of the month, right?)
• [ ] Should we collect commit metadata from GitHub like we do the PRs and issues, or get the count in a simpler way?

commits per year

• [ ] how many years back do we need to go to collect data?
• [ ] how many years do we want to show on the page?

new design

See #254

[frankwiles]

@williln I'm thinking daily is probably the right cadence. Just wanted to make sure it wasn't weekly or monthly.

[vinniefalco]

@sdarwin already has this data by the way. I settled on the number of commits because it is super easy to get

git rev-list --count <branch-name>

and also because it is efficient.

In terms of tracking, once a month is more than enough, there's no need for dailies or partially filled months... the signal to noise ratio at high sampling rates is too low. There should always be 12 months in the monthly graph for libraries that are at least 12 months old. Otherwise those older months would just show as zero.

For the yearly, only full years count, going back since the library was added.

[vinniefalco]

The graphs are the lowest priority, if we just hide them and put a solid tinted rectangle in their place I am happy for the Debut, and we can make them actually work later. Everything else needs to work though.

[williln]

@vinniefalco For the C++ version display, should we display exactly what is in meta/libraries.json or should we use the display names here? https://docs.cppalliance.org/user-guide/prev/library_metadata.html

For the display year, should we use the year of the earliest GitHub tag? Example: for asio it would be boost-1.35.0-rc1 in 2008: https://github.com/boostorg/asio/tags?after=boost-1.35.0-rc3

[vinniefalco]

In the lozenge prefer "03" and "14" to "C++03" and "C++14" if for no other reason than the repetition of "C++" weakens the design language of the page.

If we are talking about the library detail page then I think it would be ok to say "C++11", "C++14", and so on.

[williln]

@vinniefalco For the "Since 2019" display year on the library details page, should we use the year of the earliest GitHub tag? Example: for asio it would be 2008 in the boost-1.35.0-rc1 tag

[vinniefalco]

earliest repository tag is a good start. a library might be older (for example in the SVN days), but the author would have to submit a request to manually update the database or something.

[williln]

Items Accomplished:

• [x] Various small data points
• [x] Author/maintainer info
• [x] Loaded all the versions of each library, along with their GitHub data and author/maintainer data
• [x] Loaded the date of the first GitHub tag for all libraries
• [x] Made the details page version-specific

Remaining items:

• [x] Render the asciidoc in the body of the details page -- @vinniefalco where do I get that content?
• [x] Get number of commits per month going back to when the library was added and make that available to the template
• [x] Make the links version-sensitive when possible (links to docs, links to github, etc.)

[vinniefalco]

Render the asciidoc in the body of the details page -- @vinniefalco where do I get that content?

I was thinking of having it come out of a directory in the library repository. for example

doc/library-detail.adoc

or if that file is not present, then it could come from the root:

README.adoc

alternatively, we can just show nothing if "doc/library-detail.adoc" is missing, and if it is present and is also a symbolic link to another file in the repo then we can resolve the link and use that file. This will allow the repository owner to create README.adoc in the root of the repository for GitHub to preview on the repository landing page, and also allow the repository owner to link doc/library-detail.adoc to README.md so that the same content appears on the corresponding library detail page.

This would have to use the right commit or branch. If the user has selected an older version then it needs to use the git tag corresponding to that version. The user can select "latest version" which is just the highest semantically valued tag. But there are also the "develop" and "master" choices which correspond to the library repository's commit-id in the superproject.

[alandefreitas]

Render library.adoc in the root of the repository to HTML and save it into the database in a TextField which will be rendered below the graphs and other header/link elements.

where do I get that content?

To complement Vinnie's answer, the pages are in the revsys buckets under s3://***/site-pages/{branch}. For instance, s3://***/site-pages/develop/help.adoc for the FAQ page. The "?" icon currently links to https://www.boost.revsys.dev/support/ though.

[williln]

@vinniefalco @alandefreitas Thanks for your explanations. So for the library documentation --

Render the asciidoc in the body of the details page -- @vinniefalco where do I get that content?

I was thinking of having it come out of a directory in the library repository. for example

Do the asciidoc files for the library descriptions not exist in the library repos yet? Or if they do, can you point me to the one I should be using for asio, just as an example? https://github.com/boostorg/asio

[vinniefalco]

The files do not exist yet :) Alan could you please copy README.md to library-detail.adoc (and convert to Asciidoc) for URL, JSON, and Beast? And ask the Boost.Redis and Boost.MySQL authors do to the same (and merge to develop)?

[alandefreitas]

@williln library-detail.adoc is in boostorg/json/doc and boostorg/url/doc. I'll still do the other libs.

[williln]

@alandefreitas Thank you! I'll use those as test cases when working on the asciidoc conversion.

[williln]

@alandefreitas @vinniefalco The code to convert and cache asciidoc content is now in place, so I'm circling back around to load the doc/library-details.adoc content into the body of the Library Details page for the pages that have it.

For versions of the library that predate when that file was added or that otherwise don't have it, what should we show for the library description?

Local example of a library with that file:

And without that file: