Unintended files in 1.84.0 release archives

[Lastique]

In boost_1_84_0.tar.bz2, there are the following files added in libs, compared to 1.83.0 release:

• _ directory, with a set of nested directories and files with web-related content
• 404.html
• search-index.js

These files seem to be related to some online search engine, and don't seem like they should be included in a Boost release.

Please, remove them.

[Lastique]

@ashtum Perhaps, this is related to the new search facility on the website?

[ashtum]

No, the search facility doesn't add anything to the libraries documentation and it is not part of the release process. The search index is generated by crawling the documentation files in the release archives.

It appears that search-index.js is an artifact generated during the build process of one of the libraries that utilizes Antora for generating the documentation pages.

[Lastique]

Thanks. @vinniefalco, could you look into this? From the contents, it seems to be related to C++ Alliance.

[sdarwin]

one of the libraries that utilizes Antora

@Lastique this is a step towards converting boost documentation to Antora.

[Lastique]

I don't know anything about Antora, but it doesn't seem to me that this conversion should result in leaving a mess in Boost release packages.

If those files are temporary, they should be removed before packaging (and placed under bin.v2 in the build process). If they are legitimate part of the compiled documentation, they should be either in doc or in libs/<library>/doc/html.

[sdarwin]

cc: @alandefreitas

[Lastique]

The _ directory is still present in the 1.85 beta RC archives.

[alandefreitas]

https://github.com/boostorg/release-tools/pull/58 removed the files from the libs directory and the _ directory was left there because it's where the docs are.

Before creating the PR, I discussed what we could do about the _ directory with @pdimov and @sdarwin. I replicated the suggestions that came up in a https://github.com/boostorg/release-tools/pull/58#issuecomment-1885607734, which I'm transcribing here:

Considering the Antora directory layout, the solutions proposed so far were for the _ directory:

• Keep patching Antora into libs and leave _ in libs as part of Antora support. Cons: _ could be considered a library by some script.
• Find another directory for Antora documentation (where?) and copy all Antora docs to this new directory. Libraries using Antora would update their index.html files to redirect there. Pros: when moving to Antora, libraries could temporarily access both documentation folders on boost.org before removing the old scripts. Cons: Links to documentation would change considerably.
• Move _ somewhere else (where?) and include a workaround to patch all files in Antora HTML files replacing ../_ with a relative reference to this new directory. Cons: huge unstable workaround.

This conversation ended up dying because I think people simply weren't interested enough. But we can do this discussion to find the best solution here and update the scripts accordingly.

[sdarwin]

replacing ../_ with a relative reference to this new directory.

This is only an idea, but if it's currently ../ (because perhaps a script was run from ../) , could it be switched to ../../ (by running the script from ../../ , one directory higher up) and then nothing needs to be patched?

[alandefreitas]

This is only an idea, but if it's currently ../ (because perhaps a script was run from ../) , could it be switched to ../../ (by running the script from ../../ , one directory higher up) and then nothing needs to be patched?

I think I made a mistake talking about ../_ because it should be only ../ in href. So I think we ended up not talking about the same thing. The layout Antora generates is something like:

• _
• <library name>
• ...

This is nice because when it's copied into libs all other directories remain the same because they match the library name (except the _ directory which contains common files). The links to the documentation also remain the same.

So the _ directory has files with hrefs to ../<library name>/docs/<etc> and the <library name> directories has files with hrefs to ../_/<etc>.

That third suggestion was to move only _ somewhere else and somehow adjust the hrefs in all of these files. This keeps the advantage of maintaining the links but it's a very unstable solution because it involves parsing html and all other kinds of files that might contain "../" where this is a false negative.

So ../_ is about the hrefs. Not about the location where things currently are. If things go to ../../_ or any other location, the antora directories don't match the directories in libs (which would be the advantage of the first and third suggestion) then I think we have something equivalent to the second suggestion, which is just putting everything somewhere else.

[Lastique]

Sorry, I must be missing something, but I still don't understand why it is a problem to move _ along with any built docs under libs/<library>/doc/html. That is, assuming _ contains something that can be considered specific to the library documentation. Otherwise, if it's something common, we could store it under doc in Boost root, although that is not a requirement.

Note that links to library documentation are customizable in meta/libraries.json (the "documentation" parameter) and index.html (redirection target) in the library directory.

[alandefreitas]

That is, assuming _ contains something that can be considered specific to the library documentation.

Yes. This assumption is incorrect. _ is shared among more than one library.

Otherwise, if it's something common, we could store it under doc in Boost root, although that is not a requirement.

Yes. This is exactly the discussion here: https://github.com/boostorg/release-tools/issues/57#issuecomment-1984848516. That is, where this should go.

[Lastique]

That is, assuming _ contains something that can be considered specific to the library documentation.

Yes. This assumption is incorrect. _ is shared among more than one library.

Shared in what way? It looks to me that this directory does not contain anything that is specific to each of the libraries using it. Which means, we could duplicate it under each of the libs/<library>/doc/html, if putting it under doc in Boost root is difficult for some reason.

What I'm saying is that _ should be moved to one of these two locations and which of them to choose is up to you, depending on what makes more sense and easier to implement.

[alandefreitas]

Shared in what way?

The _ directory contains resources shared by all libraries that use Antora.

It looks to me that this directory does not contain anything that is specific to each of the libraries using it.

Correct. The content is not specific to any library. They're shared between the libraries.

Which means, we could duplicate it under each of the libs//doc/html, if putting it under doc in Boost root is difficult for some reason.

I think duplicating it for every library that needs it is not something we even did or should consider.

Putting it somewhere under <root>/doc is one of the alternatives we considered above (the second option).

What I'm saying is that _ should be moved to one of these two locations

Yes. That's correct. It's the second option suggested above.

and which of them to choose is up to you,

That I don't agree. I think we should need some form of consensus on where to put it. The issue is small enough so we don't need consensus from all boost developers but we need some local consensus here (assuming anyone interested could be discussing it here).

depending on what makes more sense and easier to implement.

I think any location is just as easy.

[Lastique]

It would be nice if this was resolved before the 1.85 release.

[alandefreitas]

All we need is a reasonable suggestion of a new location for these documentation files. It could be doc/antora or anything. After that, it's up to the release managers to decide if it goes into 1.85. I just don't want to make this choice by myself so that we have another issue in a few weeks.

[Lastique]

I'm sorry, but you're the one who adds support for Antora, aren't you? Who else but you is going to make this decision? As the interested party, you either do what is needed to make it work or you revert the changes and let someone else do it.

There were two reasonable locations suggested here, any of them would do. If you need more input, you could ask on the ML or mention someone particular here, but please don't stall the issue because you don't want to be responsible for the decision.

[alandefreitas]

There were two reasonable locations suggested here

Sorry... what were they?

[Lastique]

There were two reasonable locations suggested here

Sorry... what were they?

Should I mention those for the fourth time?

https://github.com/boostorg/release-tools/issues/57#issuecomment-1883226218 https://github.com/boostorg/release-tools/issues/57#issuecomment-1984957098 https://github.com/boostorg/release-tools/issues/57#issuecomment-1985286216

[alandefreitas]

That's a comment I wrote and I was the one who mentioned it. There's no proposal for any concrete location in there. All it says is "Find another directory for Antora documentation (where?)".

[pdimov]

It's not our job to propose things to you. You are the source of the problem. Fix it.

If you want to give us several solution options from which we're to choose, we can do that.

Otherwise, were I a release manager, I would have already added rm -rf libs/_ to the release script.

[Lastique]

That's a comment I wrote and I was the one who mentioned it. There's no proposal for any concrete location in there.

The three comments I linked are comments by me and they are all mentioning two locations: doc in Boost root and libs/<library>/doc/html. There, I'm listing these locations for the fourth time now.

[alandefreitas]

There were two reasonable locations suggested here

doc in Boost root and libs/<library>/doc/html

As I said, these are not reasonable locations to choose from. doc is already in use and libs/<library>/doc/html doesn't work because it's not one documentation per library.

I'm listing these locations for the fourth time now.

You could have listed it only once if you bothered to read why it doesn't work.

If you want to give us several solution options from which we're to choose, we can do that.

OK. I propose doc/antora.

I'll start writing the PR.

[pdimov]

doc/antora works for me.

[Lastique]

doc/antora works for me.

Me too.

doc is already in use

And yet you chose it.

That directory is used for libraries that build their docs there and for common resources, such as images and stylesheets shared between libraries. I see no problem storing shared Antora resources there.

You could have listed it only once if you bothered to read why it doesn't work.

I don't see where this is explained.

[alandefreitas]

And yet you chose it.

Somewhere in doc (but not root doc) was in my first proposal months ago. There's a huge difference between root doc and some specific place in doc like doc/antora. And you know that. Either that or you're just being stubborn for the sake of it.

[vinniefalco]

Please be professional

[Lastique]

And yet you chose it.

Somewhere in doc (but not root doc) was in my first proposal months ago. There's a huge difference between root doc and some specific place in doc like doc/antora. And you know that. Either that or you're just being stubborn for the sake of it.

Wow, I didn't know I had to state the obvious that no, you don't have to dump the files directly in doc. If you read my comments, I said under doc, which implies that there may be a directory structure inside doc.

But whatever, the point is moot now.