preprocessors need to be integrated with the rust build system

[ehuss]

https://github.com/rust-lang/book/pull/3918 and https://github.com/rust-lang/book/pull/3907 seem to have introduced some new preprocessors. These will need to be integrated with the rust build system so that the book gets built correctly. At a minimum, it will need to be integrated in rustbook/src/main.rs, by calling with_preprocessor when building the book. The preprocessors will need to be declared as path dependencies in rustbook/Cargo.toml. This will also probably require some wiring to detect when these custom preprocessors are enabled in book.toml (which should be accessible via something like book.config.get_preprocessor).

This should probably be done before you start using these preprocessors to ensure the published documentation isn't broken.

[chriskrycho]

Noted, and thank you, @ehuss! 👍🏼 A quick status note:

• Some changes have landed using the <Listing> preprocessor from #3918. I will make sure we either have the relevant changes to rustbook in place at the start of next week or will revert the changes which have landed until we do.
• The changes from #3907 will change the emitted output once we integrate them via with_preprocessor, but don’t make any changes to the source, so should not affect the build process directly.

I am using book.config.get_preprocessor already in tests for the preprocessors, so I have a reasonably good idea how to make sure we do that in the build. I’ll follow up with @carols10cents to make sure I don’t miss anything.

[chriskrycho]

For those following along at home: rust-lang/rust#125408 has the goods, and is (I believe!) ready to go. 👍🏼

[chriskrycho]

Status:

• The first pass on this merged last week.
• It blocked contributions because the bootstrap process did not know about needing to sync the src/doc/book submodule, which was fixed with this workaround.
• I opened a discussion with t-compiler on Zulip to figure out what the preferred path forward is here.

One highly likely possibility here is that we end up publishing these to crates.io and then pulling them in as normal (i.e. not path) dependencies in the src/tools/rustbook in the rust repo.