Add CI system and integrate into pages

[bytesnake]

We should build and publish the book automatically to rust-ml.github.io/book on each push:

• [ ] add small CI system which builds the book
• [ ] add gh branch with artifacts

[liufuyang]

I think we can use this https://github.com/peaceiris/actions-mdbook

I used it earlier in my personal mdbook note and it seems working pretty well. Should I create a PR here for you?

It's basically adding a yml file then change the repo setting like this:

Or... we are not using a standard mdbook perhaps it needs to be tweaked or? 🤔

[bytesnake]

We're using mdbook here, but with some tweaks :laughing: so there is the plugin https://github.com/rust-ml/book/tree/master/mdbook-scientific which basically adds special syntax for latex/gnuplot code blocks. The environment would need:

• mdbook
• a compiled version of the mdbook-scientific plugin
• LaTeX, perhaps gnuplot
• bib2xhtml

I know that these are a lot of dependencies, but perhaps you can use the mdbook action as baseline and then install/compile the rest

[liufuyang]

I see. I am not that experienced with github action yet but this is probably can be done.

[liufuyang]

@bytesnake I tried to make an PR https://github.com/rust-ml/book/pull/8 to make the first version CI work. How do you like the general idea? It is not fully working yet, but I guess with bib2xhtml install then we will be closer to it. It would be nice if you can help make a master-test-build branch for me to test the CI build later on?

[liufuyang]

https://github.com/rust-ml/book/pull/8 indeed can trigger github actions but the build failed. I will take a deeper look later this week :)

[liufuyang]

So I tried to let the build scrip just run twice then it can be built. Seems like a race condition thing perhaps? So when building the first time, some build process didn't wait for the needed middle files to be generated on disk? You can see the error message like this:

Run mdbook build || mdbook build
2021-04-19 17:44:06 [INFO] (mdbook::book): Book building has started
Error in chapter 1. InvalidMath("! I can\'t find file `e9ea986c569b3a41be529169.tex\'.", "", 18446744073709551615)
Error: -19 17:44:07 [ERROR] (mdbook::utils): Error: The preprocessor exited unsuccessfully
2021-04-19 17:44:07 [INFO] (mdbook::book): Book building has started
2021-04-19 17:44:07 [INFO] (mdbook::book): Running the html backend

I noticed second build can work is because I remembered days ago I made the image to work with the book on that build branch. As I am on a new computer now, I noticed the image was not working, but simply try running again without touching anything made the build works.

But I guess it still only works with the book of the version that is on the build branch. Need to solve how to make master branch also works later.

Anyway you have something here already https://rust-ml.github.io/book/

[liufuyang]

Just some helpful note here for anyone would like to try to use the image on master to build the book and see the error: Firstly checkout master branch. Then update book.toml setting with bib2xhtml = "/github/bib2xhtml/" Then you can run docker build

docker run --rm -v $(pwd):/github/workspace liufuyang/mdbook-scientific:0.3.7 build

or if you want to take a look inside docker:

docker run --rm -v $(pwd):/github/workspace --entrypoint /bin/bash -it liufuyang/mdbook-scientific:0.3.7

# then you can run: mdbook build

The error looks like this:

2021-04-19 18:22:56 [INFO] (mdbook::book): Book building has started
Block empty, but file `hyperplane` was not found!
Block empty, but file `hyperplane` was not found!
Error in chapter 1.: InvalidReference("could not find reference to `hyperplane` in line 5")
2021-04-19 18:22:56 [ERROR] (mdbook::utils): Error: The preprocessor exited unsuccessfully

I have no clue what hyperplane is about...

[bytesnake]

first: thanks for all the work! I will try to reproduce the build error inside the docker image. I have currently too many side-projects to really focus on one thing, what always works is 1-to-1 meetings and debugging code together. Perhaps I'll find time this evening, but otherwise I'll propose that you join in the bi-monthly meetings and we can look into the issue afterwards?

[liufuyang]

Oh cool. I might have time tonight as well. I haven't joined the bi-monthly meetings before but I would like to come join once and listen to your discussion. However I might be difficult to find time on Wednesday as I have some club activity on Wednesday evening. But let's see, maybe I can join someday.

[bytesnake]

I'm giving it a try now, posted a link in zulip.

However I might be difficult to find time on Wednesday as I have some club activity on Wednesday evening. But let's see, maybe I can join someday.

would be awesome!

[liufuyang]

Okay, so I made a large (if not huge) image to include all the rust build environment as well. You can see the code changes here that for building this large image.

So basically you can either try pull the image (which might take a long time)

docker pull liufuyang/mdbook-scientific:0.3.7-large

or build the image on your own (also very slow the first time) by checking out that master-test-build-large-image branch and run build script.

* If you just pull the image (the easy way)

Then I guess it's simple as you can test your Rust code changes by running the image via interaction mode, such as:

docker run --rm -v $(pwd):/github/workspace --entrypoint /bin/bash -it liufuyang/mdbook-scientific:0.3.7-large

Then you can do build and run as if you are inside an ubuntu:

cd /github/workspace/mdbook-scientific
cargo install --path .
cd /github/workspace/
mdbook build

Then your updated Rust code will run

* If you decide to build the image on your own (the harder way)

Basically you can just cp your original Dockerfile into a backup file, then use the Dockerfile from the branched I mentioned above. Then build the image on your own

docker build -t test:latest .

Each time you update the Rust code, you need to re-run the build step above so the new binary is baked into the image. Then you can run it as below to build the book (which should execute your updated Rust code!)

docker run --rm -v $(pwd):/github/workspace test:latest build
# or like this to do interaction way:
#  docker run --rm -v $(pwd):/github/workspace --entrypoint /bin/bash -it test:latest

@bytesnake Let me know whether this helps :)

[liufuyang]

Here I just put some command I ran as a demo:

 docker run --rm -v $(pwd):/github/workspace --entrypoint /bin/bash -it liufuyang/mdbook-scientific:0.3.7-large
root@1dd4dd6cf0a8:/github/workspace# ls
Dockerfile  Dockerfile.back  LICENSE-APACHE-2.0  LICENSE-CC-BY-SA  LICENSE-MIT  README.md  book  book.toml  build.docker.sh  code  fragments  literature.bib  mdbook-scientific  src
root@1dd4dd6cf0a8:/github/workspace# cd mdbook-scientific/
root@1dd4dd6cf0a8:/github/workspace/mdbook-scientific# cargo install --path .
  Installing mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)
    Updating crates.io index
   Compiling version_check v0.9.3
   Compiling cfg-if v1.0.0
   Compiling libc v0.2.94
   Compiling proc-macro2 v1.0.26
   Compiling unicode-xid v0.2.1
   Compiling autocfg v1.0.1
   Compiling syn v1.0.70
   Compiling memchr v2.3.4
   Compiling ryu v1.0.5
   Compiling bitflags v1.2.1
   Compiling proc-macro2 v0.4.30
   Compiling unicode-xid v0.1.0
   Compiling syn v0.15.44
   Compiling getrandom v0.2.2
   Compiling lexical-core v0.7.6
   ...
   Compiling mdbook v0.3.7
   Compiling mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)
    Finished release [optimized] target(s) in 7m 07s
   Replacing /root/.cargo/bin/mdbook-scientific
    Replaced package `mdbook-scientific v0.1.0 (/book/mdbook-scientific)` with `mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)` (executable `mdbook-scientific`)
root@1dd4dd6cf0a8:/github/workspace/mdbook-scientific# cd /github/workspace                  
root@1dd4dd6cf0a8:/github/workspace# mdbook build
2021-04-26 21:27:48 [INFO] (mdbook::book): Book building has started
2021-04-26 21:27:48 [INFO] (mdbook::book): Running the html backend
root@1dd4dd6cf0a8:/github/workspace#

Now you see it compiles (as it is on the build branch). Then I keep the the container running but in the editor I go to the scientific book main function add a simple println!("my hack..."); as the first line in the main function, then I run the following commands in container as: (following from the terminal prompt above)

root@1dd4dd6cf0a8:/github/workspace# cd mdbook-scientific/
root@1dd4dd6cf0a8:/github/workspace/mdbook-scientific# cargo install --path .
  Installing mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)
    Updating crates.io index
   Compiling mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)
    Finished release [optimized] target(s) in 19.33s
   Replacing /root/.cargo/bin/mdbook-scientific
    Replaced package `mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)` with `mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)` (executable `mdbook-scientific`)
root@1dd4dd6cf0a8:/github/workspace/mdbook-scientific# cd ..
root@1dd4dd6cf0a8:/github/workspace# mdbook build
2021-04-26 21:32:22 [INFO] (mdbook::book): Book building has started
my hack...
2021-04-26 21:32:22 [ERROR] (mdbook::utils): Error: Unable to parse the preprocessed book
2021-04-26 21:32:22 [ERROR] (mdbook::utils):    Caused By: expected value at line 1 column 1
root@1dd4dd6cf0a8:/github/workspace# 

You can see the changes in Rust code taking effect. Hopefully this helps! :) (The simple println seemed broke the mdbook build, but I guess you know why...)

[bytesnake]

the asset config was just set to the wrong path. Replacing assets = "mdbook-scientific" by assets = "src/ fixes the error

[liufuyang]

aha? let me give it a try the other day :) Mind if I ask why in the master branch the assets is set as assets = "mdbook-scientific"? Is it for easy development or? Otherwise perhaps we can change it to src?

[bytesnake]

I'm not sure but the use of assets is not explained .. @quietlychris

[quietlychris]

I honestly am not sure why it got changed from "src" to "mdbook scientific" although looking through my commits, I was the one to change it :sweat_smile: I think I may have been playing around with experimenting with different locations for the k_means code prior to that commit and forgot to move it back to the right position. Sorry about that!

[bytesnake]

no worries :smile: the only place where the source folder is actually used is when a latex file is loaded. So it could probably be replace, but haven't looked into it yet

[liufuyang]

CI should be ready after this https://github.com/rust-ml/book/pull/9

[quietlychris]

@bytesnake Just wanted to follow up on this, and maybe try to have the CI pushing to the Github Pages site over the next two weeks. I think that as liufuyang mentioned in his last comment in #9, would it possibly make sense to move mdbook-scientific out of this repository and call it as a dependency when needed? At the moment, I don't think that any of the content that I've added is using it as the moment, and I agree that it might make sense to move it into it's own repo like rust-ml/mdbook-scientific so as not to gate this repo on it. If you're onboard, I can try to take care of that tonight. Any thoughts?

[bytesnake]

@bytesnake Just wanted to follow up on this, and maybe try to have the CI pushing to the Github Pages site over the next two weeks. I think that as liufuyang mentioned in his last comment in #9, would it possibly make sense to move mdbook-scientific out of this repository and call it as a dependency when needed? At the moment, I don't think that any of the content that I've added is using it as the moment, and I agree that it might make sense to move it into it's own repo like rust-ml/mdbook-scientific so as not to gate this repo on it. If you're onboard, I can try to take care of that tonight. Any thoughts?

yes moving mdbook-scientific to its own repo and making it optional if not used sounds reasonable

[quietlychris]

CI updates to Github Actions with mdbook publication for new structure in #12