search

algorithm-archivists / algorithm-archive

A collaborative book on algorithms
https://www.algorithm-archive.org
MIT License
2.33k stars 352 forks source link

The Great GitBook Rewrite #611

Open Trashtalk217 opened 5 years ago

Trashtalk217 commented 5 years ago

GitBook takes the content and converts it to the HTML code webbrowsers can read. However, it became apparent that GitBook is no longer a great fit for the AAA.

Reasons

The version of GitBook we are using is no longer supported. This means that security flaws and bugs are no longer fixed by the dev team.

These problems are the main reasons for a migration away from GitBook, but there are also other potential benefits we could enjoy by switching.

Other Benefits

There are probably other advantages to switching I forgot to mention, but these are the main ones. I think I've made my case pretty clear and most of the archivists already agree with me that switching is a good move. This is not my idea, I'm just stating what has already been said before.

Now the question becomes: 'What do we switch to?"

Options

Final Advice

My final advice is to just start. The current system works, but when something works it's all to easy to become complacent. There current setup can improve and if we don't do anything, it won't improve. So I recommend Leios to open up a separate chat in the Discord server under projects called something along the lines of 'AAABook'. It's also important that we have an official place to work on this projects so I recommend Leios to open up a separate project on GitHub. And while I know that it will still take a while to properly get started, I hope that this can be a start.

leios commented 5 years ago

Thanks for the write-up, it is thorough and clear!

I think @berquist was working on an mdbook solution, but couldn't replicate the language switching behaviour. I am also not sure how @Neverik's python build is.

@c252 has also started working on a script to translate the AAA into LaTeX, which might be worth looking at too.

berquist commented 5 years ago

Here is the Rust "conversion" with https://github.com/rust-lang-nursery/mdBook:

It looks like I renamed contents/ to src/, nuked the JS config, and added a minimal mdbook config. Most notably,

To try and build it,

berquist commented 5 years ago

Maybe relevant for syntax highlighting, it indicates you can add HTML/JS and have it picked up: https://github.com/rust-lang-nursery/mdBook/issues/657

june128 commented 5 years ago

I created a project board which holds requirements and optional requirements for the AAAbook solution: https://github.com/orgs/algorithm-archivists/projects/2 Please feel free to discuss and extend the requirements, so that we end up with a good understanding of our needs.

Trashtalk217 commented 5 years ago

Thanks to the work of @Neverik, we have a working alternative to gitbook. This has been posted in the discord chat but I feel that documenting it here is important.

Minimum viable alternative: https://aaapy.herokuapp.com/

neverix commented 5 years ago

Just as an example of what aaabook can look like, here's an AAA-Py screenshot image

leios commented 5 years ago

Yeah, I have no doubt the aaa-py can work for this. I'm excited to see where it goes. I'm open to other ideas from the community as well.

Trashtalk217 commented 5 years ago

We have to make a decision between mdbook and aaa-py someday and I feel like providing some stats might help with that. Below is listed a hassel comparison.

Hassel

Hassel is defined as the difficulty in installing and using the software. To stay open towards newer programmers trying to contribute to open source software, the hassel required getting aaa to build should be as low as possible.

Mdbook

I'll describe the setup required to set up the program in these sections.

Linux setup

The Linux setup is relatively simple although it's not all out of the box.

Windows setup

Build process (both Linux and Windows)

AAA-py

Linux setup

Windows setup

Build process Linux

Build process Windows

Conclusion

The Rust for Windows installation was definitely the most painfull, since I had to figure a lot out on the fly. The "Build Tools" download was definitely the most annoying. I had to install about 5 GB of stuff to make an error go away. Mdbook had a definite advantage over aaa-py in the build department. Just one command and it just works. While you have to type like three to get the aaa-py online. AAA-py has a major advantage over mdbook in that we can in fact do something about the current setup and build steps. The pip downloads are probably easy to change.

P.S. These instructions give you a working algorithm archive, but this doesn't mean this is also the shortest way to do things. The instructions are unoptimized and some corners could potentially be cut.

Trashtalk217 commented 5 years ago

Now a short speed comparison.

Speed

All of the statistics come from a test done on Linux using the time command.

Mdbook

$ time mdbook load

Gives the following output:

real    0m0,007s
user    0m0,007s
sys 0m0,000s

I tried this several times, but the main takeaway here is that it's ridiculously small. Instant for all practical purposes.

AAA-py

$ time python3 build.py

Gives the following output in normal operations:

real    0m1,133s
user    0m2,750s
sys 0m0,144s

The same command gives the following time when the contents folder is not present (it needs to be cloned). But this barely happens since you only really need to get the full folder once. This is the worst case scenario.

real    0m4,645s
user    0m3,009s
sys 0m0,278s

Conclusion

Mdbook is instant while aaa-py is just very fast. The main thing is that they are both infinitely faster than GitBook.

Trashtalk217 commented 5 years ago

Feature comparison

Compare for yourself! Mdbook vs aaa-py

neverix commented 5 years ago

The commands you ran do different things: build.py builds the entire project from scratch and mdbook load... Shouldn't even exist? This comparison isn't really fair because mdbook load just fails. I believe you used the wrong command. But even then, aaa-py still doesn't have live reload - which is why it may sometimes be slower. I would suggest someone to try to comprehend its source and maybe add live reload. aaa-py doesn't have an integrated web server either - that would be an interesting thing to add too.

Trashtalk217 commented 5 years ago

Revision

A slight revision to what I just said. Got you @neverix !

AAA-py

AAA-py is a pipenv project so there is a slight different setup then what I described. Instead of installing all dependencies with pip install dependency-name, the following command can be used:

This also changes the build process, which now turns in this:

This also changes the speed of the build process.

`$ time pipenv run python3 build.py`

Gives the following output:

real    0m1,571s
user    0m3,062s
sys 0m0,189s

Mdbook

mdbook load doesn't actually exists, turns out. So to get accurate measurements I had to use mdbook build.

$ time mdbook build

Gives he following output:

real    0m0,271s
user    0m0,217s
sys 0m0,053s
leios commented 5 years ago

I am not convinced that the mdbook build is actually that much faster. After all, it is lacking core functionality to switch languages, which takes up a majority of the compilation time. If we add this in, it will probably be much slower.

Honestly, even if we don't get a huge performance benefit from changing to a new markdown renderer, that's ok. We are ultimately trying to find a platform that will be maintainable in the long-run.

I am leaning towards the aaa-py repo right now because it seems more maintainable and is fast enough for now. If we need to update it, we can do so because it's python and everyone knows python. On the other hand, I thought the rust build process was incredibly easy and fast.

I did a strawpoll on stream, and it was somewhat split: https://www.strawpoll.me/18273691/r

As another note: eventually, we would like to find a custom way to output the aaa into a pdf file / book. One viewer (teko2k) suggested potentially looking into orgmode. Here is an example book made in this way: https://github.com/jkitchin/pycse

leios commented 5 years ago

After a lot of discussion amongst the community, it seems like we are more or less settled on the python version. That said, there was some discussion on discord not too long ago about another js method, but right now, I think we should emphasize moving away from gitbook.

I think we need to get the aaa-py repo to have all the requirements on the project board (on which, I slightly modified some of the notes): https://github.com/orgs/algorithm-archivists/projects/2

We had a discussion about one note on stream, particularly the one that says, Be a standalone executable that can be downloaded reasonably quickly and have minimal dependencies. I believe this is useful, but others feel that it might be alright to couple the solution with the AAA, itself.

Assuming we are moving forward with the aaa-py, should we modify the project board and add a column for competed requirements?

Amaras commented 2 years ago

I feel like since we settled for Honkit instead of AAA-py, this issue is now irrelevant. I won't close it until I have confirmation from @leios, though

leios commented 2 years ago

Right, I think we can probably close this one, but I want to be clear and thank @Neverik for all the work put into this and @Trashtalk217 for leading the conversation here!

neverix commented 2 years ago

Feels like the end of an era :cry:

leios commented 2 years ago

I know, I 100% agree.

leios commented 2 years ago

Reopening this issue as we are now thinking about switching off of honkit as well.

All of the same alternative options apply here, except now I would also like to consider Franklin.jl. Right now, I see 3 viable alternatives:

  1. mdbook
  2. aaa-py
  3. franklin.jl

None of these will have language swapping and such immediately, so we are looking for the thing that is most robust and easiest to maintain in the future.

leios commented 2 years ago

So I just built both the AAA-py and mdbook versions. Here are the positives of both:

Mdbook positives:

  1. Build is super fast
  2. Dynamic search is nice

Mdbook negatives:

  1. Missing core features (language swapping and math)
  2. Built on rust, which only a few community members know

AAA-py positives:

  1. Most things work. There are a few small rendering issues, but markdown, bibliographies, math, language swapping are all basically working
  2. Home-built, which means it will be "easier" to customize

AAA-py negatives:

  1. No dynamic search
  2. It's home-built, so we need to maintain it

Franklin will obviously take more work. So I will look at that later.

ShadowMitia commented 2 years ago

Mdbook has some mathjax support: https://rust-lang.github.io/mdBook/format/mathjax.html