Move away from Gitbook - Githubissues

straight-shoota commented 6 years ago

If you want to build crystal-book locally, just setting up a working environment is not very easy. Just installing the dependencies - about 140+ npm plugins - takes quite a long time. Then you need to run npm install and gitbook install to install all gitbook plugins. Running npm install again after gitbook install removes some dependencies....

It seems like the open source version of gitbook (and the CLI) is pretty much abandoned (see https://github.com/GitbookIO/gitbook/issues/1808) and hasn't seen any substantial updates for two years.

There are good alternative documentation generators out there, and we can certainly find something that fits.

Obviously, such a change does not necessarily provide direct benefits for readers of the documentation. But it should make it easier for contributers and help stay on path for the long run.

j8r commented 6 years ago

Even more if there is one in Crystal, :100:% for it! We could have one derived from crystal docs.

vegai commented 6 years ago

Asciidoctor is liked by many, and it's a living ruby project: https://github.com/asciidoctor/asciidoctor

madranet commented 6 years ago

As an end user of the docs, rather than a contributor, Im finding the current format of the docs a bit unfriendly. Even just enabling PDF downloads would help:

https://help.gitbook.com/books/how-can-i-provide-a-pdf-version-of-my-book.html

I don't know about the rest of you but, when learning a new language or working through documentation like this, I'll have the documentation the full height of the screen with a terminal window floating on top at half the height of the screen. This allows me to type 'stuff' in the terminal while I scroll the documentation behind, up or down to refer to the text & examples.

With the way this Gitbook documentation is structured [ie. almost as a series of small flashcards], there's no vertical scrolling available, so I have to keep switching back & forth between the docs and my terminal to re-read things and also to keep clicking the 'next' arrows to go to the next flashcard sized bite of info.

2018-09-09_11-11-31

Might sound like a minor issue, but it's really interrupting my flow as I'm working through the docs.

If there was an option to view GitBooks as a single HTML page it would be a lot less hassle but I can't find any way to do that. But, failing that, at least a downloadable PDF would do as I could then set it to display as 'Continuous Scroll'

straight-shoota commented 6 years ago

@madranet Could you please open a new issue for this? This is a different problem and not entirely dependent on switching away from Gitbook as it can at least partially be fixed with the current tools available.

madranet commented 6 years ago

@straight-shoota

Done: https://github.com/crystal-lang/crystal-book/issues/282

straight-shoota commented 4 years ago

I've been experimenting with switching to Jekyll as site generator to resolve this and other issues. So far, it works really good :+1: Jekyll feels like a natural option because it's very popular, there are lots of resources and we're already using it in https://github.com(crystal-lang/crystal-website While Jekyll's initial focus is on blogging, it is very flexible and has been proven for building other things such as documentation sites. There are lot's of examples for that. One that I find appealing and has a similar feature set as we need is https://docs.sentry.io/ They use Jekyll with a minimal-mistakes theme plus a few customizations. It looks really nice.

So I've just initialized a new Jekyll project, moved the existing content to a collection folder and that's how it looks with minimal-mistakes theme:

It pretty much works out of the box with minimal changes to the existing content. I only had to replace the h1 title in the first line of each markdown file with a Jekyll frontmatter.

lunr.js-based search works out of the box. It still experiences issues similar to #242 but the search engine is plugable. Though I doubt there's a feasible alternative to lunr.js
In-page table of contents works out of the box
building a single-page document should not be hard to do (based on the navigation index) and that could be used for generating a PDF (#94, #282)
Github-flavoured Markdown works without breaking tables (https://github.com/crystal-lang/crystal-book/pull/370#issuecomment-547672653)
It adds more flexibility for organizing content. For example guides can be structured differently from reference pages (see #384)
Installation and building are faster than with gitbook

bcardiff commented 4 years ago

I like many of the features listed, and they are aligned a lot with what we want.

Jekyll is one option to go. Gatsby could also be one that I was thinking of (https://github.com/hasura/gatsby-gitbook-starter, https://www.docz.site/). We haven't had that discussion. We already use Jekyll for the main site though.

What we are also working on is in a design proposal to match the rest of the site, so I would suggest to not invest efforts in that aspect before we iterate and share the design.

bcardiff commented 4 years ago

@ftarulla suggested https://v2.docusaurus.io/ . It covers a good amount of the bullets described by @straight-shoota and offers some additional flexibility. The default look and feel is nearer to what we would like.

Disclaimer: old content in the design ;-)

The main difference is that the heading of each section has no content attached to it, it is a collapsible. So maybe as a first iteration there might need to be a "Introduction" item.

We probably will start spiking around to see if docusaurus fits as a replacement for gitbook.

straight-shoota commented 4 years ago

I'm aware of docusaurus, but it certainly wouldn't be my first option. I don't really understand why a you would need a ReactJS stack for a relatively simple static site. Also, one thing that annoys me with gitbook is the insane number of npm dependencies. Docusaurus also has a lot of dependencies. And it's an entirely different stack than Jekyll which we're already familiar from the website.

So I'm wondering, what additional flexibility do you expect from docusaurus...? I don't think a closer default look and feel is a huge argument, because it's really easy to adapt to our needs either way. And theming work is mostly a one-time task, not much of a burdon to carry around.

straight-shoota commented 3 years ago

I've recently seen some very nicely looking documentation sites made with mkdocs, particularly mkdocs-material. So that's defintely something I could very well see for crystal documentation. mkdocs is a tried and proven docs generator and it can have a pretty and modern look that fits with Crystal.

oprypin commented 3 years ago

Count me in, I know a lot about mkdocs. In fact, I think I had tried it out on Crystal ref. But not everything went smoothly.

oprypin commented 3 years ago

Really the super big issue with mkdocs is that it absolutely hates index pages of a section. And believe me that I spent a ton of time trying to make that work. It would need very advanced modding for that to work.

So for the example below there's no way to make "Literals" clickable (it's only expandable), and the next best thing is to just have "Literals → Literals" (or → Overview or whatever).

If we accept this downgrade and/or adapt docs to not need this, then maybe it's even easy.

straight-shoota commented 3 years ago

IMO that's completely acceptable. Just call it Overview and be done. There are pros and cons for having pages for subsection titles (or not), but in the end it probably doesn't affect readers much one way or the other. When I'm writing documentation envision to have pages for subsection titles, it's a bummer. But as a reader, I never care about such details. It's totally fine and maybe even better to have subsection titles only expand or contract the subsection.

oprypin commented 3 years ago

Well it's here https://github.com/oprypin/crystal-book/compare/mkdocs https://oprypin.github.io/crystal-book/index.html

j8r commented 3 years ago

An other option is mdBook, which supports index pages.

straight-shoota commented 3 years ago

That's awesome @oprypin. The source diff is similar to what I did for Jekyll, but even smaller. And the rendered result looks great.

oprypin commented 3 years ago

Really the super big issue with mkdocs is that it absolutely hates index pages of a section. And believe me that I spent a ton of time trying to make that work. It would need very advanced modding for that to work.

Well, the very advanced modding is done!

https://oprypin.github.io/crystal-book/syntax_and_semantics/literals.html

straight-shoota commented 3 years ago

I'm not sure that's an actual improvement. Like I said, subsection index pages are not necessarily a good thing because they can result in conflicting UX. But it's nice to see that you managed to make it happen.

We should be planning for is documentation versioning. Documentation for Crystal 1.0 and 2.0 (and following versions; maybe even plus versions in between) will differ and we should be able to provide different versions of the documentation. Most doc generators should support that. mkdocs and docusaurus both do.

oprypin commented 3 years ago

The branch is ready for a potential pull request. Anything that needs to be communicated, or further discussion? https://github.com/oprypin/crystal-book/compare/mkdocs https://oprypin.github.io/crystal-book/

straight-shoota commented 3 years ago

LGTM :+1: I'm really happy to see this move forward. @bcardiff Could you take a look at this?

bcardiff commented 3 years ago

Ok, let's move to material-mkdocs.

The small diff from the current state, the mature ecosystem and the result are great.

Since the mkdoc and material-mkdocs differs a lot in the look and feel I expect that later we will adapt the style of material to match better the look and feel of the main site. But this can come later.

At least we need a docker-compose.yml with livereload that contributors can use to have an environment to write. That can be used in the CI also.

oprypin commented 3 years ago

Thanks for the confirmation.

Regarding docker-compose.yml. If this was my own project, this would definitely not be what I'd advertise primarily, as I'm sure the setup without it is easier than with it. But I think I can make one, yes.

What I really need to hear about is how the uploading of the website will work. Of course, I have a GitHub Actions workflow to push to gh-pages, but that's not what is used now. Does the current Travis setup work and should we keep it?

straight-shoota commented 3 years ago

Travis should be working ATM. But migrating to GitHub Actions sounds like a good idea.

bcardiff commented 3 years ago

We can move to github actions, leaving the output in a specific folder is enough.

From there we will setup a sync to s3 bucket where the artifacts are stored.

oprypin commented 3 years ago

It doesn't really matter which to use. Sure, I'll add just CI first.

straight-shoota commented 3 years ago

The first step towards mkdocs is made, but this is not yet resolved.

straight-shoota commented 3 years ago

I just checked the output files if we need any redirects. But this looks all good :+1:

$ diff <(find _book -type f -printf '%P\n') <(find site -type f -printf '%P\n')
--- /dev/fd/63  2020-11-29 01:57:53.803974300 +0100
+++ /dev/fd/62  2020-11-29 01:57:53.803974300 +0100
@@ -3,44 +3,50 @@
 getting_started/index.html
 using_the_compiler/crystal-play.png
 using_the_compiler/index.html
-gitbook/gitbook-plugin-search/search-engine.js
-gitbook/gitbook-plugin-search/lunr.min.js
-gitbook/gitbook-plugin-search/search.js
-gitbook/gitbook-plugin-search/search.css
-gitbook/gitbook-plugin-livereload/plugin.js
-gitbook/gitbook-plugin-lunr/search-lunr.js
-gitbook/gitbook-plugin-lunr/lunr.min.js
-gitbook/gitbook-plugin-edit-link/plugin.js
-gitbook/gitbook-plugin-highlight/ebook.css
-gitbook/gitbook-plugin-highlight/website.css
-gitbook/style.css
-gitbook/gitbook.js
-gitbook/gitbook-plugin-sharing/buttons.js
-gitbook/fonts/fontawesome/fontawesome-webfont.woff
-gitbook/fonts/fontawesome/fontawesome-webfont.ttf
-gitbook/fonts/fontawesome/fontawesome-webfont.svg
-gitbook/fonts/fontawesome/FontAwesome.otf
-gitbook/fonts/fontawesome/fontawesome-webfont.woff2
-gitbook/fonts/fontawesome/fontawesome-webfont.eot
-gitbook/gitbook-plugin-fontsettings/fontsettings.js
-gitbook/gitbook-plugin-fontsettings/website.css
-gitbook/gitbook-plugin-ga/plugin.js
-gitbook/gitbook-plugin-offline/service-worker-registration.js
-gitbook/images/favicon.ico
-gitbook/images/apple-touch-icon-precomposed-152.png
-gitbook/theme.js
 database/connection.html
 database/transactions.html
 database/index.html
 database/connection_pool.html
 the_shards_command/index.html
-LICENSE
-README.md
+sitemap.xml.gz
+SUMMARY.html
 conventions/documenting_code.html
 conventions/coding_style.html
 conventions/index.html
-cover.jpg
-service-worker.js
+assets/javascripts/lunr/min/lunr.de.min.js
+assets/javascripts/lunr/min/lunr.hu.min.js
+assets/javascripts/lunr/min/lunr.fi.min.js
+assets/javascripts/lunr/min/lunr.nl.min.js
+assets/javascripts/lunr/min/lunr.multi.min.js
+assets/javascripts/lunr/min/lunr.ru.min.js
+assets/javascripts/lunr/min/lunr.no.min.js
+assets/javascripts/lunr/min/lunr.ja.min.js
+assets/javascripts/lunr/min/lunr.it.min.js
+assets/javascripts/lunr/min/lunr.ro.min.js
+assets/javascripts/lunr/min/lunr.du.min.js
+assets/javascripts/lunr/min/lunr.pt.min.js
+assets/javascripts/lunr/min/lunr.stemmer.support.min.js
+assets/javascripts/lunr/min/lunr.sv.min.js
+assets/javascripts/lunr/min/lunr.fr.min.js
+assets/javascripts/lunr/min/lunr.da.min.js
+assets/javascripts/lunr/min/lunr.ar.min.js
+assets/javascripts/lunr/min/lunr.vi.min.js
+assets/javascripts/lunr/min/lunr.jp.min.js
+assets/javascripts/lunr/min/lunr.tr.min.js
+assets/javascripts/lunr/min/lunr.es.min.js
+assets/javascripts/lunr/tinyseg.min.js
+assets/javascripts/worker/search.4ac00218.min.js.map
+assets/javascripts/worker/search.4ac00218.min.js
+assets/javascripts/vendor.fd16492e.min.js.map
+assets/javascripts/bundle.7836ba4d.min.js
+assets/javascripts/vendor.fd16492e.min.js
+assets/javascripts/bundle.7836ba4d.min.js.map
+assets/images/favicon.png
+assets/stylesheets/main.6910b76c.min.css.map
+assets/stylesheets/main.6910b76c.min.css
+assets/stylesheets/palette.196e0c26.min.css.map
+assets/stylesheets/palette.196e0c26.min.css
+sitemap.xml
 guides/writing_shards.html
 guides/continuous_integration.html
 guides/hosting/gitlab.html
@@ -52,7 +58,7 @@
 guides/ci/circleci.html
 guides/ci/travis.html
 guides/index.html
-search_index.json
+404.html
 syntax_and_semantics/type_grammar.html
 syntax_and_semantics/as_question.html
 syntax_and_semantics/splats_and_tuples.html
@@ -86,7 +92,6 @@
 syntax_and_semantics/constants.html
 syntax_and_semantics/class_methods.html
 syntax_and_semantics/as_a_suffix.html
-syntax_and_semantics/blocks_performance.md
 syntax_and_semantics/virtual_and_abstract_types.html
 syntax_and_semantics/instance_sizeof.html
 syntax_and_semantics/the_program.html
@@ -111,7 +116,6 @@
 syntax_and_semantics/or.html
 syntax_and_semantics/requiring_files.html
 syntax_and_semantics/comments.html
-syntax_and_semantics/proc_pointer.md
 syntax_and_semantics/type_inference.html
 syntax_and_semantics/truthy_and_falsey_values.html
 syntax_and_semantics/inheritance.html
@@ -165,4 +169,5 @@
 syntax_and_semantics/if_var_nil.html
 syntax_and_semantics/block_forwarding.html
 index.html
-overview/README.md
+overview/index.html
+search/search_index.json

The only thing that popped out was SUMMARY.html which is probably not useful. And LICENSE should prob be included in the generated output.

straight-shoota commented 3 years ago

When deploying we should make sure that the obsolete paths are actually removed from S3. IIRC it didn't sync deletion of files and we ended up having completely outdated sites being still accessible (not so much an issue here because content didn't change, but still).

bcardiff commented 3 years ago

I guess we should add a .js snippet to unregister gitbook's service worker, right?

j8r commented 3 years ago

Any reasons not to switch to GitHub Pages?

straight-shoota commented 3 years ago

Any reason to do that?

j8r commented 3 years ago

It is easy for everyone to fix the issue, if there are any. Plus the deployment is easy as well.

oprypin commented 3 years ago

You're forgetting that Crystal book is just one subdirectory in a site that spans several different repositories and deployments and possibly even ownership models.

j8r commented 3 years ago

Yes @oprypin , one reason why the deployment is more sensitive and complex now. This means having a subdomain. This is usually done for different sites, which is the case here.

That's just an idea, most projects use GH Pages on GitHub for convenience and simplicity.

oprypin commented 3 years ago

Subdomains are actually not an option that someone would choose willingly. Anyway, last I've seen, GH pages is not good at handling subdomains.

j8r commented 3 years ago

That's possible, haven't tested I admit. If IAM policies are done correctly, the deployment to S3 should be secure. Then, we can use aws s3 rm s3://bucket/folder --recursive before deploying.

bcardiff commented 3 years ago

The current deployment works good enough. I don't think there are enough reasons to move away from S3 as used today. Also, have in mind that this is administered by Manas and there are more people involved and reasons to why we have the setup the way it is.

Of course there is room for improvement, I wouldn't mind moving to a subdomain if that simplifies something in the future.

straight-shoota commented 3 years ago

I think this can be closed now.

obtusedev commented 3 years ago

I've recently seen some very nicely looking documentation sites made with mkdocs, particularly mkdocs-material. So that's defintely something I could very well see for crystal documentation. mkdocs is a tried and proven docs generator and it can have a pretty and modern look that fits with Crystal.

Is it possible to bring Mkdocs Material Theme to the Crystal Standard Library API documentations?
It looks very date in comparison to the Crystal Language Reference.
I open an issue here #546

crystal-lang / crystal-book

Move away from Gitbook #261