[RFC] What is the scope?

HugoGranstrom commented 3 years ago

This is a repo that's meant to guide people new to scientific programming in Nim. The possible number of different guides that would satisfy that description are many. Therefore I see it as reasonable that we try to narrow it down a bit so potential readers aren't overwhelmed by content that is irrelevant to 99% of newcomers. I propose that we start off with two section: Basics, Advanced. This way it's easy for newcomers to find what they are looking for and we can also have more niche tutorials and what not for users that want to see Nim in action (🥁).

Basics

Here we can have tutorials on the basics like reading from different files, basic plotting, and other common tasks many are looking for a quick guide on how to do. In my opinion, it should not go through basic Nim syntax. For that, we could have a Nim Crash Course or just point them to the Learn page on the Nim website. This way we can keep this section more concise and to the point.

Advanced

Here we can have nearly everything that involves coding in Nim + science. Everything from a tutorial on how to do more advanced plotting to a project-based tutorial explaining advanced deep learning to analyze an image.

This is just my thoughts, alas not any kind of truth. Think of it more as a template to either modify or discard complete.

Let's discuss!

Clonkk commented 3 years ago

Here's a list of tutorial that I would have found useful when I started Nim :

Different method of iterating over a Tensor in parallel
Performing interpolation
Pre-calculating interpolation index
Creating Grid & Regular Grid
Plotting data

I'll get it to it when I have time.

HugoGranstrom commented 3 years ago

I'm thinking about how we should structure this. Should we make a dedicated website with the guides here on Github or should just have a list of links to separate bits and pieces scattered around personal blogs and whatnot? If we go for a dedicated website, I think Nimib would be a great fit for this. Especially when it gets support for blogging. What do you think?

Clonkk commented 3 years ago

The only two constraint IMO:

we need a tool that compile / run the code examples, otherwise those examples will become obsolete.
We need an easily modifiable standard format so either markdown / RST or AsciiDoc

If Nimib fulfil both these condition, then it's okay for me

HugoGranstrom commented 3 years ago

It definitely satisfies the first requirement, it runs the code snippets and also prints out their outputs when generating the pages. So if we set up a CI to rebuild it for every commit + once in a while, we should be fine. Regarding point 2: it uses markdown for formatting, but it is inline in the Nim file in strings if it matters.

Clonkk commented 3 years ago

I tested nimib, it's good enough for me.

I would still advocate to write all the documentation in a dedicated .md file (in case we want to migrate) and use this little trick for rendering :

example.nim

import nimib
import strutils

proc mdFromFile(filename: string) : seq[string] =
  let f = open(filename)
  defer: f.close()
  result = readAll(f).split("@section")

let content = mdFromFile("parag1.md")
nbInit
nbText: content[0]
nbCode:
  let x = @[1, 2, 3, 4, 5]
  echo x
nbText: content[1]
nbSave

parag1.md

# Parag 1 

Hello world

@section

# Parag2

This is my second paragraph

Otherwise there is :

But they seem less mature

HugoGranstrom commented 3 years ago

That seems like a good compromise to me 😄 It's less flexible if one wants to add a paragraph in the middle though. One would have to shift all indices below that then. But it could be solved by keeping track of an index variable and incrementing it on every nbText. We could probably make a template like this:

var contentIndex = 0
let content = ...
template textIndexed:
  nbText: content[contentIndex]
  inc(contentIndex)

textIndexed()
nbCode: ...
textIndexed()

Haxdoc and nimdoc feel more like documentation generators to me. And I see this project more as usage tutorials instead of just a list of procs. IMO every package should be responsible for its own docs and this is just a guide on how to combine the different packages in short snippets/longer tutorials. Do we share the same view on things? 😃

Clonkk commented 3 years ago

t's less flexible if one wants to add a paragraph in the middle though.

As you pointed out, it's easily fixed either by template/proc. We can clean it up a bit to remove boilerplate code :

import nimib
import strutils

type
  MdContent = object
    filename: string
    content: seq[string]
    sep: string
    index: int

proc initMdContent(filename: string) : MdContent  =
  let f = open(filename)
  defer: f.close()
  result.index = 0
  result.sep = "@section" # Allow to easily change section string
  result.filename = filename
  result.content = readAll(f).split(result.sep)

proc next(md: var MdContent): string =
  result = md.content[md.index]
  inc(md.index)

proc reset(md: var MdContent) =
  md.index = 0

block parag1:
  var mdfile = initMdContent("parag1.md")
  nbInit
  nbText: next(mdfile)
  nbCode:
    let x = @[1, 2, 3, 4, 5]
    echo x
  nbText: next(mdfile)
  nbSave

Alternatively, Nimib print out the generated content in markdown form already so we could just use that if we ever want to migrate.

I don't have enough perspective on large documentation to know if editing markdown file is easier than editing markdown in Nim file.

Haxdoc and nimdoc feel more like documentation generators to me. And I see this project more as usage tutorials instead of just a list of procs. IMO every package should be responsible for its own docs and this is just a guide on how to combine the different packages in short snippets/longer tutorials. Do we share the same view on things?

From the description of each projects, it appears that you're right. Nimib seems like a good choice. I opened an issue to have support for a mdbook theme - that'd be the only feature we could really need.

HugoGranstrom commented 3 years ago

next looks even better :)

Yeah, that's a fair point, it would be nice not to have stuff that's connected be spread out in multiple files. I think it depends on why we would want to migrate in the future. Is it because we find a better alternative (then doing everything in Nimib and exporting md works fine) or is it in the case that Nimib would become unmaintained sometime in the future and possibly break on future Nim versions? (then it may be more trouble exporting md). I doubt the second scenario would happen in a way that breaks everything so I'd be confident in embarrassing Nimib fully. The advantage of markdown in separate files is syntax highlighting. The advantage of markdown in Nim is a clearer structure and fewer headaches counting sections. Otherwise, I don't think it matters that much if we have a large or small project as we will split tutorials about different things into different files either way. My vote is for full-on Nimib 😛

From the description of each projects, it appears that you're right. Nimib seems like a good choice. I opened an issue to have support for a mdbook theme - that'd be the only feature we could really need.

Nice! Showing support is great. I wonder how hard it would be, feels like it could be easy/not-hard enough for us to dip our toes in it and see. But that will need to wait until Pietro has done his refactoring of Nimib :)

Clonkk commented 3 years ago

I think it depends on why we would want to migrate in the future. Is it because we find a better alternative

This is the most likely (IMO only) scenario to consider.

it would be nice not to have stuff that's connected be spread out in multiple files. The advantage of markdown in separate files is syntax highlighting. The advantage of markdown in Nim is a clearer structure and fewer headaches counting sections.

We can try using Nimib with Nim file only and if editing markdown inside Nim file is too troublesome, we have a back up plan.

My vote is for full-on Nimib

Agreed.

HugoGranstrom commented 3 years ago

This is the most likely (IMO only) scenario to consider.

Agreed, but I'm having a hard time to imagine how such an alternative in Nim could be so very different than Nimib. It feels like if there will be alternatives, they will take inspiration from it and thus shouldn't be too hard to migrate.

We can try using Nimib with Nim file only and if editing markdown inside Nim file is too troublesome, we have a back up plan.

That sounds like a plan! 😄 I thought I'd try and setup a Github Pages site for this project but I couldn't find the Settings cog icon. Do I have to make a commit to count as a conreibuter to the project perhaps? 🤔

zetashift commented 3 years ago

Just wanted to chime in, I really like Nimib too and like it for tutorials too!

The advantage of markdown in separate files is syntax highlighting.

One could make a vscode nimib plugin modifying this: https://github.com/nukopy/python-string-markdown

I am a neovim user and I don't know a similiar plugin for (neo)vim.

Emacs users could use: https://github.com/polymode/polymode

HugoGranstrom commented 3 years ago

I threw together a simple demo of what it will look like: https://scinim.github.io/getting-started/basic_plotting.html Now it's just a matter of automating it. Maybe committing the Nim files to main branch and placing all the HTML and generated files in gh-page? (The automation script should do this step of course)

One could make a vscode nimib plugin modifying this: nukopy/python-string-markdown

Oh that's definitely a possible solution 😮

Clonkk commented 3 years ago

I threw together a simple demo of what it will look like: https://scinim.github.io/getting-started/basic_plotting.html Now it's just a matter of automating it. Maybe committing the Nim files to main branch and placing all the HTML and generated files in gh-page

You can take a look at https://github.com/SciNim/nimfftw3/blob/master/.github/workflows/docs.yml where I automate docs generation from master branch that gets rebased every merge (so it doesn't accumulate diff commit over time) and use github page to host it.

All that's missing is generating an index.html that contains a way to navigate between multiple html page (we could copy the way mdbook does it ?)

zetashift commented 3 years ago

All that's missing is generating an index.html that contains a way to navigate between multiple html page (we could copy the way mdbook does it ?)

Doesn't https://scinim.github.io/getting-started/ basically do this? EDIT: or do you mean a Table of Contents/index that's shown as a sidebar on every page?

HugoGranstrom commented 3 years ago

You can take a look at SciNim/nimfftw3@master/.github/workflows/docs.yml where I automate docs generation from master branch that gets rebased every merge (so it doesn't accumulate diff commit over time) and use github page to host it.

Oh, that's neat, will take a look. Thanks :)

Yeah, the index.html is a bit tricky. Either we add pages manually to it (scary stuff 😱) or we find a way to generate it from the file structure at compiletime. We could just list all files for now but in the future categories would be useful.

HugoGranstrom commented 3 years ago

Doesn't scinim.github.io/getting-started basically do this? EDIT: or do you mean a Table of Contents/index that's shown as a sidebar on every page?

A TOC would be gold!⭐ But for now just having a somewhat decent index page would suffice (that we don't have to manually edit preferably). I suspect such functionality may come to Nimib itself in the future.

Clonkk commented 3 years ago

EDIT: or do you mean a Table of Contents/index that's shown as a sidebar on every page?

This was what I meant yes :). The default https://scinim.github.io/getting-started/ does not allow you to navigate between page it just display all the existing html files.

Also, @HugoGranstrom as a rule of thumb, I wouldn't commit the generated html file instead set-up a github action to generate the html file on gh-pages branch.

HugoGranstrom commented 3 years ago

Also, @HugoGranstrom as a rule of thumb, I wouldn't commit the generated html file instead set-up a github action to generate the html file on gh-pages branch.

Yes I know 😄 just wanted to get something up there to see that it works without having to spend all day on fixing the automation 🙃

Clonkk commented 3 years ago

I'll take care of it

HugoGranstrom commented 3 years ago

Wow that was quick, thanks 😁

And yes I think we should be able to generate the index.html in the nimble file. It's just a matter of creating a template html in a string and interpolating list elements I think. The same goes for a mdbook-like sidebar, we just have to create a template html+css+js and interpolate the links.

Clonkk commented 3 years ago

@HugoGranstrom Okay, CI seems to work.

It is triggered by any push to to the main branch (pull request excluded but merge included).

It's very basic for now, it look for every .nim file in the books folder and execute it. Later, we may need to add a pattern to the nim file we execute (maybe ``tutorial.nim`` something like that... we'll see). Then the generated html files are moved to docs folder (keeping the structure).

You can test locally using nimble genbook and see how docs/ gets populated. Note that due to rsync usage it's Linux only - either we implement rsync equivalent in Nim so it works on Windows or we tell Windows user to use WSL.

You can also fork the repository and push to your forked main branch I suppose that should work.

For now, the index.html is manual. Generating index.html and a table of content should be the next step IMO.

pietroppeter commented 3 years ago

Hi all, nice to see this coming along, I will try to help!

Regarding the index.html, until there is better support for nimib, I am using a very simple approach in nblog: https://github.com/pietroppeter/nblog/blob/main/index.nim

Regarding the structure, is there a particular reason why you want to keep a separate books and docs folder and remove the nim files?

Note that nimib will automatically register as "home" directory the docs directory see https://github.com/pietroppeter/nimib#interaction-with-filesystem This is the reason why you see ../books/basic_plotting.nim at the top.

The home link in top left of default theme will then look for home (and it will then automatically point the index.html if there is one, providing basic navigation capabilities).

Clonkk commented 3 years ago

I will try to help!

Nice ! Help is always welcome :wave:

Regarding the index.html, until there is better support for nimib, I am using a very simple approach in nblog: https://github.com/pietroppeter/nblog/blob/main/index.nim

I think the index.html should reflect the table of content - which means we have to solve the table of content issue first

Regarding the structure, is there a particular reason why you want to keep a separate books and docs folder and remove the nim files?

The idea (my idea at least) is to have a books folder with the Nim files and a docs folder populated by github actions. Hence why docs should mirror books so we can nicely organize the books folder and have our generated documentation the way we set it up.

Note that nimib will automatically register as "home" directory the docs directory see https://github.com/pietroppeter/nimib#interaction-with-filesystem This is the reason why you see ../books/basic_plotting.nim at the top.

I was wondering about that. Yeah we'll need to find a solutions to that. Github pages use either docs folder or / root folder so we don't have a lot of choice here.

pietroppeter commented 3 years ago

I think the index.html should reflect the table of content - which means we have to solve the table of content issue first

yes, the simple index.nim in nblog is just to provide an easy autogenerated access to an evoving list of file. Table of contents (possibly in a sidebar, but it should be responsive) is definitely in scope in nimib, but I am a bit slow on implementing stuff.

The idea (my idea at least) is to have a books folder with the Nim files and a docs folder populated by github actions. Hence why docs should mirror books so we can nicely organize the books folder and have our generated documentation the way we set it up.

If you end up keeping the same structure in both books and docs, I do not see the advantage of keeping them separated (yes, you will end up exposing the nim files in docs folder, but I do not see harm there).

I was wondering about that. Yeah we'll need to find a solutions to that. Github pages use either docs folder or / root folder so we don't have a lot of choice here.

yes, the default home directory for nimib is done on purpose to replicate the two alternatives for github pages. The idea is that the two common structure for "nimib" projects are:

a structure like in nimib repo, where nimib is used to provide to documentation to a nim package, thus you use a docs folder
a structure like in nblog, where the repo itself is a documentation repo and I do not use a docs folder (and github pages points to root folder). I think this could be the same structure that could be used for this repo.

If anyway you want to proceed keeping the separate books and docs folder, there is a way to customize stuff automatically for a bunch of documents. I use it in adventofnim to have all documents use DarkMode. You can use it like this:

add a nbPostInit.nim file in books folder that contains something like (it should work but have not tried): nbHomeDir = nbHomeDir / "../docs".RelativeDir
add a nim.cfg or config.nims (in books folder) which adds -d:nimibCustomPostInit (this will include your nbPostInit at the end of nbInit) and --path:"." (which I think it is needed for the include to find the file).
if you also add to your nbPostInit.nim something like setCurrentDir nbHomeDir then the html and images generated by the nim files will also end up in the docs folder (and this might end up removing your dependence from rsync).

pietroppeter commented 3 years ago

mmh, wait, the above is not correct, the nbHomeDir is already the docs folder! Aggh, I will have to think a bit more about it...

HugoGranstrom commented 3 years ago

@Clonkk Great! 😄 I think rsync is ok to be Linux only, it's mostly the CI that will do the compiling. And in the other few cases, WSL is a really good thing to have installed either way.

Generating index.html and a table of content should be the next step IMO.

Yes for sure, agreed!

Hi all, nice to see this coming along, I will try to help!

Howdy! Glad to see you here! 😃 Hopefully, some of the work we do here can help you in your development of Nimib as well.

The idea (my idea at least) is to have a books folder with the Nim files and a docs folder populated by github actions. Hence why docs should mirror books so we can nicely organize the books folder and have our generated documentation the way we set it up.

I don't really see the purpose of the two separate folders either. If we do everything in /docs directly, it shouldn't matter for Github Pages, right? The only problem would be if we wanted to clean it up of course, we wouldn't be able to just delete the folder but would have to selectively delete *.html files. But that could be a Nimble task as well. nimble clean or something.

Clonkk commented 3 years ago

The separate folder was to avoid exposing the Nim file in github pages.

It also make it obvious to distinguish generated files and source file and avoid mistake like "git add docs/" that would commit generated files.

HugoGranstrom commented 3 years ago

It also make it obvious to distinguish generated files and source file and avoid mistake like "git add docs/" that would commit generated files.

I don't see the exposing part as a problem, there's no harm in having them there. Perhaps a waste of space but idk. This on the other hand is a fair reason. An automatic cleaning script on every commit would solve the problem of the generated files though. And it may be a good idea nonetheless as we could have generated files in /book as well if one only wants to check how a single file looks. But idk how such a cleaning would take place, deleting everything that isn't *.nim or just stuff that is *.html|png. But that would cause problems if we sometime would want to commit one such file :/ One way would be to check if there exists a corresponding foo.nim file for a foo.html and then delete it, otherwise not.

pietroppeter commented 3 years ago

ok, I wrapped my head around the issue and the result is #8. see details there.

I agree that a workflow like this where you have nim files and html files in separated folder should be supported and that it should allow to avoid commit html by mistake.

HugoGranstrom commented 3 years ago

Back to the original question in this RFC: how do we want to structure this page? I see two options:

Topic/difficulty ie plotting/basics and fft/basics
Difficulty/topic it basics/plotting and basics/fft

I think option 1 makes more sense as one typically wants to study a specific topic rather than a specific difficulty. It also makes it easier to see what content is missing for each topic, for example if fft lacks and advanced difficulty.

One more thing regarding structure, if we go for option 1, will we make plotting/basics a section and have entries for each individual tutorial? Ie plotting/basics/ggplotnim and plotting/basics/nimplotly?

And one last thing, I really like the idea of having a FAQ/Snippets for each topic which just contains the bare examples of common usecases. The demo that we have up atm is an example of that. It just shows you how to make a line respectively a scatter plot using ggplotnim, nothing more, nothing less. Where would one put something like this in the structure (if we include it)? plotting/snippets or plotting/basics/snippets?

Clonkk commented 3 years ago

I would just classify content by topics / subtopics. I don't think the "difficulty" is relevant in the table of content. Each author can add a difficulty label on the page, as they see fit.

Where would one put something like this in the structure (if we include it)? plotting/snippets or plotting/basics/snippets

How about plotting/quick-start ?

HugoGranstrom commented 3 years ago

That is a simpler solution indeed 👍 The only risk with it is that we can get an inflation of subtopics when submitters don't seem to find one that theirs fit in. So perhaps having a misc subtopic for posts which would have been the lone post in a subtopic?

Quick start is good as well 😄 I suspect I will have to check the quick-start guides all to often myself. Can't for the life of me remember how to plot a line plot in ggplotnim 😂🙃

Clonkk commented 3 years ago

I thin kwe can close this with #14 . The scope should be clear nenough by now.

Re-open if you disagree :smile:

SciNim / getting-started

[RFC] What is the scope? #1

Basics

Advanced