Part 1: doc restructure, introduce categories

[tomschr]

Based on the discussions from #1360, this PR contains the following changes for part 1:

• Introduce the categories "First steps", how-to guides (recipes), addendum, and references based on the Diátaxis doc framework
• Clean up the first page and move some resources into resources.md file.
• Add a high level overview on the first page. Tell the reader how the Lark documentation is organized.
• Minor changes in titles to adhere to the overall doc type

I've compiled it with make html but got some warnings. However, it seems, they exist in the master branch as well.

Let me know what you think, @erezsh.

Open questions

When I worked on this PR, I stumbled upon the following issues:

• Do we want a table of content in the first page although we have already one on the left sidebar? Should we keep the first page clean? Only difference is the depth of the TOC.
• Should we present the features on the first page instead of using the link?
• The doc is a mixture of RST and Markdown. Probably there is a reason for that ("we are get used to Markdown/don't like RST"?). Using different formats in one doc makes it hard to stay consistent. Both have a different set of syntax and features. What works with one doesn't work for the other. IMHO, this makes it also hard to contribute. But this is for a different PR (if any), but I would really recommend to use only one format.

Result

[MegaIng]

• TOC: Unsure. If it's just mostly the same as the left nav, it's pointless. If it has more depth (and this depth is actually useful. Doesn't look like it for the JSON tutorial for example), then I would say don't get rid of it. Also, can't then left nav get more depth and expand if you click on it?
• Features list: No opinion from me.
• With regard to markdown vs rst, check here: #640 . So in some sense it's mostly historic.

[tomschr]

• TOC: Unsure. If it's just mostly the same as the left nav, it's pointless. If it has more depth (and this depth is actually useful. Doesn't look like it for the JSON tutorial for example), then I would say don't get rid of it. Also, can't then left nav get more depth and expand if you click on it?

Exactly. I think we can configure Sphinx to show more content for the left sidebar/TOC. In that case, the TOC in the content would be useless and could be removed.

After I researched this issue, it can be done by adding the following lines:

diff --git i/docs/conf.py w/docs/conf.py
index aacf8bb..405af4d 100644
--- i/docs/conf.py
+++ w/docs/conf.py
@@ -102,7 +102,8 @@ html_theme = 'sphinx_rtd_theme'
 # documentation.
 #
 html_theme_options = {
-    'prev_next_buttons_location': 'both'
+    'prev_next_buttons_location': 'both',
+    'collapse_navigation': False,
 }

• With regard to markdown vs rst, check here: Please use Sphinx for documentation #640 . So in some sense it's mostly historic.

I see. As a first time contributor, I was puzzled. Usually, writers (and probably contributors as well) don't want to deal with two formats. It's just makes live harder as you have to switch context (MD <-> RST).

I've read the issue you gave me, but I'm still a bit confused. It's for historical reasons, okay. Are we forced to keep it? If not, can we use one format? Or are there any objections?

At this point, we have the following options:

1. Convert the .md files to .rst to make it consistent.
2. Stop here and merge the existing PR after all reviews are done. Make the conversion in another PR or keep it as is.

Apart from the above change, I realized there might be another question to consider. The current structure under docs/ is flat. With the current split into these four doc types, we maybe should reflect this change in the structure as well.

For example, we could create docs/references, docs/tutorials/ directories and move the files there. That would make contribution a bit clearer, hopefully. However, it would make the PR a bit bigger. But this idea is independent from the two options above and could be done also in another PR (or not, if you prefer the current structure).

It's your project, I'll give you just some ideas to think about. :slightly_smiling_face: If you think this would be beneficial, I'm willing to work on it.

What do you think?

[MegaIng]

I've read the issue you gave me, but I'm still a bit confused. It's for historical reasons, okay. Are we forced to keep it? If not, can we use one format? Or are there any objections?

We can convert the markdown files to rst, I have no objections (erezsh might). The point is more that we don't/didn't consider it worth the effort to convert them. But I don't think that should happen in this PR.

The current structure under docs/ is flat. With the current split into these four doc types, we maybe should reflect this change in the structure as well.

Sure. But IMO this should also be a separate PR to keep them all manageably small.

Let's wait for erezsh to add their opinion.

[tomschr]

Thanks MegaIng for your comments!

I agree with you. If we want to go this extra mile, I can do that in separate PRs. I'll wait for erezsh's opinion.

Thanks! :+1:

[tomschr]

I see no advantages in having two formats at the same time. Just some additional thoughts:

• Consistency is key. This is the overall parenthesis that holds it together.
• We should make the life of potential contributors easier, not harder.
• Having one format, we:
  • can use the full set of syntax. We do not have to deal with a subset or work with limitations of one format.
  • can get rid of some dependencies which are useless after the conversion.
  • looks consistent and clean
• Conversion is a matter of calling pandoc and fixing the leftovers (we have to do it once).

[tomschr]

@MegaIng / @erezsh Anything that I can do to make your life easier? Or anything that's missing? I know you are probably quite busy before the holiday season, but it would help me if I'm on the right track and if there is still interest to proceed further. :green_heart:

Thank you and have a great vacation.

[erezsh]

@tomschr Thanks! Sorry to keep you waiting. I've actually been on vacation these past 3 weeks, and might have a bit more free time in a week or two.

[tomschr]

Thank you! Hope the vacation was fantastic! :smile: No problem, have a good day.

[erezsh]

Hi @tomschr ,

I got to looking at your branch. There were some changes I liked, and some that I didn't understand. More importantly, looks like your changes from the docs. This is how it looks when I build your branch locally:

Also, looks like you intentionally removed all the links from the index page. I think that's wrong. It's arguable how much links should be there, but definitely there should be a few "call to actions" for the most common pages users would like.

[tomschr]

@erezsh Thank you for your time! I'm sorry, it seems I had accidentally committed my last experiment. That shouldn't be the case, I've corrected the TOC now. Could you check again?

[tomschr]

@erezsh Any additions, corrections, or suggestions from your side?

[erezsh]

Sorry, I've been busy. If it was as simple as just merging it, I would have already done so.

I will need to take the time to write my thoughts in detail, and explain what I think needs changing and why.

I will try to get to it soon!

[erezsh]

@tomschr Sorry again for the long wait. At an overview, I agree with these changes, but not with all of the choices you made.

A few points:

• Examples don't belong in "first steps". The way I see it, first steps should be reserved for tutorials, beginner guides, and installation instructions (if we had any). Maybe it should contain the "How to use Lark" section too. It definitely shouldn't contain a whole section called "Grammar Composition", which is an advanted topic, and definitely isn't important for beginners.

( I might be persuaded that it should contain the beginner examples, but I think examples should be together. Probably in their own separate section )

• I think "How to use Lark" is a better title than "Working with Lark"

• The portion you moved to "Resources" needs editing to adjust to the new structure

• Now that I think about it, "Parsers" section belongs in reference.

• Possibly "features" also belongs in the reference section.

[tomschr]

Thank you for your response!

After such a long time it's hard to remember all the details that I made and the reasons behind it. :laughing:

Examples don't belong in "first steps". The way I see it, first steps should be reserved for tutorials, beginner guides, and installation instructions (if we had any).

That is true, but as we don't have these guides yet I had to put at least something. Keep in mind, we are just migrating from one structure to a new one. It's an intermediate step, it's not the end. We can always adjust it later.

If you prefer to move these things, I can do that. But then we maybe end up with only one article under "First steps".

I think "How to use Lark" is a better title than "Working with Lark"

I did it on purpose, because "working" is a verb and means action. The problem with starting with "How to..." is, it propagates quite fast and you have other articles with very similar titles. That makes it harder for the reader to distinguish them. The difference in each title appears too late. In my work as a professional writer, I try to avoid starting with "How to..." if possible.

Now that I think about it, "Parsers" section belongs in reference. Possibly "features" also belongs in the reference section.

That shouldn't be a problem to move them. I'll do that later.

The portion you moved to "Resources" needs editing to adjust to the new structure.

I'm not sure I got that. Any examples?

Thanks for your time!

[erezsh]

No need to thank me! I'm glad you're trying to contribute to Lark, and like I said, I'm sorry it took so long.

After such a long time it's hard to remember all the details that I made and the reasons behind it

Now you get a glimpse of how I feel about docs I wrote 4 years ago :)

But then we maybe end up with only one article under "First steps".

That is acceptable. I think it's better than a mis-categorization, that might confuse a newcomer.

We can also include the "How to write a DSL" tutorial as an external link.

I try to avoid starting with "How to..." if possible.

Maybe there's a third option. Like "A guide to using Lark" or "best-practices for using Lark", etc.

I'm not sure I got that. Any examples?

Well, I wrote it as a "quick-links" section. Since you moved it as-is, it contains links to sections that are not really resources, like "philosophy", "features", and "examples". Also, some of these "resources" are either guides, or examples, or something in between, and would probably fit there better than to "resources".

[tomschr]

No need to thank me! I'm glad you're trying to contribute to Lark, and like I said, I'm sorry it took so long.

Haha, no problem. Now I had to say sorry as I wanted to work on it last week.

Now you get a glimpse of how I feel about docs I wrote 4 years ago :)

Haha, very true!

[... suggestions to move sections...]

This is done in commit c3470e1 now.

That is acceptable. I think it's better than a mis-categorization, that might confuse a newcomer.

Okay, no problem.

Maybe there's a third option. Like "A guide to using Lark" or "best-practices for using Lark", etc.

Hmn. :thinking: My idea was all about consistency: References and conceptual topics are in nominal style ("Grammar Reference", "JSON parser" etc.) whereas task-oriented topics uses verbs ("Contributing to Lark", "Working with Lark" etc.). Perhaps "Working" isn't the best verb and it can be improved.

Your suggestion "A guide to using Lark" or "Best-practices for using Lark" just adds more fluff to the title. :wink: From my point of view, it's just too wordy. :wink: Whether it's a guide or a section doesn't really matter to the user. It's an implementation detail. The readers want to know what is the topic about and they want to understand it fast. Anything that distracts the user reduces their understanding. :wink:

Just wanted to tell you my reasoning and why I think it matters. If you really want to go with your suggestion, I can do that change of course. :slightly_smiling_face:

I'm not sure I got that. Any examples?

Well, I wrote it as a "quick-links" section. Since you moved it as-is, it contains links to sections that are not really resources, like "philosophy", "features", and "examples". Also, some of these "resources" are either guides, or examples, or something in between, and would probably fit there better than to "resources".

Yes, I moved it "as-is" to reduce the impact of the changes. Remember, it's part 1. In this part I deal only with restructuring, maybe correcting titles here or there, but not touching the overall text. This could be done at a later point.

But perhaps I misunderstood you.

[erezsh]

My idea was all about consistency

I appreciate consistency, but I think clarity is even more important.

"Working with Lark" isn't a terrible title, but I feel that it doesn't communicate well what the content might be. If I was looking for best-practices, I might miss it if I was trying to scan the menu.

Anyway, I don't feel that strongly about it.

Yes, I moved it "as-is" to reduce the impact of the changes

It's good to keep the changes small. But I don't want the docs to end up looking wrong or broken, even if it's temporary.

If you're still confused, I can try to fix that page myself. But it needs doing.

[tomschr]

"Working with Lark" isn't a terrible title, but I feel that it doesn't communicate well what the content might be. If I was looking for best-practices, I might miss it if I was trying to scan the menu.

I see what you mean. But the titles work in combination with the guide title. I think we are talking about different things:

• Task-oriented/"recipes"/"first steps": more for beginners. Explains the topic more in a step-by-step style (if possible). Usually more task-based. Could be a "Working with Lark"
• "Best practices": more for medium users to experts. Explains the topic in a broader context. Could be something like "Optimizing your grammar" (hypothetical).

From what I see, I think these are different guides. If you have this distinction, you can focus on your titles. Beginner will be attracted by "First steps", more advanced users will gravitate the Best Practices or other guides.

But I don't want the docs to end up looking wrong or broken, even if it's temporary.

Sure, absolutely. :+1:

If you're still confused, I can try to fix that page myself. But it needs doing.

If it's not too much to ask for, please go ahead.