Closed Olshansk closed 1 year ago
Main repo: https://github.com/profishional/pocket Wiki: https://github.com/profishional/pocket/wiki Wiki git: https://github.com/profishional/pocket.wiki.git
Context Starting the conversation to double check the direction. I copied the structure from the READMEs over with some formatting changes taken from RocksDB. All internal links of the READMEs have also been updated, Subject to a more careful review as we get closer to the final implementation.
Deliverables
Github Wiki is added: I cannot do this without access to "Settings" but the steps would be:
1. Enable wiki in the core repos setting
2. Clone the wiki git
3. Reset remote to pocket repo
4. Clone Wiki from Profishional account (https://github.com/profishional/pocket.wiki.git)
5. Copy pasta from profish to pocket wiki
6. Push Pocket wiki
more info: https://stackoverflow.com/questions/10642928/how-to-pull-request-a-wiki-page-on-github
Maintenance process: I would say making the wiki the main source of content that is updated, then simplifying and pointing to the wiki from all the previous READMEs if they are going to be preserved. This seems to be inline with RocksDB (unless I'm missing some automation script in their workflow)
Optional goal
I don't have access to this. Should I?
I copied the structure from the READMEs over with some formatting changes taken from RocksDB. All internal links of the READMEs have also been updated, Subject to a more careful review as we get closer to the final implementation.
👍
Github Wiki is added: I cannot do this without access to "Settings" but the steps would be:
- Enable wiki in the core repos setting
I just enabled it. Let me know if you can see & edit.
- Clone the wiki git
- Reset remote to pocket repo
- Clone Wiki from Profishional account (https://github.com/profishional/pocket.wiki.git)
- Copy pasta from profish to pocket wiki
- Push Pocket wiki more info: https://stackoverflow.com/questions/10642928/how-to-pull-request-a-wiki-page-on-github
Makes sense to me, but would like to review it first :)
Maintenance process: I would say making the wiki the main source of content that is updated, then simplifying and pointing to the wiki from all the previous READMEs if they are going to be preserved. This seems to be inline with RocksDB (unless I'm missing some automation script in their workflow)
Here I may be misaligned. Are you suggesting that we won't have individual READMEs at all anymore?
I don't have access to this. Should I?
My bad, not sure what I was copy pasted before. I have edited the initial comment. Here are the links reposted:
Main repo: https://github.com/profishional/pocket Wiki: https://github.com/profishional/pocket/wiki Wiki git: https://github.com/profishional/pocket.wiki.git
I just enabled it. Let me know if you can see & edit.
Yep I can edit now.
Makes sense to me, but would like to review it first :)
I can transfer everything across after review. Without the above process.
Here I may be misaligned. Are you suggesting that we won't have individual READMEs at all anymore?
Yep, or much shorter individual readmes which also point to the wiki. Thats atleast how it seemed like with RockesDB at a Glance but I would be curious the rough idea in mind?
Wiki git: profishional/pocket.wiki.git
This is still a dead link for me
I can transfer everything across after review. Without the above process.
So are Wiki's just free for all to modify?
Wiki: profishional/pocket/wiki
This is 🔥
Question:
We recently add, for example, this markdown file:
https://github.com/pokt-network/pocket/blob/main/shared/docs/flows/AppHash.md
What's the process of including it in the wiki too?
Wiki git: profishional/pocket.wiki.git This is still a dead link for me
This is the cloning link for the procedure described in the first comment.
So are Wiki's just free for all to modify?
You can limit it to contributors only in the settings. But surprisingly the rocksDB is open for anyone to edit, maybe by mistake because I don't think it creates a PR or a notification (jus a version history) if it's edited. So best to restrict it. reference: https://docs.github.com/en/communities/documenting-your-project-with-wikis/changing-access-permissions-for-wikis
Question: We recently add, for example, this markdown fil: https://github.com/pokt-network/pocket/blob/main/shared/docs/flows/AppHash.md What's the process of including it in the wiki too?
From what I see in RocksDB:
So I would make the AppHash.md we short i.e.:
This repo is for x.
This is how to run it - if needed.
It relates to y and z.
For more info to the the wiki page _here_.
The wiki page would technical specs and more context (if possible)
These are my guesses tho, with my understanding I'm not sure what the absolute best practice would be.
I noticed some errors in the naming and content menu since I did a lot of copy pasting! Fixed now but will review properly as we get closer to final state.
There was one thing I wanted to double check.
I do want to keep the fact that our READMEs are version controlled (and require reviews and such). Will this not be possible once we use a wiki?
I see. Yeah Github Wikis are pretty basic (maybe enhanced in the future), any edit is a commit to the main. You can see version history, compare and revert but thats all after the fact, unlike a PR process.
There are hacks to manage it e.g.: https://www.growingwiththeweb.com/2016/07/enabling-pull-requests-on-github-wikis.html TLDR:
https://nimblehq.co/blog/create-github-wiki-pull-request (more self contained) TLDR:
I would say the implementation for this might be out of scope for this issue 😅 but it raises a question; does it make sense to go this far for the wiki given the current size of the documentation? It's cool imo, but up to you if you think it's worth it.
Yeah GitHub Wikis are pretty basic (maybe enhanced in the future), any edit is a commit to the main. You can see version history, compare and revert but thats all after the fact, unlike a PR process.
Noted.
I would say the implementation for this might be out of scope for this issue 😅 but it raises a question; does it make sense to go this far for the wiki given the current size of the documentation?
Yes, even our own team is losing track of the READMEs we have, so the screenshot I posted above would be super helpful.
It's cool imo, but up to you if you think it's worth it.
I think it's more than cool. Provides immediate value & visibility into all the .md
files we have
nimblehq.co/blog/create-github-wiki-pull-request (more self contained)
Definitely out of scope, but will sync with @okdas to see if we can do this.
Going to get some input from the team on this as well (cc @andrewnguyen22 and @deblasis).
The tl;dr is that there's an example wiki here which makes it really easy to navigate and view our documentation. The major con is that the READMEs we have would no longer be review/version controlled or live next to the code. I think this is a big minus, for example, in the context of CHANGELOGs where it should live in the corresponding repo.
@profishional I recently came by Chia's wki which also shows how great it is to have all the documentation discoverable in one place and reaffirms that it's something we need.
Could you look into ways of having the wiki be auto-generated from the .md
files in the repo so we get the best of both worlds? I found this (https://therenegadecoder.com/code/how-to-automate-your-github-wiki/) after doing a quick search but haven't doven in deep.
@profishional I recently came by Chia's wki which also shows how great it is to have all the documentation discoverable in one place and reaffirms that it's something we need.
Nice, I'll have it in mind.
Could you look into ways of having the wiki be auto-generated from the
.md
files in the repo so we get the best of both worlds? I found this (https://therenegadecoder.com/code/how-to-automate-your-github-wiki/) after doing a quick search but haven't doven in deep.
I'll have a look to compare. What did you think about the other links I posted? Happy to help with this part of the documentation too, pretty comfortable with Python but have to touch up on CI process.
I'll have a look to compare. What did you think about the other links I posted?
I'm not really sure what you're asking for exactly, but I'll reference these three:
Main repo: profishional/pocket Wiki: profishional/pocket/wiki Wiki git: profishional/pocket.wiki.git
The second link is exactly what we need with the exception of having that wiki be auto-generated by .md
files that exist in the codebase. Is that something you could take on?
Python: https://therenegadecoder.com/code/how-to-automate-your-github-wiki/ Github Actions: https://nimblehq.co/blog/create-github-wiki-pull-request
I was curious if you saw anything wrong with the links I sent, because the python approach seems to complicate it abit - had a read but would have to look at why the script aspect is even needed. Is there something you like about the python approach? Do you want the .md
files to be where they are now but still get a wiki out of it?
Do you want the .md files to be where they are now but still get a wiki out of it?
Yea, that's the first reason.
The second reason is that I was hoping for us to continue enforce a code review process to update READMEs.
Is the second piece possible without the automation approach?
I would say: Github Workflow solution (and the like) are: probably no to 1 and yes to 2 Python script: probably yes to 1 and yes to 2.
So I'm just wondering if it's worth scripting just to satisfy 1.
But I think I'll just go ahead and implement both on my fork for comparison, it'll make it easier to decide then. This would take me approx. 2 weeks probably based on current priorities.
@profishional Consider it a low priority and I don't want (2) to be too big of a time investment. It's like a "very nice to have" that I think we'll end up having sooner or later.
@profishional I was looking at https://github.com/onflow/flow-go/find/master, and realized that if I search for README.md
, there are lots of great docs that come up but they are hard to discover. Example:
This is the situation we're in right now, and is an example of what I'm hoping we can avoid, while also keeping the docs next to the soruce.
@profishional Wanted to check in to see if you have any updates on this?
No, sorry. On it this week!
Ok apologies on the delay @Olshansk. But with a full day on it here up the final updates:
Github Wiki is a separate repo within a main repo e.g. Main: github.com/.../pocket.git
VS Wiki: github.com/.../pocket.wiki.git
.
But the wiki does not require pull requests to edit. Also, we want to only maintain the docs in the main repo which are then converted into a wiki page.
Repo: https://github.com/profishional/pocket
Wiki: https://github.com/profishional/pocket/wiki
EDIT: pushed files in tools/
that the py script generates as an example, since they are not stored through the Github Actions script.
find | grep
line that we could use to derive a wiki structure. Using the ./README.md
structure the paths of the files are mapped in various way to create a sidebar menu and wiki files to update.Pre-requisites: Create an account level token with correct permissions and create a repo level secret.
Github Actions - file
find | grep
to get readme and changelog file pathsPython Script - file
README.md
structure and pattern in the file paths../tools
that the command can read instead. This will determine what is in the wiki.
find . -name "README.md" -o -name "CHANGELOG.md" | grep -v -e "vendor" -e "prototype" -e "SUMMARY.md" -e "TASTE.md" > ./tools/output.txt
./docs
for sake of testing, but it needs to be related to the list from above. Help around Github Actions on:
syntax would be appreciated../README.md
has to be changed.doc
folder naming more consistent ./app/client/cli/doc/README.md
vs ./runtime/docs/README.md
Besides improving the python script and Github Action a bit more (which I would like some help on as well), I won't be able to make it as robust as this solution. It would take decent amount of time that I don't have currently.
@profishional Appreciate all the work you've put into this and the detailed play-by-play and instructions. I think it really sets it up for someone else to pick up the task and take it over the finish line.
CCing @b3nnnp to see if you potentially know someone who could pick this up?
Update: synced with @profishional and we decided that as a next step, I'll do a detailed review of what we have to see if it provides enough guidance to get it over the finish line. If it results in too much work, we'll open it up to someone else.
@profishional Let me know if I missed anything.
But the wiki does not require pull requests to edit. Also, we want to only maintain the docs in the main repo which are then converted into a wiki page.
Got it, thanks for the explanation.
Pre-requisites: Create an account level token with correct permissions and create a repo level secret.
ACK. Action item on @Olshansk once the scripts are ready.
Better definition of excluded docs. Maybe a file in ./tools that the command can read instead. This will determine what is in the wiki.
I think this is good enough.
'find . -name "*.md" | grep -v -e ".git" -e ".vendor""'
Note: that if you have SUMMARY
and prototype
in your directory, you likely have a very old version of the v1 repo.
Better triggering of the wiki update. I just have it as any changes to ./docs for sake of testing, but it needs to be related to the list from above. Help around Github Actions on: syntax would be appreciated.
Why not just do it on all markdown changes?
on:
push:
branches:
- main
paths:
- '**/*.md'
Improving the READMEs because they are a 1:1 for the wiki, No changes can be made beside copy/pasting them.
Let's just do a 1:1 for now.
Reviewing the py script
See the code below. I'm not guaranteeing its fully functional but just where my mind was at. We should go with this approach and make the file name titles semantic in the actual source code.
Reviewing the github actions
Can you verify if we can remove is_github_workflow
in the python script and just do everything in python? This will simplify the github action
consider file diff before generating wiki files
No need to overcomplicate. Any markdown changes on push to main should retrigger this.
@profishional Do you think this is enough info / guidance to take this over the finish line?
import os
from collections import defaultdict
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, List
WIKI_DIR = "tools/wiki"
# TODO: Consider using enums?
HOME = "Home"
GUIDE = "Guides"
ARCHITECTURE = "Architecture"
CHANGELOGS = "Changelogs"
CLI = "CLI"
@dataclass
class DocumentationFile:
file_name: str
display_name: str
path: Path
def get_all_raw_markdown_file_paths(is_github_workflow: bool) -> List[str]:
if not is_github_workflow:
return os.popen(
'find . -name "*.md" | grep -v -e ".git" -e ".vendor"'
).readlines()
with open("tools/output.txt") as file:
return file.readlines
def get_all_markdown_file_paths(is_github_workflow: bool) -> List[Path]:
for raw_path in get_all_raw_markdown_file_paths(is_github_workflow):
path = Path(raw_path.strip())
if path.is_file():
yield path
def get_display_name(category: str, filename: str) -> str:
filename = filename.split(".")[0]
file_display_name = " ".join(
[s.title() for s in filename.replace("_", " ").split(" ")]
)
if category != HOME:
return f"{category} - {file_display_name}"
return file_display_name
def categorize_paths(is_github_workflow: bool) -> Dict[str, List[DocumentationFile]]:
sidebar = defaultdict(list)
for path in get_all_markdown_file_paths(is_github_workflow):
filename = os.path.basename(path)
dirname = os.path.dirname(path)
top_level_dirname = dirname.split("/")[0]
category = None
if filename == "CHANGELOG.md":
category = CHANGELOGS
elif "cli" in dirname:
category = CLI
elif top_level_dirname == "docs":
category = GUIDE
elif len(dirname) == 0 or top_level_dirname == ".":
category = HOME
else:
category = top_level_dirname.title()
# display_name = get_display_name(category, os.path.splitext(filename)[0])
display_name = get_display_name(category, filename)
sidebar[category].append(DocumentationFile(filename, display_name, path))
return sidebar
def write_sidebar_file(sidebar: Dict[str, List[DocumentationFile]]) -> None:
sidebar_format = "'''Contents'''\n"
sidebar_format += "* [[Home | Home]]\n"
print(sidebar)
for category, doc_files in sidebar.items():
sidebar_format += f"* '''{category}'''\n"
for doc_file in doc_files:
sidebar_format += (
f"** [[ {doc_file.file_name} | {doc_file.display_name} ]]\n"
)
with open(f"{WIKI_DIR}/_Sidebar.mediawiki", "w") as f:
f.write(sidebar_format)
def write_wiki_pages(sidebar: Dict[str, List[DocumentationFile]]) -> None:
for _, doc_files in sidebar.items():
for doc_file in doc_files:
with open(doc_file.path) as source:
target = f"{WIKI_DIR}/{doc_file.file_name}"
with open(target, "w") as dest:
dest.write(source.read())
if __name__ == "__main__":
is_github_workflow = False
sidebar_category_dict = categorize_paths(is_github_workflow)
for category, items in sidebar_category_dict.items():
for i in items:
print(i.display_name)
os.makedirs(WIKI_DIR, exist_ok=True)
write_sidebar_file(sidebar_category_dict)
write_wiki_pages(sidebar_category_dict)
'find . -name "*.md" | grep -v -e ".git" -e ".vendor""'
Ok I'll go with that.
Why not just do it on all markdown changes?
paths: - '**/*.md'
Yep, great! that's perfect.
See the code below. I'm not guaranteeing its fully functional but just where my mind was at. We should go with this approach and make the file name titles semantic in the actual source code.
Can you verify if we can remove
is_github_workflow
in the python script and just do everything in python? This will simplify the github action
alright I am going to try to make this work.
@profishional Do you think this is enough info / guidance to take this over the finish line?
Yep definitely I think this help a lot.
Quick update;
The mapping wasn't working properly so with only changing the variable in the current mapping flow we get the below. To make Changelog it own category I think I have to change it a bit more. wiki
Category
- Changelog
- Readme
- Readme
- ...
Category
- Changelog
- ...
Maybe we should fully define what the structure of the menu be at this point. Especially since we have collisions, for example if we only take the top level folder persistence
the docs
and indexer
readmes clash.
├── persistence/
│ ├── CHANGELOG.md
│ ├── docs/
│ │ ├── CHANGELOG.md
│ │ └── README.md
│ └── indexer/
│ └── doc/
│ └── README.md
My deafault would be trying to include the subcategory in the wiki file name/display name too. Is there any other preferences, maybe subcategory?
Also how much of the Github Actions do we want to remove, the easiest is to pass the params to the same script (fixing remaining bug on the push step). Do we want to remove the whole Action file?
on:
...
env:
USER_TOKEN: ${{ secrets.WIKI_SYNC_SECRET }}
...
jobs:
publish_docs_to_wiki:
run: |
python ./tools/wiki_sync.py -s $USER_TOKEN etc ...
The mapping wasn't working properly so with only changing the variable in the current mapping flow we get the below. To make Changelog it own category I think I have to change it a bit more
I just want to re-iterate one of my previous points:
I'm not guaranteeing its fully functional but just where my mind was a
I was just trying to clean up the code a bit but not provide a full solution. Do you need me to update the python file so the mapping works 100% of the time?
Maybe we should fully define what the structure of the menu be at this point. Especially since we have collisions, for example if we only take the top level folder persistence the docs and indexer readmes clash
Again, to reiterate one of my previous points:
We should go with this approach and make the file name titles semantic in the actual source code.
I think the rules we will find ourselves in can get very complex:
I would much rather we modify the source code, keep it simpler without maintaining an arbitrary list of rules, and easier to follow. For example, an idea I had is including a comment at the bottom of each markdown like so:
<-- GITHUB_WIKI: TITLE wiki/category/subcategory/path -->
This way it's very simple, very explicit, and we can determine how the wiki is updated based on comments in the markdown in the source code. It won't have to follow subdirectory rules.
Wdyt?
Also how much of the Github Actions do we want to remove, the easiest is to pass the params to the same script (fixing remaining bug on the push step). Do we want to remove the whole Action file?
Less code ==> better
Let's remove as much of it as we can as long as the github wiki still autogenerates whenever changes are merged to main.
I was just trying to clean up the code a bit but not provide a full solution. Do you need me to update the python file so the mapping works 100% of the time?
Haha no I understand, I didn't expect it to be fully working. I think it's good compared to mine so I didn't want to change a whole lot, but looking at the nuances of the repo I figured there are design choices that we would like to talk about.
I would much rather we modify the source code, keep it simpler without maintaining an arbitrary list of rules, and easier to follow. For example, an idea I had is including a comment at the bottom of each markdown like so:
<-- GITHUB_WIKI: TITLE wiki/category/subcategory/path -->
This way it's very simple, very explicit, and we can determine how the wiki is updated based on comments in the markdown in the source code. It won't have to follow subdirectory rules. Wdyt?
I think that is the easier more manual path, to avoid a general solution for all formats in the script. I wasn't sure if you'd be ok with the coupling; as in preferred the script to figure it out.
So that I understand, for the persistence
example above it would be:
<-- GITHUB_WIKI: Persistence Indexer wiki/persistence/indexer/readme.md
Which would become:
Persistence
- Indexer
- Readme
Less code ==> better Let's remove as much of it as we can as long as the github wiki still auto generated whenever changes are merged to main.
Ok, Ill try to remove the env
part too without importing new dependencies.
Not Quite (not I'm changing things per your suggestion).
In this file:
./persistence/indexer/doc/README.md
You would append the following to the bottom:
<--! GITHUB_WIKI: wiki/persistence/indexer/specs -->
Which would translate to the following:
Persistence
- Indexer
- Specs
@profishional Do you think this is enough context to get this over the finish line?
I think so. I'll have a proper go at it tomorrow by assuming that line is at the end of the reamdes and changelogs.
Ok I think I might be misunderstanding the <---! GITHUB_WIKI --->
solution. I'm, just going to think about the mapping from scratch again before sending through better questions.
Also, the github actions has been minimised and is working. So functionality wise just getting the mapping thing to work.
Ok I think I might be misunderstanding the
Could you elaborate what you're not understanding?
Here's the rationale:
- Directory structure in the core repo: complex (i.e. hard to parse), dynamic (files move around), non-deterministic (we're not going to enforce a naming structure for the sake of the wiki)
A file located under /root/module1/dir1/dir2/dir3
with the comment <--! GITHUB_WIKI: wiki/persistence/indexer/specs -->
will map to the following in the wiki:
Persistence
- Indexer
Specs
A file located under /root/module2/dir7/dir8/dir9
with the comment <--! GITHUB_WIKI: wiki/persistence/indexer/interface -->
will map to the following in the wiki:
Persistence
- Indexer
Interface
Please let me know if that clears things up.
Ok so e.g. a person will manually go through all the relevant md
files (based on find | grep
command) and add a comment?
What is the workflow for changing these comments? Do people have to check locally what the changes will do to the wiki before a merge?
Will there be rules for the structure in the comment e.g. avoiding creation of a wiki menu with 5 nested subcategories?
How will the timeline of those comments matter for closing this issue?
Would it be possible force that comment to be the last line in a md
file? (not necessary, but would make it easier)
Edit; I have also just realised, the internal links of the md
files need to be changed, if we want them to be pointing to wiki files (something I did manually for the first attempt). I'll come up with a potential solution.
Ok so e.g. a person will manually go through all the relevant md files (based on find | grepcommand) and add a comment?
Correct.
What is the workflow for changing these comments? Do people have to check locally what the changes will do to the wiki before a merge?
The github action in this PR will trigger after changes are merged to main.
Will there be rules for the structure in the comment e.g. avoiding creation of a wiki menu with 5 nested subcategories?
When we hit issues/problems, I believe the answer will be yes, but let's cross that bridge when we get there.
Edit; I have also just realised, the internal links of the md files need to be changed, if we want them to be pointing to wiki files (something I did manually for the first attempt). I'll come up with a potential solution.
I don't think we need to deal with this. Let's just finish something that's good enough.
How will the timeline of those comments matter for closing this issue?
Finish writing python script that parses an expected <--! GITHUB_WIKI: wiki/persistence/indexer/specs -->
line and copies the file appropriately.
Finish writing the github action that copies the file from the source repo to the readme when PRs are merged to main
Submit a PR in the source repo that includes the following:
3.1 The python script
3.2. The github action
3.3. The addition of the GITHUB_WIKI
comment in markdown files matching this: find . -name "*.md" | grep -v -e "vendor" -e "app" | wc -l
3.4 Update the documentation in docs/development/README.md
explaining how this works, why it was implemented and how new markdown files should be configured if the author wants them to be part of the readme
3.5 Update the main README
in the root repo so it points to the wiki
During the code review process:
4.1 If there are changes that need to made to the python script -> we'll leave a comment
4.2 If there are changes that need to made to the github action -> we'll leave a comment
4.3 If the directory structure you came up with in the comments in GITHUB_WIKI
is not correct or should be changed -> we'll leave a comment
If we need to creat issues for followup tasks (e.g. making the links also point to wiki links instead of the core repo, we can do that in a followup task)
If we need to change the directory structures in the future, anyone can do it.
@profishional Does that plan of action make sense?
Ok great, got it. That was helpful, thank you.
@b3nnnp This ticket seemed to have gotten closed by error.
The PR for this is up for review at #419, but it'll probably require a few rounds of back & forths & review.
@profishional I'll aim to take a look at it this week and will follow up directly there. We can close out this ticket once its merged.
Ok @Olshansk apologies.. it was due to a status change in dework but will not change any developer bounties in future
Objective
Create a Github Wiki for Pocket V1
Origin Document
The main page for https://github.com/pokt-network/pocket contains links to several READMEs spread through the network.
There are a few problems with the current documentation:
Starting point
A good starting point would be to view the structure of https://github.com/pokt-network/pocket.
In order to find all the existing READMEs, one could execute
find . -name "README.md" | grep -v "vendor"
which outputs the following:References
A good example or reference of this is the RockdDB Wiki Page, which currently looks like this:
Goals / Deliverables
Optional Goals
Non-goals
Creator: @Olshansk Co-Owners: ???