Closed Keith-CY closed 8 months ago
Now there is an issue regarding the design of the label scheme.
First, let's talk about the prerequisites: the GitHub REST API
uses an AND query
for the labels
parameter instead of an OR query
. It may be possible to achieve an OR query
with the GitHub GraphQL API
, but it would increase complexity.
Therefore, if we need to iterate over multiple labels
, it would result in a higher number of API requests.
By default, with a token, GitHub allows 5000 requests per hour.
There are several design options for the labels:
troubleshoot
), and then 1 to n additional labels to determine the menu hierarchy. For example, adding a label like guide
would place the issue under the "New User Guide" menu, and adding create wallet
would indicate that it belongs to the "Create Wallet" submenu under the guide. If we add the label faq
, the issue can also appear under the "FAQ" submenu, specifically under "Create Wallet" (if the FAQ has such a submenu).guide - create wallet
. This way, a single label can express the complete menu hierarchy. The drawback is that if an issue belongs to multiple top-level menus, an additional label is needed. For example, guide - create wallet, faq - create wallet
.guide
and faq
and only use labels for n-level menus like create wallet
. The downside is that two top-level menus cannot have submenus with the same name, and it may reduce readability.troubleshoot
label. The drawback is that it would consume a higher number of API requests.guide
, faq
, or announcement
), followed by 1 to n labels to determine the menu hierarchy.Personally, I lean towards option 2, which is troubleshoot, guide - create wallet
.
@Danie0918 @Keith-CY Do you have any other suggestions or recommendations?
I have questions about the prerequisite
First, let's talk about the prerequisites: the GitHub REST API uses an AND query for the labels parameter instead of an OR query. It may be possible to achieve an OR query with the GitHub GraphQL API, but it would increase complexity. Therefore, if we need to iterate over multiple labels, it would result in a higher number of API requests. By default, with a token, GitHub allows 5000 requests per hour.
Why not query various labels separately to collect all related issues? Suppose we have 50 labels(it would be too many to manage, in my opinion), we can request 50 times to construct a catalog.
Why not query various labels separately to collect all related issues? Suppose we have 50 labels(it would be too many to manage, in my opinion), we can request 50 times to construct a catalog.
This is a method similar to option 4, where the number of requests made during a full collection is related to the number of labels and issues (with a maximum of 100 issues returned per request). We may need to perform a full collection in various scenarios, such as multiple runs during a deployment process (e.g., getStaticProps and getStaticPaths for different pages, which could potentially be optimized), or when triggering index uploads.
On the other hand, if we base the solution on the "troubleshoot" label, the number of requests made during a full collection will only be related to the number of issues. I expect it to be around 1 to 3 requests.
By adopting this approach, we can have a sufficient number of remaining requests to prevent exceeding GitHub's request limits during frequent deployments.
This is a method similar to option 4, where the number of requests made during a full collection is related to the number of labels and issues (with a maximum of 100 issues returned per request). We may need to perform a full collection in various scenarios, such as multiple runs during a deployment process (e.g., getStaticProps and getStaticPaths for different pages, which could potentially be optimized), or when triggering index uploads.
There is an important point that I forgot to mention: the getStaticProps
of the issues/[id].page.tsx
page triggers requests for issue details corresponding to the number of issues. This consumes a significant number of requests, so I am trying to minimize the usage of requests.
Why not query various labels separately to collect all related issues? Suppose we have 50 labels(it would be too many to manage, in my opinion), we can request 50 times to construct a catalog.
This is a method similar to option 4, where the number of requests made during a full collection is related to the number of labels and issues (with a maximum of 100 issues returned per request). We may need to perform a full collection in various scenarios, such as multiple runs during a deployment process (e.g., getStaticProps and getStaticPaths for different pages, which could potentially be optimized), or when triggering index uploads.
On the other hand, if we base the solution on the "troubleshoot" label, the number of requests made during a full collection will only be related to the number of issues. I expect it to be around 1 to 3 requests.
By adopting this approach, we can have a sufficient number of remaining requests to prevent exceeding GitHub's request limits during frequent deployments.
100 issues/topics are quite enough in our case, I believe.
And suppose the issue list is fixed, the count of requests to cover the catalog won't fluctuate too much if the labels are not overlapped too many. It's similar to
1 label => 100 issues
and
1 label => 10 issues
1 label => 20 issues
1 label => 30 issues
1 label => 40 issues
1 request to 4 requests
Besides, the catalog could be cached to avoid requesting issue list repeatly
With the cache, we can only update the issue list on label change event and update the catalog without scraping issue list.
This is a method similar to option 4, where the number of requests made during a full collection is related to the number of labels and issues (with a maximum of 100 issues returned per request). We may need to perform a full collection in various scenarios, such as multiple runs during a deployment process (e.g., getStaticProps and getStaticPaths for different pages, which could potentially be optimized), or when triggering index uploads.
There is an important point that I forgot to mention: the
getStaticProps
of theissues/[id].page.tsx
page triggers requests for issue details corresponding to the number of issues. This consumes a significant number of requests, so I am trying to minimize the usage of requests.
We won't have so many articles.
And the requests could be reduced by Incremental Static Generation(https://nextjs.org/docs/pages/building-your-application/data-fetching/incremental-static-regeneration)
We can use algolia to trigger page generation progressively
IMO, we won't have too many issues to be collected, and full-text scraping is enough
And suppose the issue list is fixed, the count of requests to cover the catalog won't fluctuate too much if the labels are not overlapped too many. It's similar to
1 label => 100 issues and 1 label => 10 issues 1 label => 20 issues 1 label => 30 issues 1 label => 40 issues
1 request to 4 requests
Considering the possibility of having multiple submenus, there would likely be more than just a few in number.
100 issues/topics are quite enough in our case, I believe. IMO, we won't have too many issues to be collected, and full-text scraping is enough.
I agree with your assessment. That's exactly why we need to consider the frequency of requests. If we use webhook + storage, we don't need to worry about this issue since it would generate only a minimal number of requests.
With the cache, we can only update the issue list on label change event and update the catalog without scraping issue list.
My plan is to use storage only when our number of issues becomes large enough to necessitate it.
And the requests could be reduced by Incremental Static Generation(https://nextjs.org/docs/pages/building-your-application/data-fetching/incremental-static-regeneration)
I have implemented ISR, but considering that this project will be deployed together with the official website project, we still need to consider the possibility of request limitations during frequent deployments.
100 issues/topics are quite enough in our case, I believe. IMO, we won't have too many issues to be collected, and full-text scraping is enough.
I agree with your assessment. That's exactly why we need to consider the frequency of requests. If we use webhook + storage, we don't need to worry about this issue since it would generate only a minimal number of requests.
With the cache, we can only update the issue list on label change event and update the catalog without scraping issue list.
My plan is to use storage only when our number of issues becomes large enough to necessitate it.
And the requests could be reduced by Incremental Static Generation(nextjs.org/docs/pages/building-your-application/data-fetching/incremental-static-regeneration)
I have implemented ISR, but considering that this project will be deployed together with the official website project, we still need to consider the possibility of request limitations during frequent deployments.
I would suggest starting from the simple assumption "we have limited articles to update", and consider the rate limit when requests are rejected for it.
I would suggest starting from the simple assumption "we have limited articles to update", and consider the rate limit when requests are rejected for it.
Alright, let's consider this issue once we encounter request rejections. I initially raised this concern because I noticed that there would be at least 8 submenus according to the design drafts. If we calculate based on 8 labels + 50 issues, it would result in approximately 100 requests for a single deployment. Once the number increases, there is a probability of triggering request rejections during frequent deployments on the official website.
@Keith-CY So we use a single label like guide - create wallet
?
I would suggest starting from the simple assumption "we have limited articles to update", and consider the rate limit when requests are rejected for it.
Alright, let's consider this issue once we encounter request rejections. I initially raised this concern because I noticed that there would be at least 8 submenus according to the design drafts. If we calculate based on 8 labels + 50 issues, it would result in approximately 100 requests for a single deployment. Once the number increases, there is a probability of triggering request rejections during frequent deployments on the official website.
@Keith-CY So we use a single label like
guide - create wallet
?
You've mentioned
the GitHub REST API uses an
AND query
for the labels parameter instead of anOR query
.
What if label this article with guide
and create wallet
I would suggest starting from the simple assumption "we have limited articles to update", and consider the rate limit when requests are rejected for it.
Alright, let's consider this issue once we encounter request rejections. I initially raised this concern because I noticed that there would be at least 8 submenus according to the design drafts. If we calculate based on 8 labels + 50 issues, it would result in approximately 100 requests for a single deployment. Once the number increases, there is a probability of triggering request rejections during frequent deployments on the official website. @Keith-CY So we use a single label like
guide - create wallet
?You've mentioned
the GitHub REST API uses an
AND query
for the labels parameter instead of anOR query
.What if label this article with
guide
andcreate wallet
In fact, I'll prefer to move create wallet
into title
the GitHub REST API uses an
AND query
for the labels parameter instead of anOR query
.What if label this article with
guide
andcreate wallet
If we use the format guide, create wallet
, then I would prefer fetching the issues with the guide
label and categorizing them locally based on their labels. Otherwise, I feel that the guide
label would serve no meaningful purpose.
In fact, I'll prefer to move
create wallet
intotitle
In reality, "create wallet" is actually a submenu. Perhaps I should provide a different example.
the GitHub REST API uses an
AND query
for the labels parameter instead of anOR query
.What if label this article with
guide
andcreate wallet
If we use the format
guide, create wallet
, then I would prefer fetching the issues with theguide
label and categorizing them locally based on their labels. Otherwise, I feel that theguide
label would serve no meaningful purpose.In fact, I'll prefer to move
create wallet
intotitle
In reality, "create wallet" is actually a submenu. Perhaps I should provide a different example.
Clear, I think gathering by guide
and categorizing them based on create wallet
is good for us.
Currently, a simple demo has been implemented: https://neuron-troubleshooting.vercel.app/ Repo: https://github.com/Magickbase/neuron-troubleshooting
Next, it is necessary to improve the data source (issues in https://github.com/nervosnetwork/neuron) for troubleshooting and assign them specific labels. This will require the participation of @Keith-CY and @Danie0918. This is in order to apply for https://docsearch.algolia.com/, which provides a comprehensive documentation search solution (frontend implementation + indexing crawler implementation). If the application is not approved, we will need to either use Algolia Pro's crawler feature, or deploy our own indexing service using Docker based on the instructions in https://docsearch.algolia.com/docs/legacy/run-your-own/, or attempt to deploy https://github.com/algolia/docsearch-scraper on Vercel to run periodically (not sure if it is feasible).
This is the currently proposed menu and label structure: https://github.com/Magickbase/neuron-troubleshooting/blob/a170da4eb3758f4b04b19b99cf6fecfa81ce508b/src/utils/posts.ts#L12-L39
I have added 3 main tags and 2 sub tags under FAQ, and marked 3 issues.If you need more tags or tagged issues, please contact me.
https://github.com/nervosnetwork/neuron/issues/2384 https://github.com/nervosnetwork/neuron/issues/2388 https://github.com/nervosnetwork/neuron/issues/2557
How is the troubleshooting page going? It was expected to be linked in nervos official website
Ref: https://github.com/Magickbase/nervos-official-website/issues/237
The latest progress has been pushed to the repository and deployed to https://neuron-troubleshooting.vercel.app/.
Since we have not received a reply from the Docsearch team, I plan to refer to https://github.com/algolia/docsearch-scraper to write a TypeScript version of the Docsearch indexer.
Alternatively, we could set up a machine to run Docsearch-scraper periodically. The advantage of implementing it in TypeScript is that we can avoid the crawler part and do not need an additional server. Instead, we can provide the markdown to HTML results directly during deployment to quickly create the index.
Since we have not received a reply from the Docsearch team, I plan to refer to algolia/docsearch-scraper to write a TypeScript version of the Docsearch indexer.
Alternatively, we could set up a machine to run Docsearch-scraper periodically. The advantage of implementing it in TypeScript is that we can avoid the crawler part and do not need an additional server. Instead, we can provide the markdown to HTML results directly during deployment to quickly create the index.
What's the problem with scraping the page? I remember that algolia will scan pages daily to update the database by setting an API token
What's the problem with scraping the page? I remember that algolia will scan pages daily to update the database by setting an API token
The DocSearch team needs to approve our application before they can provide us with this service. Otherwise, we will need to deploy it ourselves or implement the indexer on our own.
BTW, their application processing procedure is not public and there is no way to check the progress except for waiting for their email reply.
Perhaps we can consider using a self-deployed Docker version for a few weeks during this waiting period, and if it still hasn't been approved, then we can consider implementing it in TypeScript?
The Docsearch application has been approved and is currently integrated.
I have implemented the mobile version of the article details page and now need to implement the mobile version of other pages as well.
Here are 2 question:
Here are 2 question:
- This version is only available in English.
- The search here on the home page has changed from the previous one and does not use the UI interaction.
Yes, this may be an issues that would be dealt with last.
Because i18n can try to translate all content at once with some tools at the end, and the search UI may take more time and have lower returns, it can also be postponed.
A Chinese version of the Beginner's Guide has been completed, please check it out.The content of the newbie guide is categorized into three categories: About Neuron, Usage Tutorials, and Advanced Functions.
If there are no problems, I will upload the English version of the newissue later for the service to grab.
URL:[Google Docs]
@Keith-CY @yanguoyu
A Chinese version of the Beginner's Guide has been completed, please check it out.The content of the newbie guide is categorized into three categories: About Neuron, Usage Tutorials, and Advanced Functions.
If there are no problems, I will upload the English version of the newissue later for the service to grab.
URL:[Google Docs]
@Keith-CY @yanguoyu
Please authorize us to add comments
A Chinese version of the Beginner's Guide has been completed, please check it out.The content of the newbie guide is categorized into three categories: About Neuron, Usage Tutorials, and Advanced Functions. If there are no problems, I will upload the English version of the newissue later for the service to grab. URL:[Google Docs] @Keith-CY @yanguoyu
Please authorize us to add comments
Already adapted.
Basically it's completed for now. There are still some known minor issues:
i18n now will (in console) report hydration errors in non-English cases.
The clickable element effects did not consider multi-line text cases, which may underperform when there are multiple lines of text. Such cases are rare but could happen in the changelog page.
The i18n rules for help center articles are not implemented yet. It should be configured based on labels and can be done when needed.
The search box effects are not fully implemented according to the design draft.
I think these issues should not block the launch and can be fixed afterwards.
The FAQ document has been polished FAQ. @Danie0918 @WhiteMinds @Keith-CY .
@Sven-TBD
The initial version of the menu hierarchy was as follows: Top-level menu Guide, with second-level categories Create wallet, Backup wallet, Transfer and receive. Top-level menu FAQ, with second-level categories Sync, Transaction, Migration, CKBNode. Top-level menu Announcements, with second-level categories Change log, Develop guide.
Now it seems that a new Document is to be added under FAQ. Are there any other menu changes? For example, removing some, adding some.
Here'are the labels used in FAQ:
@WhiteMinds
Okay, currently the Migration
and CKBNode
have been removed from FAQ
, and Assets
, Safety
, Report
have been added.
Is this the latest testing website :https://neuron.magickbase.com/ ? Does neuron has entrance like this @WhiteMinds ?
Bug issues link:https://github.com/Magickbase/neuron-public-issues/issues/339
add a troubleshooting page as https://support.google.com/chrome/?p=help&ctx=menu#topic=7439538, better to be supported by https://www.algolia.com/
add an entry to the troubleshooting page as
A troubleshooting page is desired because many users don't visit GitHub. We've received many FAQs since we added
contact us
inmenu => help
, and many of them are pinned in the issue list. But users are not familiar with GitHub, so they still seek help from us directly. A troubleshooting page supported by algolia should ameliorate it.The page will be an entry of neuron in the future, including download and user manual