Open orbeckst opened 2 years ago
emailed support to ask how to link mdanalysis index to my dashboard and/or get access to the new(?) Crawler management interface to see(?) the index
Users will receive access to the new infrastructure on a random basis. Sorry, we are a bit early in the process right now to do bigger batches/deploy certain configs! You can read more here: https://docsearch.algolia.com/docs/migrating-from-legacy#migration-seems-to-have-started-but-i-dont-have-received-any-emails
created algolia account to have access to dashboard and possibly see index
We don't grant access to the dashboard but only the Analytics, sorry! (But it will be available in the new infra :D)
start_urls
and stop_urls
works as matching pattern/substring detection, so this URL for example https://www.mdanalysis.org/distopia/index.html
will be skipped because of this stop_urls
: https://www.mdanalysis.org/.*index.html$
.
I'd suggest you to adapt the stop_urls
to make sure we don't exclude URLs you'd potentially like to keep!
Below, the URLs matching distopia
:
We can see that some selectors don't match certain pages. To make it more specific, you can use selectors_key
There's was a typo in your config (sorry I didn't saw it), I've fixed it in: https://github.com/algolia/docsearch-configs/pull/4712
I installed the docsearch-scraper locally and I’m able to run it so I can now debug more easily.
Well... maybe not that simple:
$ ./docsearch run ../docsearch-configs/configs/mdanalysis.json
...
algoliasearch.exceptions.RequestException: Record quota exceeded. Change plan or delete records.
Nb hits: 10415
previous nb_hits: 85975
Will need to see how to work within these limitations.
I am now using a scraper with disabled index submission for testing, see https://github.com/orbeckst/docsearch-scraper/pull/1.
I am now using a scraper with disabled index submission for testing, see https://github.com/orbeckst/docsearch-scraper/pull/1.
That's a good idea! It would be nice to see it as an option indeed
Let me know if I can help you debug your issue
Many of the missing terms are due to broken sitemaps. Apparently, somewhere our Sphinx + GitHub actions based doc deployment changed and the sitemaps now contain a version information that is not actually present in the deployment URL. That's a problem on our end.
I'll leave this issue open for the moment.
PR #4751 addresses some of the problems and we also fixed sitemaps. The PR has some more comments on what still seems to be missing, including the output from the scraper (for 0 record pages). Any insights why we're still missing content would be appreciated. Thanks!
As mentioned in PR #4751 there are still a number of "0 records" pages, namely under
From https://github.com/algolia/docsearch-configs/pull/4751#issuecomment-947445757 :
When pages are retrieved but without records, it's usually related to the selectors.
Testing
document.querySelectorAll("[itemprop='articleBody'] > .section h1, .page h1, .post h1, .body > .section h1");
on https://www.mdanalysis.org/GridDataFormats/gridData/basic.html for example, returns no results.You can either make your selectors broader (we often go with
.class heading
) to also retrieve content form these pages, or add a newselectors_key
field in thestart_urls
Debugging the selectors is the next step...
Bug: no search results after adding new sitemaps
If it is a DocSearch index issue, what is the related
index_name
?index_name
= mdanalysisWhat is the current behaviour?
We added two new sitemaps to our siteindex (see Any other feedback below for details) for "distopia" and "pytng". Searching for new unique content does not give any results:
If the current behaviour is a bug, please provide all the steps to reproduce and screenshots with context.
To perform search, go to https://www.mdanalysis.org/ and use the search box as show in the screen shot:
The screen shot shows that the unique term CalcBondsOrtho (for distopia) is not found, as explained in more detail below:
distopia content is not found
distopia failed example text
distopia failed example API docs
Note that this example probably fails because content is in a dl (definition list) with dt/dd tags:
pytng content is not found
pytng failed example text
pytng failed example API docs
Note that this example probably fails to find the API doc because content is in a dl (definition list) with dt/dd tags, the text use should have been found
What is the expected behaviour?
Relevant pages from the distopia and pytng docs are found, as indicated above. (It was clearer to include the expected results above for the individual examples).
What have you tried to solve it?
Any quick clues?
Some content (for example, the API docs in https://www.mdanalysis.org/distopia/api/distopia.html) are stored in definition lists (dl with dt/dd elements) and the dt tags are NOT included as selectors in the algolia config file yet. (see PR #4700)
However, no idea why standard text is not appearing; seeing the scraper output might help but that requires algolia staff help.
Any other feedback / questions ?
We added two new sitemaps to our sitemap index https://www.mdanalysis.org/sitemapindex.xml for
Our open issues: