Open mantou132 opened 3 years ago
shortcuts
commented
3 years ago Hi,
It indeed doesn't seem like it is possible to access to dom via query selectors with the shadow-root tag open. I don't know much about shadowDOM but it might be possible to make it work.
As long as you can query selector something from the console, our scraper will be able to get it so you will be able to use DocSearch!
mantou132
commented
3 years ago Cannot select the content of shadowDOM through css selector or xpath.
To select shadowDOM content like a css selector, need to extend the css selector, such as using >>(outdated specification) instead of shdowDOM boundary: gem-book >> gem-book-sidebar >> gem-active-link，when using this selector, replace >> with shadowRoot, for example:
'body gem-book >> gem-book-sidebar >> gem-active-link >> a[href]'.split('>>').reduce(
(p, c, index, arr) => {
const isLastSelector = index === arr.length - 1;
return p.map((e) => [...e.querySelectorAll(c)].map((ce) => (isLastSelector ? ce : ce.shadowRoot))).flat();
},
[document],
);
This is also an example of use in the browser, if it is selenium, there should be a similar API
mantou132
commented
3 years ago Hi, I viewed the source code today, i found only a little update can support ShadowDOM.
Can use custom downloaders to pull all DOM:
# pseudocode
driver.execute_script("return document.documentElement.getInnerHTML();")Will get result:
<head>...</head>
<body>
<gem-book>
<template shadowroot="open">
... content
</template>
</gem-book>
</body>~Next, we only need to delete the all <template> tag(don't delete content), may be a regular expression~
Hello Algolia Devs,
I tried to add search function to my website, but I got the reply that no content can be crawled. Is it because my website uses shadowdom?
Thank you.