Unfortunately the Terms and Privacy Policy are located on the same page, and not in separate DOM elements with their own IDs. I believe that theoretically it should be possible to select all elements in between different elements (i.e. everything between h2#terms-of-service and h2#privacy-policy for the ToS, and everything beyond h2#privacy-policy in that element for the privacy policy), but my XPath-fu is not good enough for that. Thus, it now just crawls both at once, and hopes the layout doesn't change significantly.
michielbdejong
commented
6 years ago
Note that I haven't been able to test the XPath.
Unfortunately the Terms and Privacy Policy are located on the same page, and not in separate DOM elements with their own IDs. I believe that theoretically it should be possible to select all elements in between different elements (i.e. everything between
h2#terms-of-serviceandh2#privacy-policyfor the ToS, and everything beyondh2#privacy-policyin that element for the privacy policy), but my XPath-fu is not good enough for that. Thus, it now just crawls both at once, and hopes the layout doesn't change significantly.