XPath attributes without evaluation()

[n1xn]

Simple description about the feature

The title is simplest description possible.

Descrption

I am scrapping a pretty big list of product links (> 1000) and have issues that puppeteer throws cannot find context with specified id undefined. This is because I use XPath for collecting the nodes and evaluate the array in the next step. I would like to avoid the evaluate operation after using Sx() and access / map the desired attribute to a variable. Therefore I played around with the options we have with Sx()[] and found out that the attributes are actually all loaded - which means the evaluation step afterwards is not needed. My problem is that I have to replace JSHandle: when accessing an attribute within its string.

Puppeteer reference

• Here is a reference on the same issue within puppeteer itself

Current issue

1. Here is an example, which I would like to show you. This example is the one actually failing after some 'hundred-ish' iterations.

   paginations.each do |pagination_step|
   xpath_links = '(//a[contains(concat(" ", normalize-space(@class), " "), " productlist__link ")])'
   link_nodes = page.Sx(xpath_links)
   
   link_nodes.each do |product_link|
   
   # evalution will be called thousands of times.      <---------
   href = page.evaluate('e => e.href', product_link)
   # href = product_link.evaluate('e => e.href')
   
   product = { href:, category: }
   products.push(product)
   end
   end

2. After realizing that this is exceeding some limitations by browsers / puppeteer I have tried to optimize the evaluation to execute only once and setting the desired attribute href.

   paginations.each do |pagination_step|
   xpath_links = '(//a[contains(concat(" ", normalize-space(@class), " "), " productlist__link ")])'
   link_nodes = page.Sx(xpath_links)
   
   # executing now only on each pagination step - better.       <---------
   product_links = page.evaluate('e => e.map((el) => el.href)', link_nodes)
   product_links.each do |product_link|
   
   # but product_link is actually empty.       <---------
   product = { href: product_link, category: }
   products.push(product)
   end
   end

3. As mentioned in the comment in code 2. the problem is that the mapped evaluation does not contain any values (tried also el.getAttribute('href')). So I tried to access the properties from Sx directly in ruby via property('href') and actually got the value but prefixed with JSHandle: - which I replaced and got it working.

   paginations.each do |pagination_step|
   xpath_links = '(//a[contains(concat(" ", normalize-space(@class), " "), " productlist__link ")])'
   link_nodes = page.Sx(xpath_links)
   
   # do not evaluate anything - loop through nodes
   link_nodes.each do |product_link|
   
   # access the current nodes property and remove JSHandle: prefix.      <---------
   href = product_link.property("href").to_s.gsub('JSHandle:', '')
   
   product = { href:, category: }
   products.push(product)
   end
   end

Usecase / Motivation

I am not sure if I am using this right or missed a conzept, but as mentioned I have a problem with page.evaluate(). I would like to get attributes by xpath without hacking .to_s.gsub('JSHandle:'.''). See the code below for my suggestion.

xpath = '//expression'
xpath_nodes = Sx(xpath)

xpath_nodes.each do |node|
  href = node.attribute('href')
end