Open dkipping opened 4 years ago
For reference, the yielding in the spider's start_requests
:
def start_requests(self):
... manually getting some parameters from a webpage with selenium ...
total = results_from_above
pagination_start = 0
while pagination_start < total:
url = f'{self.my_start_url}&from={pagination_start}'
pagination_start += self.pagination_size
yield SeleniumRequest(
url=url,
callback=self.parse1,
wait_time=5,
wait_until=EC.visibility_of_element_located((By.XPATH, self.xpaths[1])),
)
And yielding in the parse1
function:
def parse1(self, response):
urls_to_follow = response.selector.xpath(self.xpaths[2]).extract()
for url in urls_to_follow:
yield SeleniumRequest(
url=url,
callback=self.parse2,
wait_time=10,
wait_until= EC.presence_of_element_located((By.XPATH, self.xpaths[3]))
)
A custom crawling strategy was not really necessary until now, as with this approach the link filtering happens via xpaths already...
Following the distributed quickstart in the documentation (except the seed injection step), I am monitoring the prints from the ZeroMQ broker:
2020-07-21 11:38:15 {'started': 1595324105.494039, 'spiders_out_recvd': 0, 'spiders_in_recvd': 0, 'db_in_recvd': 1, 'db_out_recvd': 0, 'sw_in_recvd': 1, 'sw_out_recvd': 0}
... wich means, that the spider is never registered by frontera? Could that be the point breaking it? (And what could cause that? The configuration also mostly follows the general-spider example
After some debugging I can say that at least the start_requests seem to be working properly, the issue arises from the yielded requests from the parse function
Issue might be related to #337
Hi,
I have already read in discussions here, that the scheduling of requests should be done by frontera and apparently even the creation should be done by the frontier and not by the spider. However, in the documentation of scrapy and frontera it is written that requests shall be yielded in the spider
parse
function.How should the process look like, if requests are to be created by the crawling strategy and not yielded by the spider? How does the spider trigger that?
In my use case, I am using scrapy-selenium with scrapy and frontera (I use SeleniumRequests to be able to wait for JS loaded elements).
I have to generate the URLs I want to scrape in two phases: I am yielding them firstly in the
start_requests()
method of the spider instead of a seeds file and yield requests for extracted links in the first of twoparse
functions.Yielding SeleniumRequests from
start_requests
works, but yielding SeleniumRequests from theparse
function afterwards results in the following error (only pasted an extract, as the iterable error prints the same errors over and over):Very thankful for all hints and examples!