Closed CristianMR closed 7 months ago
Look, I totally understand, but it'll break the current API using, also if one does not await queue
, 'unhandled promise' warning will always be there. What's worse, it to confuse the API when providing promise and callback at the same time. To be hoest, it is better to deduplicate outside the crawler, which means should be handled by the developer. That keeps the flexibility and consistency. Hope it helpful.
Thanks for your answer Mike. I already did it. It took some hours to find out this issue so probably others will do it too. Have a nice year btw ✨
Sorry to hear that you spent hours on this issue, so let's keep the issue details here to help others. Thanks, and the same to you, have a nice year.
When
skipDuplicates
is set totrue
, the 'drain' event is emitted before it's checked whetherqueue(uri)
has been seen. The solution is quite straightforward: provide a promise to determine whether queue(uri)
has been resolved. This would allow us toawait c.queue(uri)
inside the callback function before callingdone()
.