search

qascade / yast

Yet Another Streaming Tool
BSD 3-Clause "New" or "Revised" License
33 stars 10 forks source link

Make use of single collector while scraping 1337x target #42

Closed qascade closed 2 years ago

qascade commented 2 years ago

We are declaring a collector for parsing the result page and then we are initialising another collector again to parse the magnet. All of this should be handled using single collector only.

ash-kamrip commented 2 years ago

Hi @qascade,

I would like to work on this one.

qascade commented 2 years ago

@ash-kamrip Sure, assigning this to you. Thanks for taking interest in my project. You can understand some of the control flow of the collectors by debugging test codes that are verifiying query results. Uncomment them ofcourse. Make sure to test your code locally and make sure that the results are coming back. Then Comment the code back before pushing the code as the scraper_test will fail at github actions due to actions not able to access the target, due to lack of VPN.

ash-kamrip commented 2 years ago

@qascade Let me know if you have any questions

qascade commented 2 years ago

@ash-kamrip Hey, I have looked at the code, looks good. Will pull it and test it at my end just to be sure. Other than that LGTM. Thanks for your contribution. Will close this by today. You can look at other outstanding issues, if you want.