Open jajajasalu2 opened 5 years ago
One possibility for this:
Use requests/urllib in test cases and create fake Scrapy responses from these responses. Decorate the test methods with vcrpy, so that response bodies can be taken from VCR cassettes later, and updated when necessary. This allows to "switch the mock on and off".
Current unit tests use HTML files stored in
tests/mocks
. While using HTML mocks is helpful in testing the parsing logic, there is no way to check if a current online response is parsed correctly by the xpaths. Testing real time data would enable testing the xpaths inpatchfinder.entrypoint
.A solution like https://github.com/kevin1024/vcrpy is optimal. vcrpy, unfortunately, does not currently support Scrapy's networking framework Twisted.
Thus, an efficient method to test if a current response is parsed properly is required.