UI Integration Checklist

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

http://scrapy-cluster.readthedocs.io/

MIT License

1.18k stars 324 forks source link

UI Integration Checklist #116

Open madisonb opened 7 years ago

madisonb commented 7 years ago

Checklist for items that I know need worked on before the ui branch can be merged into the dev branch

[x] Create documentation
[x] Add offline unit tests
[x] Add online integration tests
[x] Add tests to run_*_tests.sh
[ ] Can we add actual ui tests? (aka Selenium or some other framework)
[x] Can we use the local flask app to pass requests to the Rest Endpoint, instead of the browser? (only exposing the UI port on the cluster)
[x] Build out primitive functionality to submit a crawl request, or get basic information about the cluster
[x] Ensure travis can run the unit tests

This comment here outlines FULL functionality, however I don't think we need that to provide a basic UI for the cluster to be merged into dev.

This checklist will be updated as we come across other smaller items that prevent the ui branch from merging into dev

damienkilgannon commented 7 years ago

This PR #129 satisfies the following items from the above checklist; although additional unit tests are required for better code coverage.

[x] Create documentation
[x] Add offline unit tests
[x] Add online integration tests
[x] Add tests to run_*_tests.sh
[ ] Can we add actual ui tests? (aka Selenium or some other framework)
[x] Can we use the local flask app to pass requests to the Rest Endpoint, instead of the browser? (only exposing the UI port on the cluster)
[x] Build out primitive functionality to submit a crawl request, or get basic information about the cluster
[x] Ensure travis can run the unit tests

backtrack-5 commented 6 years ago

Hi @madisonb , @damienkilgannon ,

I have written UI test using selenium and python for all pages available in Scrapy Cluster. (index, Kafka, Redis and Crawler)

and created my PR #174

Added my code under : ~ui\tests\uitest

Please let me know if i need to do any changes

Thanks,

damienkilgannon commented 6 years ago

Hi @sornalingam nice .... been a while since I worked on this, I must get back to it. I think the branch you created the PR on is the most appropriate. I will check out your branch in the next day or two and have a look.

backtrack-5 commented 6 years ago

Hi @damienkilgannon, Thanks, I have commited my new code. Please have a look when you get a chance and let me know.