Command line arguments - Githubissues

rivermont / spidy

The simple, easy to use command line web crawler.

GNU General Public License v3.0

334 stars 69 forks source link

Command line arguments #46

Open rivermont opened 6 years ago

rivermont commented 6 years ago

Arguments for:

[ ] Overwrite existing save files
[ ] Raise Errors (possibly only for different severity levels?)
[ ] Save pages
[ ] Save words
[ ] Zip files
[ ] Override file size check.
[ ] Domain restriction
[ ] Respect robots.txt
[ ] Custom save file locations
[ ] Autosave count
[ ] HTTP headers
[ ] Max errors
[ ] Starting page

NOTES: Don't try to use sys.argv.

dasdachs commented 6 years ago

If you are ok with (click)[http://click.pocoo.org/] I'd be happy to work on it.

rivermont commented 6 years ago

Sure! Just make sure to add click to requirements.txt and update setup.py.

dasdachs commented 6 years ago

Cool. I'll get at it.

realazizk commented 6 years ago

It is a pain to work on this project, someone should fix it, adding this feature is pretty easy but because it has a lot of GLOBAL variables spattered all over the place it's not going to be that easy.

rivermont commented 6 years ago

@mohamed-aziz I agree, actually. This started as a simple script that I whipped in the spring, but it jumped to a full-blown project this past week. Would love some pointers on where to go from here!