Open buren opened 6 years ago
I've created a spidr_cli
gem which includes the above mentioned functionality, plus accept/reject hosts
, ports
, links
and urls
arguments and ability to chose what method to use: Spidr::site|host|start_at
.
Sorry for not noticing this. If I were to add a CLI it would need to be a class called Spidr::CLI
. It would also need to catch Interrupt
and Errno::EPIPE
exceptions (see: how command_kit handles this). Also, would need a --format
or --output-format
option to control plain text, CSV, or JSON. Would also need specs that invoke the command and uses RSpec's .to output(...).to_stdout
.
Not to plug my own code too much, but you might want to consider using command_kit for your spidr-cli gem?
If you want to get this merged, checkout the CLI
class from wordlist.rb. Feel free to copy it's zero-dependency boilerplate CLI code.
Side note: First of all thank you for an awesome gem. Over the past years and I've reached for this gem numerous times for various purposes big and small, its always a joy to use - thank you! 🙌
Simple Command Line Interface (CLI)
Rationale I've found myself wanting to do a "quick and dirty" crawl of different websites quite often. For example to find 4X, 5XX etc. So far I've written small Ruby scripts using
spidr
with the things I need. Many of these use cases could be solved with a fairly simple CLI.Examples
it supports all
Spdir::Agent
argumentsyou can output multiple values (CSV-style), the columns argument map to methods on
page
Usage
todo
--[no-]robots
option requiresgem install robots
?If you don't want to include this here then this could be a separate gem, something like
spidr_cli
(~with your blessing~ unless you object?). However it would probably be easier for others to find it if its here.Thanks!