search

tomnomnom / waybackurls

Fetch all the URLs that the Wayback Machine knows about for a domain
3.43k stars 457 forks source link

[Feature Request] Switch to enable/disable urls from subdomains of base domain #3

Closed Rhynorater closed 6 years ago

Rhynorater commented 6 years ago

Hey Tom!

Great tool! I was wondering if it would be possible for you to add a command-line arg to select URLs only from the base domain provided. The new WayBack Machine query would look like this:

http://web.archive.org/cdx/search/cdx?url=%s/*&output=json&collapse=urlkey

as opposed to the one that pulls all subdomains too (currently in your code) which is this:

http://web.archive.org/cdx/search/cdx?url=*.%s/*&output=json&collapse=urlkey

I propose that this param should be called "with_subs" and should be Boolean defaulting to True.

Thanks, Justin

Rhynorater commented 6 years ago

Also, note, I did try to implement this, but Go doesn't like me...

I tried to add a CMD param like so:

var dates bool                                                                                                                                                                                            
var withSubs bool                                                                                                                                                                                         
flag.BoolVar(&dates, "dates", false, "show date of fetch in the first column")                                                                                                                            
flag.BoolVar(&withSubs, "withSubs", true, "pull URLs for subdomains of the hosts provided as well")

and then pass it into the functions getWaybackURLs and getCommonCrawlURLs via this:

resp, err := fetch(domain, withSubs)

Once I modified the resulting functions to look like this:

func getWaybackURLs(domain string, withSubs bool) ([]wurl, error) {
        if withSubs {
                res, err := http.Get(
                        fmt.Sprintf("http://web.archive.org/cdx/search/cdx?url=%s/*&output=json&collapse=urlkey", domain),
                )
        } else {
                res, err := http.Get(
                        fmt.Sprintf("http://web.archive.org/cdx/search/cdx?url=*.%s/*&output=json&collapse=urlkey", domain),
                )
        }

But Go didn't like that... I don't speak Go hahaha.

tomnomnom commented 6 years ago

Hey, @Rhynorater! Good idea :)

I'll have a poke at it now

tomnomnom commented 6 years ago

I've added a --no-subs option in 41bb26a

The only real problem with it is that the searches still often include subdomains, so the actual behaviour doesn't quite line up with the expected behaviour.

I'll add some additional filtering I think

tomnomnom commented 6 years ago

@Rhynorater I've added some output filtering so you only get results for the exact input domain if you specify the --no-subs option in 58bbafe

Give it a try and let me know if you hit any problems :)

Rhynorater commented 6 years ago

Thanks Tom! Sounds great!

On Tue, Aug 28, 2018 at 11:53 AM, Tom Hudson notifications@github.com wrote:

I've added a --no-subs option in 41bb26a https://github.com/tomnomnom/waybackurls/commit/41bb26a77e181fbc571aeb1d67fed4ba74ccd560

The only real problem with it is that the searches still often include subdomains, so the actual behaviour doesn't quite line up with the expected behaviour.

I'll add some additional filtering I think

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tomnomnom/waybackurls/issues/3#issuecomment-416638840, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2_r0a33rDEPuQeWmSFHeicYg0MYgs9ks5uVWdwgaJpZM4WPnr3 .

-- Justin Gardner Cell: (804)525-8089