Closed grzesiekrzaca closed 8 years ago
Try wayback_machine_downloader http://starkreality.smackjeeves.com
, it happens that you will get bad results specifying a directory.
Ok it started crawling but many pages still produce empty index.html, some don't. Is there a way to get images from other hosts instead of leaving src value in img tag?
Downloading http://starkreality.smackjeeves.com to
websites/starkreality.smackje
eves.com/ from Wayback Machine...
http://starkreality.smackjeeves.com/robots.txt ->
websites/starkreality.smackjee
ves.com/robots.txt (1/62)
http://starkreality.smackjeeves.com:80/otherworks/index.html # 404 Not Found
http://starkreality.smackjeeves.com:80/otherworks/index.html ->
websites/starkre
ality.smackjeeves.com/otherworks/index.html (2/62)
http://starkreality.smackjeeves.com:80/news-archive/ -> websites/
starkreality.sm
ackjeeves.com/news-archive/index.html (3/62)
http://starkreality.smackjeeves.com:80/ -> websites/
starkreality.smackjeeves.com
/index.html (4/62)
http://starkreality.smackjeeves.com:80/home/ #
websites/starkreality.smackjeeves
.com/home/index.html already exists. (5/62)
http://starkreality.smackjeeves.com:80/monster/ ->
websites/starkreality.smackje
eves.com/monster/index.html (6/62)
http://starkreality.smackjeeves.com:80/cubby/ ->
websites/starkreality.smackjeev
es.com/cubby/index.html (7/62)
http://starkreality.smackjeeves.com:80/angle/ ->
websites/starkreality.smackjeev
es.com/angle/index.html (8/62)
http://starkreality.smackjeeves.com:80/aloof/ ->
websites/starkreality.smackjeev
es.com/aloof/index.html (9/62)
http://starkreality.smackjeeves.com:80/merchshirts/ ->
websites/starkreality.sma
ckjeeves.com/merchshirts/index.html (10/62)
http://starkreality.smackjeeves.com:80/merchcomics/ ->
websites/starkreality.sma
ckjeeves.com/merchcomics/index.html (11/62)
http://starkreality.smackjeeves.com:80/merchart/ ->
websites/starkreality.smackj
eeves.com/merchart/index.html (12/62)
http://starkreality.smackjeeves.com:80/summary/ ->
websites/starkreality.smackje
eves.com/summary/index.html (13/62)
http://starkreality.smackjeeves.com:80/merch/ ->
websites/starkreality.smackjeev
es.com/merch/index.html (14/62)
http://starkreality.smackjeeves.com:80/links/ ->
websites/starkreality.smackjeev
es.com/links/index.html (15/62)
http://starkreality.smackjeeves.com:80/pointfive/ ->
websites/starkreality.smack
jeeves.com/pointfive/index.html (16/62)
http://starkreality.smackjeeves.com:80/latest-story/ -> websites/
starkreality.sm
ackjeeves.com/latest-story/index.html (17/62)
http://starkreality.smackjeeves.com:80/archivepage/ #
websites/starkreality.smac
kjeeves.com/archivepage/index.html already exists. (18/62)
http://starkreality.smackjeeves.com:80/Archivepage/ #
websites/starkreality.smac
kjeeves.com/Archivepage/index.html already exists. (19/62)
http://starkreality.smackjeeves.com/born # incorrect header check
http://starkreality.smackjeeves.com/born -> websites/
starkreality.smackjeeves.co
m/born/index.html (20/62)
http://starkreality.smackjeeves.com/dive # incorrect header check
http://starkreality.smackjeeves.com/dive -> websites/
starkreality.smackjeeves.co
m/dive/index.html (21/62)
http://starkreality.smackjeeves.com/augment # incorrect header check
http://starkreality.smackjeeves.com/augment ->
websites/starkreality.smackjeeves
.com/augment/index.html (22/62)
http://starkreality.smackjeeves.com/hitchhiker # incorrect header check
http://starkreality.smackjeeves.com/hitchhiker ->
websites/starkreality.smackjee
ves.com/hitchhiker/index.html (23/62)
http://starkreality.smackjeeves.com/trophy # incorrect header check
http://starkreality.smackjeeves.com/trophy ->
websites/starkreality.smackjeeves.
com/trophy/index.html (24/62)
http://starkreality.smackjeeves.com/notice # incorrect header check
http://starkreality.smackjeeves.com/notice ->
websites/starkreality.smackjeeves.
com/notice/index.html (25/62)
http://starkreality.smackjeeves.com/anti-theft # incorrect header check
http://starkreality.smackjeeves.com/anti-theft ->
websites/starkreality.smackjee
ves.com/anti-theft/index.html (26/62)
http://starkreality.smackjeeves.com/brekky # incorrect header check
http://starkreality.smackjeeves.com/brekky ->
websites/starkreality.smackjeeves.
com/brekky/index.html (27/62)
http://starkreality.smackjeeves.com/bigman # incorrect header check
http://starkreality.smackjeeves.com/bigman ->
websites/starkreality.smackjeeves.
com/bigman/index.html (28/62)
http://starkreality.smackjeeves.com/Sale # websites/
starkreality.smackjeeves.com
/Sale/index.html already exists. (29/62)
http://starkreality.smackjeeves.com/Misplaced # incorrect header check
http://starkreality.smackjeeves.com/Misplaced ->
websites/starkreality.smackjeev
es.com/Misplaced/index.html (30/62)
http://starkreality.smackjeeves.com/nice # incorrect header check
http://starkreality.smackjeeves.com/nice -> websites/
starkreality.smackjeeves.co
m/nice/index.html (31/62)
http://starkreality.smackjeeves.com/place # incorrect header check
http://starkreality.smackjeeves.com/place ->
websites/starkreality.smackjeeves.c
om/place/index.html (32/62)
http://starkreality.smackjeeves.com/laundry # incorrect header check
http://starkreality.smackjeeves.com/laundry ->
websites/starkreality.smackjeeves
.com/laundry/index.html (33/62)
http://starkreality.smackjeeves.com/hobby # incorrect header check
http://starkreality.smackjeeves.com/hobby ->
websites/starkreality.smackjeeves.c
om/hobby/index.html (34/62)
http://starkreality.smackjeeves.com/bitefight # incorrect header check
http://starkreality.smackjeeves.com/bitefight ->
websites/starkreality.smackjeev
es.com/bitefight/index.html (35/62)
http://starkreality.smackjeeves.com/sequential # incorrect header check
http://starkreality.smackjeeves.com/sequential ->
websites/starkreality.smackjee
ves.com/sequential/index.html (36/62)
http://starkreality.smackjeeves.com/round # incorrect header check
http://starkreality.smackjeeves.com/round ->
websites/starkreality.smackjeeves.c
om/round/index.html (37/62)
http://starkreality.smackjeeves.com/drag # incorrect header check
http://starkreality.smackjeeves.com/drag -> websites/
starkreality.smackjeeves.co
m/drag/index.html (38/62)
http://starkreality.smackjeeves.com/numberone # incorrect header check
http://starkreality.smackjeeves.com/numberone ->
websites/starkreality.smackjeev
es.com/numberone/index.html (39/62)
http://starkreality.smackjeeves.com/sphere # incorrect header check
http://starkreality.smackjeeves.com/sphere ->
websites/starkreality.smackjeeves.
com/sphere/index.html (40/62)
http://starkreality.smackjeeves.com/faces # incorrect header check
http://starkreality.smackjeeves.com/faces ->
websites/starkreality.smackjeeves.c
om/faces/index.html (41/62)
http://starkreality.smackjeeves.com/buttsinchairs # incorrect header check
http://starkreality.smackjeeves.com/buttsinchairs ->
websites/starkreality.smack
jeeves.com/buttsinchairs/index.html (42/62)
http://starkreality.smackjeeves.com/parking # incorrect header check
http://starkreality.smackjeeves.com/parking ->
websites/starkreality.smackjeeves
.com/parking/index.html (43/62)
http://starkreality.smackjeeves.com/cost # incorrect header check
http://starkreality.smackjeeves.com/cost -> websites/
starkreality.smackjeeves.co
m/cost/index.html (44/62)
http://starkreality.smackjeeves.com/humanspam # incorrect header check
http://starkreality.smackjeeves.com/humanspam ->
websites/starkreality.smackjeev
es.com/humanspam/index.html (45/62)
http://starkreality.smackjeeves.com/gutter # incorrect header check
http://starkreality.smackjeeves.com/gutter ->
websites/starkreality.smackjeeves.
com/gutter/index.html (46/62)
http://starkreality.smackjeeves.com/behind-the-scenes # incorrect header
check
http://starkreality.smackjeeves.com/behind-the-scenes ->
websites/starkreality.s
mackjeeves.com/behind-the-scenes/index.html (47/62)
http://starkreality.smackjeeves.com:80/misplaced/ #
websites/starkreality.smackj
eeves.com/misplaced/index.html already exists. (48/62)
http://starkreality.smackjeeves.com:80/Hobby/ #
websites/starkreality.smackjeeve
s.com/Hobby/index.html already exists. (49/62)
http://starkreality.smackjeeves.com:80/Augment/ #
websites/starkreality.smackjee
ves.com/Augment/index.html already exists. (50/62)
http://starkreality.smackjeeves.com:80/Faces2/ ->
websites/starkreality.smackjee
ves.com/Faces2/index.html (51/62)
http://starkreality.smackjeeves.com:80/bigman2/ ->
websites/starkreality.smackje
eves.com/bigman2/index.html (52/62)
http://starkreality.smackjeeves.com:80/Notice/ #
websites/starkreality.smackjeev
es.com/Notice/index.html already exists. (53/62)
http://starkreality.smackjeeves.com:80/bitefight2/ ->
websites/starkreality.smac
kjeeves.com/bitefight2/index.html (54/62)
http://starkreality.smackjeeves.com:80/bitefight3/ ->
websites/starkreality.smac
kjeeves.com/bitefight3/index.html (55/62)
http://starkreality.smackjeeves.com:80/Trophy/ #
websites/starkreality.smackjeev
es.com/Trophy/index.html already exists. (56/62)
http://starkreality.smackjeeves.com:80/ButtsInChairs/ # websites/
starkreality.sm
ackjeeves.com/ButtsInChairs/index.html already exists. (57/62)
http://starkreality.smackjeeves.com:80/BiteFight/ #
websites/starkreality.smackj
eeves.com/BiteFight/index.html already exists. (58/62)
http://starkreality.smackjeeves.com:80/Anti-Theft/ #
websites/starkreality.smack
jeeves.com/Anti-Theft/index.html already exists. (59/62)
http://starkreality.smackjeeves.com:80/Born/ #
websites/starkreality.smackjeeves
.com/Born/index.html already exists. (60/62)
http://starkreality.smackjeeves.com:80/sale/ #
websites/starkreality.smackjeeves
.com/sale/index.html already exists. (61/62)
http://starkreality.smackjeeves.com:80/faces2 #
websites/starkreality.smackjeeve
s.com/faces2/index.html already exists. (62/62)
Download complete, saved in websites/starkreality.smackjeeves.com/ (62
files)
On Fri, Mar 11, 2016 at 2:51 AM, hartator notifications@github.com wrote:
Try wayback_machine_downloader http://starkreality.smackjeeves.com, it happens that you will get bad results specifying a directory.
— Reply to this email directly or view it on GitHub https://github.com/hartator/wayback-machine-downloader/issues/31#issuecomment-195139316 .
What do you mean by "leaving src value in img tag?" ?
If there aren't downloaded now by the backup tool, it probably mean the images haven't been backup by the Internet Archive.
On Thu, Mar 10, 2016 at 8:09 PM, grzesiekrzaca notifications@github.com wrote:
Ok it started crawling but many pages still produce empty index.html, some don't. Is there a way to get images from other hosts instead of leaving src value in img tag?
Downloading http://starkreality.smackjeeves.com to websites/starkreality.smackje eves.com/ from Wayback Machine... http://starkreality.smackjeeves.com/robots.txt -> websites/starkreality.smackjee ves.com/robots.txt (1/62) http://starkreality.smackjeeves.com:80/otherworks/index.html # 404 Not Found http://starkreality.smackjeeves.com:80/otherworks/index.html -> websites/starkre ality.smackjeeves.com/otherworks/index.html (2/62) http://starkreality.smackjeeves.com:80/news-archive/ -> websites/ starkreality.sm ackjeeves.com/news-archive/index.html (3/62) http://starkreality.smackjeeves.com:80/ -> websites/ starkreality.smackjeeves.com /index.html (4/62) http://starkreality.smackjeeves.com:80/home/ # websites/starkreality.smackjeeves .com/home/index.html already exists. (5/62) http://starkreality.smackjeeves.com:80/monster/ -> websites/starkreality.smackje eves.com/monster/index.html (6/62) http://starkreality.smackjeeves.com:80/cubby/ -> websites/starkreality.smackjeev es.com/cubby/index.html (7/62) http://starkreality.smackjeeves.com:80/angle/ -> websites/starkreality.smackjeev es.com/angle/index.html (8/62) http://starkreality.smackjeeves.com:80/aloof/ -> websites/starkreality.smackjeev es.com/aloof/index.html (9/62) http://starkreality.smackjeeves.com:80/merchshirts/ -> websites/starkreality.sma ckjeeves.com/merchshirts/index.html (10/62) http://starkreality.smackjeeves.com:80/merchcomics/ -> websites/starkreality.sma ckjeeves.com/merchcomics/index.html (11/62) http://starkreality.smackjeeves.com:80/merchart/ -> websites/starkreality.smackj eeves.com/merchart/index.html (12/62) http://starkreality.smackjeeves.com:80/summary/ -> websites/starkreality.smackje eves.com/summary/index.html (13/62) http://starkreality.smackjeeves.com:80/merch/ -> websites/starkreality.smackjeev es.com/merch/index.html (14/62) http://starkreality.smackjeeves.com:80/links/ -> websites/starkreality.smackjeev es.com/links/index.html (15/62) http://starkreality.smackjeeves.com:80/pointfive/ -> websites/starkreality.smack jeeves.com/pointfive/index.html (16/62) http://starkreality.smackjeeves.com:80/latest-story/ -> websites/ starkreality.sm ackjeeves.com/latest-story/index.html (17/62) http://starkreality.smackjeeves.com:80/archivepage/ # websites/starkreality.smac kjeeves.com/archivepage/index.html already exists. (18/62) http://starkreality.smackjeeves.com:80/Archivepage/ # websites/starkreality.smac kjeeves.com/Archivepage/index.html already exists. (19/62) http://starkreality.smackjeeves.com/born # incorrect header check http://starkreality.smackjeeves.com/born -> websites/ starkreality.smackjeeves.co m/born/index.html (20/62) http://starkreality.smackjeeves.com/dive # incorrect header check http://starkreality.smackjeeves.com/dive -> websites/ starkreality.smackjeeves.co m/dive/index.html (21/62) http://starkreality.smackjeeves.com/augment # incorrect header check http://starkreality.smackjeeves.com/augment -> websites/starkreality.smackjeeves .com/augment/index.html (22/62) http://starkreality.smackjeeves.com/hitchhiker # incorrect header check http://starkreality.smackjeeves.com/hitchhiker -> websites/starkreality.smackjee ves.com/hitchhiker/index.html (23/62) http://starkreality.smackjeeves.com/trophy # incorrect header check http://starkreality.smackjeeves.com/trophy -> websites/starkreality.smackjeeves. com/trophy/index.html (24/62) http://starkreality.smackjeeves.com/notice # incorrect header check http://starkreality.smackjeeves.com/notice -> websites/starkreality.smackjeeves. com/notice/index.html (25/62) http://starkreality.smackjeeves.com/anti-theft # incorrect header check http://starkreality.smackjeeves.com/anti-theft -> websites/starkreality.smackjee ves.com/anti-theft/index.html (26/62) http://starkreality.smackjeeves.com/brekky # incorrect header check http://starkreality.smackjeeves.com/brekky -> websites/starkreality.smackjeeves. com/brekky/index.html (27/62) http://starkreality.smackjeeves.com/bigman # incorrect header check http://starkreality.smackjeeves.com/bigman -> websites/starkreality.smackjeeves. com/bigman/index.html (28/62) http://starkreality.smackjeeves.com/Sale # websites/ starkreality.smackjeeves.com /Sale/index.html already exists. (29/62) http://starkreality.smackjeeves.com/Misplaced # incorrect header check http://starkreality.smackjeeves.com/Misplaced -> websites/starkreality.smackjeev es.com/Misplaced/index.html (30/62) http://starkreality.smackjeeves.com/nice # incorrect header check http://starkreality.smackjeeves.com/nice -> websites/ starkreality.smackjeeves.co m/nice/index.html (31/62) http://starkreality.smackjeeves.com/place # incorrect header check http://starkreality.smackjeeves.com/place -> websites/starkreality.smackjeeves.c om/place/index.html (32/62) http://starkreality.smackjeeves.com/laundry # incorrect header check http://starkreality.smackjeeves.com/laundry -> websites/starkreality.smackjeeves .com/laundry/index.html (33/62) http://starkreality.smackjeeves.com/hobby # incorrect header check http://starkreality.smackjeeves.com/hobby -> websites/starkreality.smackjeeves.c om/hobby/index.html (34/62) http://starkreality.smackjeeves.com/bitefight # incorrect header check http://starkreality.smackjeeves.com/bitefight -> websites/starkreality.smackjeev es.com/bitefight/index.html (35/62) http://starkreality.smackjeeves.com/sequential # incorrect header check http://starkreality.smackjeeves.com/sequential -> websites/starkreality.smackjee ves.com/sequential/index.html (36/62) http://starkreality.smackjeeves.com/round # incorrect header check http://starkreality.smackjeeves.com/round -> websites/starkreality.smackjeeves.c om/round/index.html (37/62) http://starkreality.smackjeeves.com/drag # incorrect header check http://starkreality.smackjeeves.com/drag -> websites/ starkreality.smackjeeves.co m/drag/index.html (38/62) http://starkreality.smackjeeves.com/numberone # incorrect header check http://starkreality.smackjeeves.com/numberone -> websites/starkreality.smackjeev es.com/numberone/index.html (39/62) http://starkreality.smackjeeves.com/sphere # incorrect header check http://starkreality.smackjeeves.com/sphere -> websites/starkreality.smackjeeves. com/sphere/index.html (40/62) http://starkreality.smackjeeves.com/faces # incorrect header check http://starkreality.smackjeeves.com/faces -> websites/starkreality.smackjeeves.c om/faces/index.html (41/62) http://starkreality.smackjeeves.com/buttsinchairs # incorrect header check http://starkreality.smackjeeves.com/buttsinchairs -> websites/starkreality.smack jeeves.com/buttsinchairs/index.html (42/62) http://starkreality.smackjeeves.com/parking # incorrect header check http://starkreality.smackjeeves.com/parking -> websites/starkreality.smackjeeves .com/parking/index.html (43/62) http://starkreality.smackjeeves.com/cost # incorrect header check http://starkreality.smackjeeves.com/cost -> websites/ starkreality.smackjeeves.co m/cost/index.html (44/62) http://starkreality.smackjeeves.com/humanspam # incorrect header check http://starkreality.smackjeeves.com/humanspam -> websites/starkreality.smackjeev es.com/humanspam/index.html (45/62) http://starkreality.smackjeeves.com/gutter # incorrect header check http://starkreality.smackjeeves.com/gutter -> websites/starkreality.smackjeeves. com/gutter/index.html (46/62) http://starkreality.smackjeeves.com/behind-the-scenes # incorrect header check http://starkreality.smackjeeves.com/behind-the-scenes -> websites/starkreality.s mackjeeves.com/behind-the-scenes/index.html (47/62) http://starkreality.smackjeeves.com:80/misplaced/ # websites/starkreality.smackj eeves.com/misplaced/index.html already exists. (48/62) http://starkreality.smackjeeves.com:80/Hobby/ # websites/starkreality.smackjeeve s.com/Hobby/index.html already exists. (49/62) http://starkreality.smackjeeves.com:80/Augment/ # websites/starkreality.smackjee ves.com/Augment/index.html already exists. (50/62) http://starkreality.smackjeeves.com:80/Faces2/ -> websites/starkreality.smackjee ves.com/Faces2/index.html (51/62) http://starkreality.smackjeeves.com:80/bigman2/ -> websites/starkreality.smackje eves.com/bigman2/index.html (52/62) http://starkreality.smackjeeves.com:80/Notice/ # websites/starkreality.smackjeev es.com/Notice/index.html already exists. (53/62) http://starkreality.smackjeeves.com:80/bitefight2/ -> websites/starkreality.smac kjeeves.com/bitefight2/index.html (54/62) http://starkreality.smackjeeves.com:80/bitefight3/ -> websites/starkreality.smac kjeeves.com/bitefight3/index.html (55/62) http://starkreality.smackjeeves.com:80/Trophy/ # websites/starkreality.smackjeev es.com/Trophy/index.html already exists. (56/62) http://starkreality.smackjeeves.com:80/ButtsInChairs/ # websites/ starkreality.sm ackjeeves.com/ButtsInChairs/index.html already exists. (57/62) http://starkreality.smackjeeves.com:80/BiteFight/ # websites/starkreality.smackj eeves.com/BiteFight/index.html already exists. (58/62) http://starkreality.smackjeeves.com:80/Anti-Theft/ # websites/starkreality.smack jeeves.com/Anti-Theft/index.html already exists. (59/62) http://starkreality.smackjeeves.com:80/Born/ # websites/starkreality.smackjeeves .com/Born/index.html already exists. (60/62) http://starkreality.smackjeeves.com:80/sale/ # websites/starkreality.smackjeeves .com/sale/index.html already exists. (61/62) http://starkreality.smackjeeves.com:80/faces2 # websites/starkreality.smackjeeve s.com/faces2/index.html already exists. (62/62) Download complete, saved in websites/starkreality.smackjeeves.com/ (62 files)
On Fri, Mar 11, 2016 at 2:51 AM, hartator notifications@github.com wrote:
Try wayback_machine_downloader http://starkreality.smackjeeves.com, it happens that you will get bad results specifying a directory.
— Reply to this email directly or view it on GitHub < https://github.com/hartator/wayback-machine-downloader/issues/31#issuecomment-195139316
.
— Reply to this email directly or view it on GitHub https://github.com/hartator/wayback-machine-downloader/issues/31#issuecomment-195144621 .
Seems to be related to #33.
Be advised: this comic contains voilence.
Windows 10, Ruby 2.2
So i tried:
I have those 2 pages but that's it.
I tried to manually download an archived comic but that gives me a sole 0kb index.html