Closed sercansayitoglu closed 4 years ago
please provide some details
First of all, I realized that the proxy option was not working while crawling. While digging the code saw that the argv parameter takes all the arguments from the command line but can't understand the URL and result file variables. Because, they don't fit to this options part -> 'hc:t:jn:x:A:p:d:BGR:U:wD:s:m:C:qr:SIHFP:OvelE:L:M'
crawler.py (line 432):
try:
opts, args = getopt.getopt(
argv, 'hc:t:jn:x:A:p:d:BGR:U:wD:s:m:C:qr:SIHFP:OvelE:L:M')
except getopt.GetoptError as err:
print str(err)
sys.exit(1)
and all for loop starting at line 444 wasn't working for that reason
changing argv
(at line 433) to argv[2:]
fixing the getopt problem (if URL and result file variables are second and third place on the command line(first one is crawl option btw))
Also changing this code on line 596
Shared.starturl = normalize_url(args[0])
out_file = args[1]
to this
Shared.starturl = normalize_url(argv[0])
out_file = argv[1]
fixing the whole problem
as far as I remember I only edited these parts of code
if you have any questions I would like to answer
cheers!
could you please share the commandline you are using? It should be something like
python htcap.py crawl [options] https://something.com out.db
thanks!
It is:
htcap/htcap.py crawl https://******.com /tmp/test5wmjsl3ipd9u -n 4 -m aggressive -t 30 -p http:127.0.0.1:8080
ok I see.. the correct syntax is htcap.py [options] url outfile
you cannot put the option list at the end...
lol! But after this edit, I can :D
sorry I didn't pay attention to the order, instead of that it was easier to change the code to work according to my own will
thanks for your time!
Greetings!
no problem! cheers!
I wonder why using proxy affect results? Am I doing something wrong?
Code:
python htcap.py crawl -v -p http:127.0.0.1:8080 "https://X.X.X.X:8443" /tmp/test5rvygonk0u98
Result: 2 requests
Code:
python htcap.py crawl -v -p socks5:127.0.0.1:8081 "https://X.X.X.X:8443" /tmp/test5rvygonk0u98
Result: 2 requests
Code:
python htcap.py crawl -v "https://X.X.X.X:8443" /tmp/test5rvygonk0u98
Result: Hundreds requests
I tried on Windows 10, Kali Linux, and Debian 10. All are the same result.
Can you help?
I believe the problem is about how many seconds this program waits for the puppeteer to let the browser runs java script. If we can specify the time by a parameter, the problem will be fixed.
It should be like this:
-- coding: utf-8 --
""" HTCAP - beta 1 Author: filippo.cavallarin@wearesegment.com
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. """
from future import unicode_literals import sys import os import datetime import time import getopt import json import re from urlparse import urlsplit, urljoin from urllib import unquote import urllib2 import threading import subprocess from random import choice import string import ssl import signal
from core.lib.exception import * from core.lib.cookie import Cookie from core.lib.database import Database
from lib.shared import from lib.crawl_result import from core.lib.request import Request from core.lib.http_get import HttpGet from core.lib.shell import CommandExecutor
from crawler_thread import CrawlerThread
from core.lib.shingleprint import ShinglePrint
from core.lib.texthash import TextHash from core.lib.request_pattern import RequestPattern from core.lib.utils import from core.constants import from lib.utils import * from core.lib.progressbar import Progressbar
class Crawler: