fail to clone web page

[Haotian-Shi-cyber]

when i try to clone, it alerts me "aiohttp" is not installed. after then, use "python3.6 -m pip install aiohttp".. and i got this couldn't find any solutions

[lordlabuckdas]

Can you try running sudo apt install python3.6-dev before the setup?

[Haotian-Shi-cyber]

just found it is ./bin/clone right? not built-in command... its ok, is there requirements for pages im going to clone? eg, must be http and not for https?

[lordlabuckdas]

./bin/clone contains the raw file for the cloner which will be built into a pip module by setuptools in setup.py. The recommended way to set up and use Snare (and Cloner) is to run [sudo] python setup.py install and then use the clone command directly. Also, I don't think there are no requirements regarding the protocol (http/https), you can clone any site.

[Haotian-Shi-cyber]

i cloned youtube.com but still can't access page.

[Haotian-Shi-cyber]

every time i cloned a page, it got so many warnings and errors, is there anything wrong

[mzfr]

@Haotian-Shi-cyber Please post the output of the log files.

[Haotian-Shi-cyber]

some examples

[mzfr]

@Haotian-Shi-cyber it would be nice if you can post the complete output on Pastebin and then give the link. The screenshot you provided doesn't give much information.

[Haotian-Shi-cyber]

http://pastebin.com/28XTaM98

[afeena]

@Haotian-Shi-cyber most of the warnings come from the css parsing library, it is not the from cloner itself. Also please remember that cloner has really limited functionality, so not every page can be successfully cloned.

Regarding the page serving, could you please provide the snare logs instead of the cloning logs? From your cloning logs I can not say anything regarding your problem with page access.

Thanks

[aabulaeva]

i cloned youtube.com but still can't access page.

Hello, i have the same issue do you find a solution ? thanks

[Haotian-Shi-cyber]

hi,

no, i gave it up.

thanks

[mzfr]

I just tested this and had no issues except the following error:

Traceback (most recent call last):                                                                                                                           
  File "/home/mzfr/dev/snare/venv/bin/clone", line 4, in <module>                                                                                            
    __import__('pkg_resources').run_script('Snare==0.3.0', 'clone')                                                                                          
  File "/home/mzfr/dev/snare/venv/lib/python3.10/site-packages/pkg_resources/__init__.py", line 672, in run_script                                           
    self.require(requires)[0].run_script(script_name, ns)                                                                                                    
  File "/home/mzfr/dev/snare/venv/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1479, in run_script                                          
    exec(script_code, namespace, namespace)                                                                                                                  
  File "/home/mzfr/dev/snare/venv/lib/python3.10/site-packages/Snare-0.3.0-py3.10.egg/EGG-INFO/scripts/clone", line 81, in <module>                          
  File "/home/mzfr/dev/snare/venv/lib/python3.10/site-packages/Snare-0.3.0-py3.10.egg/EGG-INFO/scripts/clone", line 67, in main                              
UnboundLocalError: local variable 'end' referenced before assignment

That should be fixed https://github.com/mushorg/snare/pull/320