search

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.31k stars 129 forks source link

cannot import name 'SSLCertificateError' #163

Closed mkrzmr closed 4 years ago

mkrzmr commented 4 years ago

See below, gs-server installed successfully. Grab-site still seems to miss something:

Traceback (most recent call last): File "/usr/local/bin/grab-site", line 4, in import('pkg_resources').run_script('grab-site==2.1.17', 'grab-site') File "/home/michael/.local/lib/python3.6/site-packages/pkg_resources/init.py", line 666, in run_script self.require(requires)[0].run_script(script_name, ns) File "/home/michael/.local/lib/python3.6/site-packages/pkg_resources/init.py", line 1462, in run_script exec(code, namespace, namespace) File "/usr/local/lib/python3.6/dist-packages/grab_site-2.1.17-py3.6.egg/EGG-INFO/scripts/grab-site", line 4, in main.main() File "/usr/lib/python3/dist-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 535, in invoke return callback(args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/grab_site-2.1.17-py3.6.egg/libgrabsite/main.py", line 369, in main from wpull.processor.web import WebProcessor File "/home/michael/.local/lib/python3.6/site-packages/wpull/processor/web.py", line 18, in from wpull.processor.coprocessor.phantomjs import PhantomJSCoprocessor File "/home/michael/.local/lib/python3.6/site-packages/wpull/processor/coprocessor/phantomjs.py", line 22, in from wpull.processor.rule import ProcessingRule File "/home/michael/.local/lib/python3.6/site-packages/wpull/processor/rule.py", line 21, in from wpull.protocol.http.robots import RobotsTxtChecker File "/home/michael/.local/lib/python3.6/site-packages/wpull/protocol/http/robots.py", line 14, in from wpull.protocol.http.web import WebClient File "/home/michael/.local/lib/python3.6/site-packages/wpull/protocol/http/web.py", line 13, in from wpull.protocol.http.client import Client File "/home/michael/.local/lib/python3.6/site-packages/wpull/protocol/http/client.py", line 14, in from wpull.protocol.abstract.client import BaseClient, BaseSession, DurationTimeout File "/home/michael/.local/lib/python3.6/site-packages/wpull/protocol/abstract/client.py", line 12, in from wpull.network.pool import ConnectionPool File "/home/michael/.local/lib/python3.6/site-packages/wpull/network/pool.py", line 10, in from wpull.network.connection import Connection, SSLConnection File "/home/michael/.local/lib/python3.6/site-packages/wpull/network/connection.py", line 13, in from tornado.netutil import SSLCertificateError ImportError: cannot import name 'SSLCertificateError'

ivan commented 4 years ago

OS? Which install instructions did you follow in the README?

The install should not produce /usr/local/bin/grab-site or use Python 3.6; Python 3.7 is required.

If they are present, /usr/local/bin/{grab-site,gs-server,gs-dump-urls} should be removed so that the correctly-installed versions are used from the user-local install path.

mkrzmr commented 4 years ago

thanks, I did change to python3.7 and reinstall, now getting a different error:

btw running on Ubuntu, followed the Ubuntu install steps

Traceback (most recent call last): File "/usr/local/bin/grab-site", line 4, in import('pkg_resources').run_script('grab-site==2.1.17', 'grab-site') File "/usr/local/lib/python3.7/site-packages/pkg_resources/init.py", line 666, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/local/lib/python3.7/site-packages/pkg_resources/init.py", line 1446, in run_script exec(code, namespace, namespace) File "/usr/local/lib/python3.7/site-packages/grab_site-2.1.17-py3.7.egg/EGG-INFO/scripts/grab-site", line 4, in main.main() File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 764, in call return self.main(args, kwargs) File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/usr/local/lib/python3.7/site-packages/grab_site-2.1.17-py3.7.egg/libgrabsite/main.py", line 369, in main from wpull.processor.web import WebProcessor File "/usr/local/lib/python3.7/site-packages/wpull-2.0.1-py3.7.egg/wpull/processor/web.py", line 18, in from wpull.processor.coprocessor.phantomjs import PhantomJSCoprocessor File "/usr/local/lib/python3.7/site-packages/wpull-2.0.1-py3.7.egg/wpull/processor/coprocessor/phantomjs.py", line 19, in from wpull.driver.phantomjs import PhantomJSDriverParams, PhantomJSDriver File "/usr/local/lib/python3.7/site-packages/wpull-2.0.1-py3.7.egg/wpull/driver/phantomjs.py", line 10, in from wpull.driver.process import Process File "/usr/local/lib/python3.7/site-packages/wpull-2.0.1-py3.7.egg/wpull/driver/process.py", line 56 self._stderr_reader = asyncio.async(self._read_stderr()) ^ SyntaxError: invalid syntax

ivan commented 4 years ago

That error indicates that wpull was installed instead of ludios_wpull. grab-site can only work with ludios_wpull, which has the fixed syntax for Python 3.7.

The install instructions are only designed to work if followed exactly. There shouldn't be anything getting installed to /usr/local. Note also the 'non-root' for the remaining steps.

If any of the previous install steps failed, please let me know, they cannot be substituted for another similar step. If you would like to share terminal output of the entire install procedure, that would help.

ivan commented 4 years ago

I used a digitalocean VM to verify that the install steps work on Ubuntu 19.10, at least up to the point where 512MB wasn't enough to compile lxml, but I'm pretty sure things work fine after that point as well, on a machine with >= 1GB RAM.

There was some kind of glitch with apt and I had to repeat the update/install steps because of a transient error.

# doctl compute droplet create ctest --region sfo2 --image ubuntu-19-10-x64 --size 512mb --ssh-keys 53:c9:3f:8b:6d:d5:76:37:94:b2:bf:17:9f:d0:1b:4d

# doctl compute droplet list
ID           Name     Public IPv4        Private IPv4    Public IPv6    Memory    VCPUs    Disk    Region    Image               Status    Tags    Features    Volumes
166017114    ctest    167.172.203.xxx                                   512       1        20      sfo2      Ubuntu 19.10 x64    new
# ssh root@167.172.203.xxx
The authenticity of host '167.172.203.xxx (167.172.203.xxx)' can't be established.
ECDSA key fingerprint is SHA256:/LPgYHq/qUdxLew9gTYhLtfcedzyF1NFo/r2x74ZwAQ.
Are you sure you want to continue connecting (yes/no)? yes 
Warning: Permanently added '167.172.203.xxx' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 19.10 (GNU/Linux 5.3.0-18-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Fri Nov  8 00:51:21 UTC 2019

  System load:  0.38              Processes:           99
  Usage of /:   6.0% of 19.21GB   Users logged in:     0
  Memory usage: 36%               IP address for eth0: 167.172.203.xxx
  Swap usage:   0%

0 updates can be installed immediately.
0 of these updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@ctest:~# sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu eoan-security InRelease [97.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu eoan InRelease [255 kB]
Get:3 http://security.ubuntu.com/ubuntu eoan-security/main amd64 Packages [45.5 kB]
Get:4 http://security.ubuntu.com/ubuntu eoan-security/main Translation-en [18.5 kB]                                            
Get:5 http://archive.ubuntu.com/ubuntu eoan-updates InRelease [97.5 kB]                                 
Get:6 http://security.ubuntu.com/ubuntu eoan-security/main amd64 c-n-f Metadata [876 B]                             
Get:7 http://security.ubuntu.com/ubuntu eoan-security/restricted amd64 Packages [3456 B]
Get:8 http://security.ubuntu.com/ubuntu eoan-security/restricted Translation-en [996 B]                 
Get:9 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 Packages [12.2 kB]                 
Get:10 http://archive.ubuntu.com/ubuntu eoan-backports InRelease [88.8 kB]                        
Get:11 http://security.ubuntu.com/ubuntu eoan-security/universe Translation-en [8672 B] 
Get:12 http://security.ubuntu.com/ubuntu eoan-security/universe amd64 c-n-f Metadata [588 B]                          
Get:13 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 Packages [1164 B]                             
Get:14 http://security.ubuntu.com/ubuntu eoan-security/multiverse Translation-en [632 B]                
Get:15 http://archive.ubuntu.com/ubuntu eoan/main amd64 Packages [969 kB]   
Get:16 http://security.ubuntu.com/ubuntu eoan-security/multiverse amd64 c-n-f Metadata [116 B]
Get:17 http://archive.ubuntu.com/ubuntu eoan/main amd64 c-n-f Metadata [29.7 kB]      
Get:18 http://archive.ubuntu.com/ubuntu eoan/universe amd64 Packages [8798 kB]      
Get:19 http://archive.ubuntu.com/ubuntu eoan/universe Translation-en [5198 kB]
Get:20 http://archive.ubuntu.com/ubuntu eoan/universe amd64 c-n-f Metadata [271 kB]                                                                                                                                              
Get:21 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 Packages [153 kB]                                                                                                                                                  
Get:22 http://archive.ubuntu.com/ubuntu eoan/multiverse Translation-en [111 kB]                                                                                                                                                  
Get:23 http://archive.ubuntu.com/ubuntu eoan/multiverse amd64 c-n-f Metadata [9424 B]                                                                                                                                            
Get:24 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages [48.7 kB]                                                                                                                                               
Get:25 http://archive.ubuntu.com/ubuntu eoan-updates/main Translation-en [19.9 kB]                                                                                                                                               
Get:26 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 c-n-f Metadata [892 B]                                                                                                                                           
Get:27 http://archive.ubuntu.com/ubuntu eoan-updates/restricted amd64 Packages [3456 B]                                                                                                                                          
Get:28 http://archive.ubuntu.com/ubuntu eoan-updates/restricted Translation-en [996 B]                                                                                                                                           
Get:29 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 Packages [17.4 kB]                                                                                                                                           
Get:30 http://archive.ubuntu.com/ubuntu eoan-updates/universe Translation-en [10.7 kB]                                                                                                                                           
Get:31 http://archive.ubuntu.com/ubuntu eoan-updates/universe amd64 c-n-f Metadata [588 B]                                                                                                                                       
Get:32 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 Packages [1164 B]                                                                                                                                          
Get:33 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse Translation-en [632 B]                                                                                                                                           
Get:34 http://archive.ubuntu.com/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata [112 B]                                                                                                                                     
Get:35 http://archive.ubuntu.com/ubuntu eoan-backports/main amd64 c-n-f Metadata [108 B]                                                                                                                                         
Get:36 http://archive.ubuntu.com/ubuntu eoan-backports/restricted amd64 c-n-f Metadata [116 B]                                                                                                                                   
Get:37 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 Packages [2936 B]                                                                                                                                          
Get:38 http://archive.ubuntu.com/ubuntu eoan-backports/universe Translation-en [1268 B]                                                                                                                                          
Get:39 http://archive.ubuntu.com/ubuntu eoan-backports/universe amd64 c-n-f Metadata [112 B]                                                                                                                                     
Get:40 http://archive.ubuntu.com/ubuntu eoan-backports/multiverse amd64 c-n-f Metadata [116 B]                                                                                                                                   
Fetched 16.3 MB in 15s (1122 kB/s)                                                                                                                                                                                               
Reading package lists... Done
root@ctest:~# sudo apt-get install --no-install-recommends \
>     git build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev \
>     libsqlite3-dev libffi-dev libxml2-dev libxslt1-dev libre2-dev pkg-config
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package libxml2-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libssl-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package build-essential
E: Package 'libssl-dev' has no installation candidate
E: Unable to locate package zlib1g-dev
E: Unable to locate package libbz2-dev
E: Unable to locate package libreadline-dev
E: Unable to locate package libsqlite3-dev
E: Unable to locate package libffi-dev
E: Package 'libxml2-dev' has no installation candidate
E: Unable to locate package libre2-dev
E: Unable to locate package pkg-config
root@ctest:~# apt-get update
Get:1 http://mirrors.digitalocean.com/ubuntu eoan InRelease [255 kB]
Get:2 http://mirrors.digitalocean.com/ubuntu eoan-updates InRelease [97.5 kB]    
Hit:3 http://security.ubuntu.com/ubuntu eoan-security InRelease
Get:4 http://mirrors.digitalocean.com/ubuntu eoan-backports InRelease [88.8 kB]
Get:5 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 Packages [969 kB]
Get:6 http://mirrors.digitalocean.com/ubuntu eoan/main Translation-en [505 kB]
Get:7 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 c-n-f Metadata [29.7 kB]
Get:8 http://mirrors.digitalocean.com/ubuntu eoan/restricted amd64 Packages [16.3 kB]
Get:9 http://mirrors.digitalocean.com/ubuntu eoan/restricted Translation-en [5196 B]
Get:10 http://mirrors.digitalocean.com/ubuntu eoan/restricted amd64 c-n-f Metadata [364 B]
Get:11 http://mirrors.digitalocean.com/ubuntu eoan/universe amd64 Packages [8798 kB]
Get:12 http://mirrors.digitalocean.com/ubuntu eoan/universe Translation-en [5198 kB]
Get:13 http://mirrors.digitalocean.com/ubuntu eoan/universe amd64 c-n-f Metadata [271 kB]
Get:14 http://mirrors.digitalocean.com/ubuntu eoan/multiverse amd64 Packages [153 kB]
Get:15 http://mirrors.digitalocean.com/ubuntu eoan/multiverse Translation-en [111 kB]
Get:16 http://mirrors.digitalocean.com/ubuntu eoan/multiverse amd64 c-n-f Metadata [9424 B]
Get:17 http://mirrors.digitalocean.com/ubuntu eoan-updates/main amd64 Packages [48.4 kB]
Get:18 http://mirrors.digitalocean.com/ubuntu eoan-updates/main Translation-en [19.7 kB]
Get:19 http://mirrors.digitalocean.com/ubuntu eoan-updates/main amd64 c-n-f Metadata [892 B]
Get:20 http://mirrors.digitalocean.com/ubuntu eoan-updates/restricted amd64 Packages [3456 B]
Get:21 http://mirrors.digitalocean.com/ubuntu eoan-updates/restricted Translation-en [996 B]
Get:22 http://mirrors.digitalocean.com/ubuntu eoan-updates/restricted amd64 c-n-f Metadata [112 B]
Get:23 http://mirrors.digitalocean.com/ubuntu eoan-updates/universe amd64 Packages [17.2 kB]
Get:24 http://mirrors.digitalocean.com/ubuntu eoan-updates/universe Translation-en [10.6 kB]
Get:25 http://mirrors.digitalocean.com/ubuntu eoan-updates/universe amd64 c-n-f Metadata [588 B]
Get:26 http://mirrors.digitalocean.com/ubuntu eoan-updates/multiverse amd64 Packages [1164 B]
Get:27 http://mirrors.digitalocean.com/ubuntu eoan-updates/multiverse Translation-en [632 B]
Get:28 http://mirrors.digitalocean.com/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata [112 B]
Get:29 http://mirrors.digitalocean.com/ubuntu eoan-backports/main amd64 c-n-f Metadata [108 B]
Get:30 http://mirrors.digitalocean.com/ubuntu eoan-backports/restricted amd64 c-n-f Metadata [116 B]
Get:31 http://mirrors.digitalocean.com/ubuntu eoan-backports/universe amd64 Packages [2936 B]
Get:32 http://mirrors.digitalocean.com/ubuntu eoan-backports/universe Translation-en [1268 B]
Get:33 http://mirrors.digitalocean.com/ubuntu eoan-backports/universe amd64 c-n-f Metadata [112 B]
Get:34 http://mirrors.digitalocean.com/ubuntu eoan-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 16.6 MB in 3s (5356 kB/s)                
Reading package lists... Done
root@ctest:~# sudo apt-get install --no-install-recommends     git build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev     libsqlite3-dev libffi-dev libxml2-dev libxslt1-dev libre2-dev pkg-config
Reading package lists... Done
Building dependency tree       
Reading state information... Done
git is already the newest version (1:2.20.1-2ubuntu1).
git set to manually installed.
The following additional packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu cpp cpp-9 dpkg-dev g++ g++-9 gcc gcc-9 icu-devtools libasan5 libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libdpkg-perl libgcc-9-dev libgomp1 libicu-dev libisl21
  libitm1 liblsan0 libmpc3 libncurses-dev libquadmath0 libre2-5 libstdc++-9-dev libtsan0 libubsan1 libxslt1.1 linux-libc-dev make
Suggested packages:
  binutils-doc cpp-doc gcc-9-locales debian-keyring g++-multilib g++-9-multilib gcc-9-doc libstdc++6-9-dbg gcc-multilib manpages-dev autoconf automake libtool flex bison gdb gcc-doc gcc-9-multilib libgcc1-dbg libgomp1-dbg
  libitm1-dbg libatomic1-dbg libasan5-dbg liblsan0-dbg libtsan0-dbg libubsan1-dbg libquadmath0-dbg glibc-doc bzr icu-doc ncurses-doc readline-doc sqlite3-doc libssl-doc libstdc++-9-doc make-doc
Recommended packages:
  fakeroot libalgorithm-merge-perl bzip2-doc manpages-dev libfile-fcntllock-perl
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-9 dpkg-dev g++ g++-9 gcc gcc-9 icu-devtools libasan5 libatomic1 libbinutils libbz2-dev libc-dev-bin libc6-dev libcc1-0 libdpkg-perl libffi-dev
  libgcc-9-dev libgomp1 libicu-dev libisl21 libitm1 liblsan0 libmpc3 libncurses-dev libquadmath0 libre2-5 libre2-dev libreadline-dev libsqlite3-dev libssl-dev libstdc++-9-dev libtsan0 libubsan1 libxml2-dev libxslt1-dev
  linux-libc-dev make pkg-config zlib1g-dev
The following packages will be upgraded:
  libxslt1.1
1 upgraded, 44 newly installed, 0 to remove and 18 not upgraded.
Need to get 56.1 MB of archives.
After this operation, 237 MB of additional disk space will be used.
Do you want to continue? [Y/n] y         
Get:1 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 binutils-common amd64 2.33-2ubuntu1 [203 kB]
Get:2 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libbinutils amd64 2.33-2ubuntu1 [474 kB]
Get:3 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 binutils-x86-64-linux-gnu amd64 2.33-2ubuntu1 [1639 kB]
Get:4 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 binutils amd64 2.33-2ubuntu1 [3400 B]
Get:5 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libc-dev-bin amd64 2.30-0ubuntu2 [71.6 kB]
Get:6 http://mirrors.digitalocean.com/ubuntu eoan-updates/main amd64 linux-libc-dev amd64 5.3.0-19.20 [1065 kB]
Get:7 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libc6-dev amd64 2.30-0ubuntu2 [2548 kB]
Get:8 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libisl21 amd64 0.21-2 [576 kB]
Get:9 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libmpc3 amd64 1.1.0-1 [40.8 kB]
Get:10 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 cpp-9 amd64 9.2.1-9ubuntu2 [8957 kB]
Get:11 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 cpp amd64 4:9.2.1-3.1ubuntu1 [27.6 kB]
Get:12 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libcc1-0 amd64 9.2.1-9ubuntu2 [49.1 kB]
Get:13 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libgomp1 amd64 9.2.1-9ubuntu2 [89.3 kB]
Get:14 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libitm1 amd64 9.2.1-9ubuntu2 [28.2 kB]
Get:15 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libatomic1 amd64 9.2.1-9ubuntu2 [9220 B]
Get:16 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libasan5 amd64 9.2.1-9ubuntu2 [395 kB]
Get:17 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 liblsan0 amd64 9.2.1-9ubuntu2 [139 kB]
Get:18 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libtsan0 amd64 9.2.1-9ubuntu2 [301 kB]
Get:19 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libubsan1 amd64 9.2.1-9ubuntu2 [130 kB]
Get:20 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libquadmath0 amd64 9.2.1-9ubuntu2 [146 kB]
Get:21 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libgcc-9-dev amd64 9.2.1-9ubuntu2 [2359 kB]
Get:22 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 gcc-9 amd64 9.2.1-9ubuntu2 [9763 kB]
Get:23 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 gcc amd64 4:9.2.1-3.1ubuntu1 [5216 B]
Get:24 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libstdc++-9-dev amd64 9.2.1-9ubuntu2 [1707 kB]
Get:25 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 g++-9 amd64 9.2.1-9ubuntu2 [10.2 MB]
Get:26 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 g++ amd64 4:9.2.1-3.1ubuntu1 [1612 B]
Get:27 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 make amd64 4.2.1-1.2 [162 kB]
Get:28 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libdpkg-perl all 1.19.7ubuntu2 [230 kB]
Get:29 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 dpkg-dev all 1.19.7ubuntu2 [679 kB]
Get:30 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 build-essential amd64 12.8ubuntu1 [4624 B]
Get:31 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 icu-devtools amd64 63.2-2 [180 kB]
Get:32 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libbz2-dev amd64 1.0.6-9.2 [30.2 kB]
Get:33 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libicu-dev amd64 63.2-2 [9178 kB]
Get:34 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libncurses-dev amd64 6.1+20190803-1ubuntu1 [338 kB]
Get:35 http://mirrors.digitalocean.com/ubuntu eoan/universe amd64 libre2-5 amd64 20190801+dfsg-1 [162 kB]
Get:36 http://mirrors.digitalocean.com/ubuntu eoan/universe amd64 libre2-dev amd64 20190801+dfsg-1 [225 kB]
Get:37 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libreadline-dev amd64 8.0-3 [141 kB]
Get:38 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libsqlite3-dev amd64 3.29.0-2 [680 kB]
Get:39 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libssl-dev amd64 1.1.1c-1ubuntu4 [1578 kB]
Get:40 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libxml2-dev amd64 2.9.4+dfsg1-7ubuntu3 [719 kB]
Get:41 http://mirrors.digitalocean.com/ubuntu eoan-updates/main amd64 libxslt1.1 amd64 1.1.33-0ubuntu1.1 [150 kB]
Get:42 http://mirrors.digitalocean.com/ubuntu eoan-updates/main amd64 libxslt1-dev amd64 1.1.33-0ubuntu1.1 [412 kB]
Get:43 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 pkg-config amd64 0.29.1-0ubuntu3 [45.2 kB]
Get:44 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-1ubuntu3 [172 kB]
Get:45 http://mirrors.digitalocean.com/ubuntu eoan/main amd64 libffi-dev amd64 3.2.1-9 [157 kB]
Fetched 56.1 MB in 2s (24.0 MB/s)
Extracting templates from packages: 100%
Selecting previously unselected package binutils-common:amd64.
(Reading database ... 61924 files and directories currently installed.)
Preparing to unpack .../00-binutils-common_2.33-2ubuntu1_amd64.deb ...
Unpacking binutils-common:amd64 (2.33-2ubuntu1) ...
Selecting previously unselected package libbinutils:amd64.
Preparing to unpack .../01-libbinutils_2.33-2ubuntu1_amd64.deb ...
Unpacking libbinutils:amd64 (2.33-2ubuntu1) ...
Selecting previously unselected package binutils-x86-64-linux-gnu.
Preparing to unpack .../02-binutils-x86-64-linux-gnu_2.33-2ubuntu1_amd64.deb ...
Unpacking binutils-x86-64-linux-gnu (2.33-2ubuntu1) ...
Selecting previously unselected package binutils.
Preparing to unpack .../03-binutils_2.33-2ubuntu1_amd64.deb ...
Unpacking binutils (2.33-2ubuntu1) ...
Selecting previously unselected package libc-dev-bin.
Preparing to unpack .../04-libc-dev-bin_2.30-0ubuntu2_amd64.deb ...
Unpacking libc-dev-bin (2.30-0ubuntu2) ...
Selecting previously unselected package linux-libc-dev:amd64.
Preparing to unpack .../05-linux-libc-dev_5.3.0-19.20_amd64.deb ...
Unpacking linux-libc-dev:amd64 (5.3.0-19.20) ...
Selecting previously unselected package libc6-dev:amd64.
Preparing to unpack .../06-libc6-dev_2.30-0ubuntu2_amd64.deb ...
Unpacking libc6-dev:amd64 (2.30-0ubuntu2) ...
Selecting previously unselected package libisl21:amd64.
Preparing to unpack .../07-libisl21_0.21-2_amd64.deb ...
Unpacking libisl21:amd64 (0.21-2) ...
Selecting previously unselected package libmpc3:amd64.
Preparing to unpack .../08-libmpc3_1.1.0-1_amd64.deb ...
Unpacking libmpc3:amd64 (1.1.0-1) ...
Selecting previously unselected package cpp-9.
Preparing to unpack .../09-cpp-9_9.2.1-9ubuntu2_amd64.deb ...
Unpacking cpp-9 (9.2.1-9ubuntu2) ...
Selecting previously unselected package cpp.
Preparing to unpack .../10-cpp_4%3a9.2.1-3.1ubuntu1_amd64.deb ...
Unpacking cpp (4:9.2.1-3.1ubuntu1) ...
Selecting previously unselected package libcc1-0:amd64.
Preparing to unpack .../11-libcc1-0_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libcc1-0:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package libgomp1:amd64.
Preparing to unpack .../12-libgomp1_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libgomp1:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package libitm1:amd64.
Preparing to unpack .../13-libitm1_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libitm1:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package libatomic1:amd64.
Preparing to unpack .../14-libatomic1_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libatomic1:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package libasan5:amd64.
Preparing to unpack .../15-libasan5_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libasan5:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package liblsan0:amd64.
Preparing to unpack .../16-liblsan0_9.2.1-9ubuntu2_amd64.deb ...
Unpacking liblsan0:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package libtsan0:amd64.
Preparing to unpack .../17-libtsan0_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libtsan0:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package libubsan1:amd64.
Preparing to unpack .../18-libubsan1_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libubsan1:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package libquadmath0:amd64.
Preparing to unpack .../19-libquadmath0_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libquadmath0:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package libgcc-9-dev:amd64.
Preparing to unpack .../20-libgcc-9-dev_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libgcc-9-dev:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package gcc-9.
Preparing to unpack .../21-gcc-9_9.2.1-9ubuntu2_amd64.deb ...
Unpacking gcc-9 (9.2.1-9ubuntu2) ...
Selecting previously unselected package gcc.
Preparing to unpack .../22-gcc_4%3a9.2.1-3.1ubuntu1_amd64.deb ...
Unpacking gcc (4:9.2.1-3.1ubuntu1) ...
Selecting previously unselected package libstdc++-9-dev:amd64.
Preparing to unpack .../23-libstdc++-9-dev_9.2.1-9ubuntu2_amd64.deb ...
Unpacking libstdc++-9-dev:amd64 (9.2.1-9ubuntu2) ...
Selecting previously unselected package g++-9.
Preparing to unpack .../24-g++-9_9.2.1-9ubuntu2_amd64.deb ...
Unpacking g++-9 (9.2.1-9ubuntu2) ...
Selecting previously unselected package g++.
Preparing to unpack .../25-g++_4%3a9.2.1-3.1ubuntu1_amd64.deb ...
Unpacking g++ (4:9.2.1-3.1ubuntu1) ...
Selecting previously unselected package make.
Preparing to unpack .../26-make_4.2.1-1.2_amd64.deb ...
Unpacking make (4.2.1-1.2) ...
Selecting previously unselected package libdpkg-perl.
Preparing to unpack .../27-libdpkg-perl_1.19.7ubuntu2_all.deb ...
Unpacking libdpkg-perl (1.19.7ubuntu2) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../28-dpkg-dev_1.19.7ubuntu2_all.deb ...
Unpacking dpkg-dev (1.19.7ubuntu2) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../29-build-essential_12.8ubuntu1_amd64.deb ...
Unpacking build-essential (12.8ubuntu1) ...
Selecting previously unselected package icu-devtools.
Preparing to unpack .../30-icu-devtools_63.2-2_amd64.deb ...
Unpacking icu-devtools (63.2-2) ...
Selecting previously unselected package libbz2-dev:amd64.
Preparing to unpack .../31-libbz2-dev_1.0.6-9.2_amd64.deb ...
Unpacking libbz2-dev:amd64 (1.0.6-9.2) ...
Selecting previously unselected package libicu-dev:amd64.
Preparing to unpack .../32-libicu-dev_63.2-2_amd64.deb ...
Unpacking libicu-dev:amd64 (63.2-2) ...
Selecting previously unselected package libncurses-dev:amd64.
Preparing to unpack .../33-libncurses-dev_6.1+20190803-1ubuntu1_amd64.deb ...
Unpacking libncurses-dev:amd64 (6.1+20190803-1ubuntu1) ...
Selecting previously unselected package libre2-5:amd64.
Preparing to unpack .../34-libre2-5_20190801+dfsg-1_amd64.deb ...
Unpacking libre2-5:amd64 (20190801+dfsg-1) ...
Selecting previously unselected package libre2-dev:amd64.
Preparing to unpack .../35-libre2-dev_20190801+dfsg-1_amd64.deb ...
Unpacking libre2-dev:amd64 (20190801+dfsg-1) ...
Selecting previously unselected package libreadline-dev:amd64.
Preparing to unpack .../36-libreadline-dev_8.0-3_amd64.deb ...
Unpacking libreadline-dev:amd64 (8.0-3) ...
Selecting previously unselected package libsqlite3-dev:amd64.
Preparing to unpack .../37-libsqlite3-dev_3.29.0-2_amd64.deb ...
Unpacking libsqlite3-dev:amd64 (3.29.0-2) ...
Selecting previously unselected package libssl-dev:amd64.
Preparing to unpack .../38-libssl-dev_1.1.1c-1ubuntu4_amd64.deb ...
Unpacking libssl-dev:amd64 (1.1.1c-1ubuntu4) ...
Selecting previously unselected package libxml2-dev:amd64.
Preparing to unpack .../39-libxml2-dev_2.9.4+dfsg1-7ubuntu3_amd64.deb ...
Unpacking libxml2-dev:amd64 (2.9.4+dfsg1-7ubuntu3) ...
Preparing to unpack .../40-libxslt1.1_1.1.33-0ubuntu1.1_amd64.deb ...
Unpacking libxslt1.1:amd64 (1.1.33-0ubuntu1.1) over (1.1.33-0ubuntu1) ...
Selecting previously unselected package libxslt1-dev:amd64.
Preparing to unpack .../41-libxslt1-dev_1.1.33-0ubuntu1.1_amd64.deb ...
Unpacking libxslt1-dev:amd64 (1.1.33-0ubuntu1.1) ...
Selecting previously unselected package pkg-config.
Preparing to unpack .../42-pkg-config_0.29.1-0ubuntu3_amd64.deb ...
Unpacking pkg-config (0.29.1-0ubuntu3) ...
Selecting previously unselected package zlib1g-dev:amd64.
Preparing to unpack .../43-zlib1g-dev_1%3a1.2.11.dfsg-1ubuntu3_amd64.deb ...
Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-1ubuntu3) ...
Selecting previously unselected package libffi-dev:amd64.
Preparing to unpack .../44-libffi-dev_3.2.1-9_amd64.deb ...
Unpacking libffi-dev:amd64 (3.2.1-9) ...
Setting up libisl21:amd64 (0.21-2) ...
Setting up binutils-common:amd64 (2.33-2ubuntu1) ...
Setting up linux-libc-dev:amd64 (5.3.0-19.20) ...
Setting up libgomp1:amd64 (9.2.1-9ubuntu2) ...
Setting up libffi-dev:amd64 (3.2.1-9) ...
Setting up libasan5:amd64 (9.2.1-9ubuntu2) ...
Setting up make (4.2.1-1.2) ...
Setting up libquadmath0:amd64 (9.2.1-9ubuntu2) ...
Setting up libssl-dev:amd64 (1.1.1c-1ubuntu4) ...
Setting up libmpc3:amd64 (1.1.0-1) ...
Setting up libatomic1:amd64 (9.2.1-9ubuntu2) ...
Setting up icu-devtools (63.2-2) ...
Setting up libdpkg-perl (1.19.7ubuntu2) ...
Setting up libubsan1:amd64 (9.2.1-9ubuntu2) ...
Setting up libxslt1.1:amd64 (1.1.33-0ubuntu1.1) ...
Setting up libre2-5:amd64 (20190801+dfsg-1) ...
Setting up cpp-9 (9.2.1-9ubuntu2) ...
Setting up libbinutils:amd64 (2.33-2ubuntu1) ...
Setting up libc-dev-bin (2.30-0ubuntu2) ...
Setting up libre2-dev:amd64 (20190801+dfsg-1) ...
Setting up libcc1-0:amd64 (9.2.1-9ubuntu2) ...
Setting up liblsan0:amd64 (9.2.1-9ubuntu2) ...
Setting up libitm1:amd64 (9.2.1-9ubuntu2) ...
Setting up binutils-x86-64-linux-gnu (2.33-2ubuntu1) ...
Setting up libtsan0:amd64 (9.2.1-9ubuntu2) ...
Setting up binutils (2.33-2ubuntu1) ...
Setting up dpkg-dev (1.19.7ubuntu2) ...
Setting up libgcc-9-dev:amd64 (9.2.1-9ubuntu2) ...
Setting up pkg-config (0.29.1-0ubuntu3) ...
Setting up cpp (4:9.2.1-3.1ubuntu1) ...
Setting up libc6-dev:amd64 (2.30-0ubuntu2) ...
Setting up libicu-dev:amd64 (63.2-2) ...
Setting up libbz2-dev:amd64 (1.0.6-9.2) ...
Setting up gcc-9 (9.2.1-9ubuntu2) ...
Setting up libncurses-dev:amd64 (6.1+20190803-1ubuntu1) ...
Setting up libstdc++-9-dev:amd64 (9.2.1-9ubuntu2) ...
Setting up libreadline-dev:amd64 (8.0-3) ...
Setting up gcc (4:9.2.1-3.1ubuntu1) ...
Setting up libxml2-dev:amd64 (2.9.4+dfsg1-7ubuntu3) ...
Setting up libsqlite3-dev:amd64 (3.29.0-2) ...
Setting up g++-9 (9.2.1-9ubuntu2) ...
Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-1ubuntu3) ...
Setting up g++ (4:9.2.1-3.1ubuntu1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.8ubuntu1) ...
Setting up libxslt1-dev:amd64 (1.1.33-0ubuntu1.1) ...
Processing triggers for man-db (2.8.7-3) ...
Processing triggers for install-info (6.6.0.dfsg.1-2ubuntu2) ...
Processing triggers for libc-bin (2.30-0ubuntu2) ...

# adduser grab
Adding user `grab' ...
Adding new group `grab' (1000) ...
Adding new user `grab' (1000) with group `grab' ...
Creating home directory `/home/grab' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
No password supplied
New password:
Retype new password:
No password supplied
New password:
Retype new password:
No password supplied
passwd: Authentication token manipulation error
passwd: password unchanged
Try again? [y/N]
Changing the user information for grab
Enter the new value, or press ENTER for the default
        Full Name []:
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n]
root@ctest:~#
root@ctest:~# su grab
grab@ctest:/root$ cd
grab@ctest:~$
grab@ctest:~$

grab@ctest:~$ wget https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer
--2019-11-08 00:58:58--  https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.188.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.188.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2454 (2.4K) [text/plain]
Saving to: ‘pyenv-installer’

pyenv-installer                                          100%[================================================================================================================================>]   2.40K  --.-KB/s    in 0s      

2019-11-08 00:58:58 (19.5 MB/s) - ‘pyenv-installer’ saved [2454/2454]

grab@ctest:~$ chmod +x pyenv-installer
grab@ctest:~$ ./pyenv-installer
Cloning into '/home/grab/.pyenv'...
remote: Enumerating objects: 660, done.
remote: Counting objects: 100% (660/660), done.
remote: Compressing objects: 100% (494/494), done.
remote: Total 660 (delta 329), reused 255 (delta 75), pack-reused 0
Receiving objects: 100% (660/660), 377.06 KiB | 7.25 MiB/s, done.
Resolving deltas: 100% (329/329), done.
Cloning into '/home/grab/.pyenv/plugins/pyenv-doctor'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 11 (delta 1), reused 3 (delta 0), pack-reused 0
Unpacking objects: 100% (11/11), done.
Cloning into '/home/grab/.pyenv/plugins/pyenv-installer'...
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 16 (delta 1), reused 8 (delta 0), pack-reused 0
Unpacking objects: 100% (16/16), done.
Cloning into '/home/grab/.pyenv/plugins/pyenv-update'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 10 (delta 1), reused 6 (delta 0), pack-reused 0
Unpacking objects: 100% (10/10), done.
Cloning into '/home/grab/.pyenv/plugins/pyenv-virtualenv'...
remote: Enumerating objects: 57, done.
remote: Counting objects: 100% (57/57), done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 57 (delta 11), reused 21 (delta 0), pack-reused 0
Unpacking objects: 100% (57/57), done.
Cloning into '/home/grab/.pyenv/plugins/pyenv-which-ext'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 10 (delta 1), reused 6 (delta 0), pack-reused 0
Unpacking objects: 100% (10/10), done.

WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="/home/grab/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

grab@ctest:~$ ~/.pyenv/bin/pyenv install 3.7.5
Downloading Python-3.7.5.tar.xz...
-> https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tar.xz
Installing Python-3.7.5...
Installed Python-3.7.5 to /home/grab/.pyenv/versions/3.7.5

grab@ctest:~$ ~/.pyenv/versions/3.7.5/bin/python -m venv ~/gs-venv 
grab@ctest:~$ ~/gs-venv/bin/pip install --no-binary lxml --upgrade git+https://github.com/ArchiveTeam/grab-site
Collecting git+https://github.com/ArchiveTeam/grab-site
  Cloning https://github.com/ArchiveTeam/grab-site to /tmp/pip-req-build-k5xkn_10
  Running command git clone -q https://github.com/ArchiveTeam/grab-site /tmp/pip-req-build-k5xkn_10

Collecting click>=6.3 (from grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
     |████████████████████████████████| 81kB 8.1MB/s 
Collecting wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7 from https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7 (from grab-site==2.1.17)
  Downloading https://github.com/ArchiveTeam/ludios_wpull/tarball/master
     / 819kB 1.1MB/s
Collecting manhole>=1.0.0 (from grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/f7/fe/32d484b370ec162c68f83d3efaead1ef84dee106b024b94d068cbea92cf1/manhole-1.6.0-py2.py3-none-any.whl
Collecting lmdb>=0.89 (from grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/c0/5c/d56dbc2532ecf14fa004c543927500c0f645eaca8bd7ec39420c7546396a/lmdb-0.98.tar.gz (869kB)
     |████████████████████████████████| 870kB 14.0MB/s 
Collecting autobahn>=0.12.1 (from grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/35/77/1bf9b3c4c0850fd6f00996afcf5e71e35295c145833f4e2430d0a7fddbaa/autobahn-19.11.1-py2.py3-none-any.whl (769kB)
     |████████████████████████████████| 778kB 29.4MB/s 
Collecting fb-re2>=1.0.6 (from grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/4b/39/e2f57ddd1c00c54e2a3a868d2ebb2c8674325e42308e88e2ae4842d03bc0/fb-re2-1.0.7.tar.gz
Collecting websockets>=6.0 (from grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/df/19/2573653fb823929324667c936541be610869b2b389eb9dabfe8c874ea80f/websockets-8.1-cp37-cp37m-manylinux1_x86_64.whl (73kB)
     |████████████████████████████████| 81kB 22.4MB/s 
Collecting cchardet>=1.0.0 (from grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/de/3e/9744f7aeed36226cd26101b26bdba890c2673f1956396b5c33e059785410/cchardet-2.1.4-cp37-cp37m-manylinux1_x86_64.whl (205kB)
     |████████████████████████████████| 215kB 30.8MB/s 
Collecting chardet (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
     |████████████████████████████████| 143kB 31.7MB/s 
Collecting dnspython (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/ec/d3/3aa0e7213ef72b8585747aa0e271a9523e713813b9a20177ebe1e939deb0/dnspython-1.16.0-py2.py3-none-any.whl (188kB)
     |████████████████████████████████| 194kB 13.1MB/s 
Collecting html5-parser (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/a6/78/6a80c13be6c02fd2fb615b473a6ceff5b28bafca184f428ab197f677fd7a/html5-parser-0.4.9.tar.gz (274kB)
     |████████████████████████████████| 276kB 12.7MB/s 
Collecting lxml (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/c4/43/3f1e7d742e2a7925be180b6af5e0f67d38de2f37560365ac1a0b9a04c015/lxml-4.4.1.tar.gz (4.5MB)
     |████████████████████████████████| 4.5MB 22.5MB/s 
Collecting namedlist (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/88/49/f7db251a949311c4f09f583e1b3c5a7e377220d5913607e6ab453446fe7e/namedlist-1.7.tar.gz
Collecting sqlalchemy (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/14/0e/487f7fc1e432cec50d2678f94e4133f2b9e9356e35bacc30d73e8cb831fc/SQLAlchemy-1.3.10.tar.gz (6.0MB)
     |████████████████████████████████| 6.0MB 30.7MB/s 
Collecting tornado==4.5.3 (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/e3/7b/e29ab3d51c8df66922fea216e2bddfcb6430fb29620e5165b16a216e0d3c/tornado-4.5.3.tar.gz (484kB)
     |████████████████████████████████| 491kB 13.9MB/s 
Collecting yapsy (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/tarball/master#egg=wpull-3.0.7->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/f1/4c/c771fbc77045a45678cdd78f57fd4006259767c363c1f884071debd200f7/Yapsy-1.12.2.tar.gz (83kB)
     |████████████████████████████████| 92kB 27.1MB/s 
Collecting txaio>=18.8.1 (from autobahn>=0.12.1->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/e9/6d/e1a6f7835cde86728e5bb1f577be9b2d7d273fdb33c286e70b087d418ded/txaio-18.8.1-py2.py3-none-any.whl
Collecting cryptography>=2.7 (from autobahn>=0.12.1->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/ca/9a/7cece52c46546e214e10811b36b2da52ce1ea7fa203203a629b8dfadad53/cryptography-2.8-cp34-abi3-manylinux2010_x86_64.whl (2.3MB)
     |████████████████████████████████| 2.3MB 35.4MB/s 
Collecting six>=1.11.0 (from autobahn>=0.12.1->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.7->autobahn>=0.12.1->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/16/cd/1f4ddf6be8300713c676bb9f3a2d3b8eb8accc0a6a24f57d4f6c4cd59d34/cffi-1.13.2-cp37-cp37m-manylinux1_x86_64.whl (398kB)
     |████████████████████████████████| 399kB 31.5MB/s 
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=2.7->autobahn>=0.12.1->grab-site==2.1.17)
  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
     |████████████████████████████████| 163kB 34.4MB/s 
Installing collected packages: click, chardet, dnspython, lxml, html5-parser, namedlist, sqlalchemy, tornado, yapsy, wpull, manhole, lmdb, six, txaio, pycparser, cffi, cryptography, autobahn, fb-re2, websockets, cchardet, grab-site

[let's just pretend the rest of this works]
ivan commented 4 years ago

https://github.com/ArchiveTeam/grab-site#install-on-another-distribution-lacking-python-37x the nix-based install can also be used on any Linux if the apt/pyenv/pip stuff is too tedious or non-working.

mkrzmr commented 4 years ago

thank you, I was able to set it up using nix.