scrapyard-backend is « not installed »

[NicolasWladyslaw]

I'm having a hard time trying to install scrapyard-backend v2.1 on Brave Browser Version 1.45.127 Chromium: 107.0.5304.110 (Build officiel) (64 bits) on Linux Mint 20.1 x86_64 Cinnamon.

I closed Brave Browser and followed your instructions in the readme.txt file. I didn't see any error messages, some dependencies were already installed but Scrapyard v2.1.3 complains that the backend is not installed.

I faced the same issue with scrapyard-helper a few weeks ago and managed to get it working by moving (or copying ?) scrapyard_helper.json from /home/nicolas/.config/google-chrome/NativeMessagingHosts/ to /home/nicolas/.config/BraveSoftware/Brave-Browser/NativeMessagingHosts. I also had to install pip and some python packages that were not in the distro repositories.

I therefore moved scrapyard_backend.json to /home/nicolas/.config/BraveSoftware/Brave-Browser/NativeMessagingHosts but it didn't do the trick. I used sudo pip check and installed the missing packages numpy and testresources ($ sudo pip install numpy testresources, not sure if those are actually needed by Scrapyard) and also updated beautifulsoup4, Clik and MarkupSafe ($ sudo pip install -U beautifulsoup4 Clik MarkupSafe). I've just realized it might have been Click that should have been updated so I did just that. There's quite a few oudated python packages on my system :

Package                  Version       Latest      Type 
------------------------ ------------- ----------- -----
bcrypt                   3.1.7         4.0.1       wheel
blinker                  1.4           1.5         wheel
Brotli                   1.0.7         1.0.9       wheel
cachetools               4.0.0         5.2.0       wheel
certifi                  2019.11.28    2022.9.24   wheel
chardet                  3.0.4         5.0.0       wheel
colorama                 0.4.3         0.4.6       wheel
cryptography             2.8           38.0.3      wheel
dbus-python              1.2.16        1.3.2       sdist
distro                   1.4.0         1.8.0       wheel
distro-info              0.23ubuntu1   1.0         wheel
duplicity                0.8.12.0      1.0.1       sdist
entrypoints              0.3           0.4         wheel
fasteners                0.14.1        0.18        wheel
feedparser               5.2.1         6.0.10      wheel
google-api-python-client 1.7.11        2.66.0      wheel
google-auth              1.5.1         2.14.1      wheel
google-auth-httplib2     0.0.3         0.1.0       wheel
grpcio                   1.16.1        1.51.0      wheel
html5lib                 1.0.1         1.1         wheel
httplib2                 0.14.0        0.21.0      wheel
idna                     2.8           3.4         wheel
ifaddr                   0.1.6         0.2.0       wheel
img2pdf                  0.3.3         0.4.4       sdist
keyring                  18.0.1        23.11.0     wheel
launchpadlib             1.10.13       1.10.18     wheel
lazr.restfulclient       0.14.2        0.14.5      wheel
lazr.uri                 1.0.3         1.0.6       sdist
lxml                     4.5.0         4.9.1       wheel
Mako                     1.1.0         1.2.4       wheel
Markdown                 3.1.1         3.4.1       wheel
monotonic                1.5           1.6         wheel
mutagen                  1.44.0        1.46.0      wheel
netaddr                  0.7.19        0.8.0       wheel
netifaces                0.10.4        0.11.0      wheel
notify2                  0.3           0.3.1       wheel
oauth2client             4.1.2         4.1.3       wheel
oauthlib                 3.1.0         3.2.2       wheel
packaging                20.3          21.3        wheel
paramiko                 2.6.0         2.12.0      wheel
pexpect                  4.6.0         4.8.0       wheel
pikepdf                  1.10.3+dfsg   6.2.4       wheel
Pillow                   7.0.0         9.3.0       wheel
pip                      20.0.2        22.3.1      wheel
protobuf                 3.6.1         4.21.9      wheel
psutil                   5.5.1         5.9.4       wheel
pyasn1                   0.4.2         0.4.8       wheel
pyasn1-modules           0.2.1         0.2.8       wheel
pycairo                  1.16.2        1.22.0      sdist
pycryptodomex            3.6.1         3.15.0      wheel
pycups                   1.9.73        2.0.1       sdist
pycurl                   7.43.0.2      7.45.1      sdist
Pygments                 2.3.1         2.13.0      wheel
PyGObject                3.36.0        3.42.2      sdist
PyICU                    2.4.2         2.10.2      sdist
PyJWT                    1.7.1         2.6.0       wheel
PyNaCl                   1.3.0         1.5.0       wheel
pyOpenSSL                19.0.0        22.1.0      wheel
pyparsing                2.4.6         3.0.9       wheel
pyparted                 3.11.2        3.12.0      sdist
python-debian            0.1.36ubuntu1 0.1.49      wheel
python-magic             0.4.16        0.4.27      wheel
python-xlib              0.23          0.32        wheel
pytz                     2019.3        2022.6      wheel
pyxattr                  0.6.1         0.7.2       sdist
pyxdg                    0.26          0.28        wheel
PyYAML                   5.3.1         6.0         wheel
reportlab                3.5.34        3.6.12      wheel
requests                 2.22.0        2.28.1      wheel
requests-unixsocket      0.2.0         0.3.0       wheel
rsa                      4.0           4.9         wheel
SecretStorage            2.3.1         3.3.3       wheel
setproctitle             1.1.10        1.3.2       wheel
setuptools               45.2.0        65.6.0      wheel
simplejson               3.16.0        3.18.0      wheel
sip                      4.19.21       6.7.5       wheel
six                      1.14.0        1.16.0      wheel
soupsieve                1.9.5         2.3.2.post1 wheel
SQLAlchemy               1.3.12        1.4.44      wheel
tinycss2                 1.0.2         1.2.1       wheel
Unidecode                1.1.1         1.3.6       wheel
uritemplate              0.6           4.1.1       wheel
urllib3                  1.25.8        1.26.12     wheel
wadllib                  1.3.3         1.3.6       sdist
websockets               8.1           10.4        wheel
wheel                    0.34.2        0.38.4      wheel
wxPython                 4.0.7         4.2.0       sdist
zeroconf                 0.30.0        0.39.4      wheel

Scrapyard v2.1.3 still complains that the backend is not installed. Both scrapyard_backend.json and scrapyard_helper.json are in /home/nicolas/.config/BraveSoftware/Brave-Browser/NativeMessagingHosts, /home/nicolas/.config/google-chrome/NativeMessagingHosts/ and also in /home/nicolas/.config/chromium/NativeMessagingHosts/.

This means I can't use Scrapyard to save pages, what else should I do to finalize the installation of scrapyard-backend v2.1 ?

[GChristensen]

There is no scrapyard-backend package. This means that it is not installed. The full installation output may say more. May be wrong user or Python version/environment?

[NicolasWladyslaw]

This is the archive file I downloaded when clicking on "CLI-based installer (MacOS, Linux, Windows)" in the "Backend" section in Scrapyards settings and the extracted folder :

nicolas@T-Funkshun:~/Téléchargements$ ls
scrapyard-backend-2.1  scrapyard-backend-2.1.tgz

This is what I get when I execute install.sh :

nicolas@T-Funkshun:~/Téléchargements/scrapyard-backend-2.1$ ./install.sh
Processing ./scrapyard_backend
Requirement already satisfied: Flask in /home/nicolas/.local/lib/python3.8/site-packages (from scrapyard-backend==2.1) (2.2.2)
Requirement already satisfied: bs4 in /home/nicolas/.local/lib/python3.8/site-packages (from scrapyard-backend==2.1) (0.0.1)
Requirement already satisfied: regex in /home/nicolas/.local/lib/python3.8/site-packages (from scrapyard-backend==2.1) (2022.9.13)
Requirement already satisfied: Jinja2>=3.0 in /home/nicolas/.local/lib/python3.8/site-packages (from Flask->scrapyard-backend==2.1) (3.1.2)
Requirement already satisfied: importlib-metadata>=3.6.0; python_version < "3.10" in /home/nicolas/.local/lib/python3.8/site-packages (from Flask->scrapyard-backend==2.1) (5.0.0)
Requirement already satisfied: itsdangerous>=2.0 in /home/nicolas/.local/lib/python3.8/site-packages (from Flask->scrapyard-backend==2.1) (2.1.2)
Requirement already satisfied: Werkzeug>=2.2.2 in /home/nicolas/.local/lib/python3.8/site-packages (from Flask->scrapyard-backend==2.1) (2.2.2)
Requirement already satisfied: click>=8.0 in /home/nicolas/.local/lib/python3.8/site-packages (from Flask->scrapyard-backend==2.1) (8.1.3)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.8/dist-packages (from bs4->scrapyard-backend==2.1) (4.11.1)
Requirement already satisfied: MarkupSafe>=2.0 in /home/nicolas/.local/lib/python3.8/site-packages (from Jinja2>=3.0->Flask->scrapyard-backend==2.1) (2.1.1)
Requirement already satisfied: zipp>=0.5 in /home/nicolas/.local/lib/python3.8/site-packages (from importlib-metadata>=3.6.0; python_version < "3.10"->Flask->scrapyard-backend==2.1) (3.9.0)
Requirement already satisfied: soupsieve>1.2 in /usr/lib/python3/dist-packages (from beautifulsoup4->bs4->scrapyard-backend==2.1) (1.9.5)
Building wheels for collected packages: scrapyard-backend
  Building wheel for scrapyard-backend (setup.py) ... done
  Created wheel for scrapyard-backend: filename=scrapyard_backend-2.1-py3-none-any.whl size=122365 sha256=f750f07b163b8b1770c1002d5641486f84e394631f7135872c1b8f1de651c74a
  Stored in directory: /tmp/pip-ephem-wheel-cache-batlozml/wheels/37/e7/b0/e98d85b8eaa661d12f9c5fe75fbf4a35e37033542d3fcbd947
Successfully built scrapyard-backend
Installing collected packages: scrapyard-backend
  Attempting uninstall: scrapyard-backend
    Found existing installation: scrapyard-backend 2.1
    Uninstalling scrapyard-backend-2.1:
      Successfully uninstalled scrapyard-backend-2.1
Successfully installed scrapyard-backend-2.1

[GChristensen]

It seems that there are no errors. Somewhere in /home/nicolas/.local/bin should be a script named scrapyard-backend.sh or so. If it presents, and pip3 list lists the scrapyard-backend package, it is installed. The next move is to check the path to the scrapyard-backend.sh in scrapyard_helper.json If the script presents and paths are the right, this behavior is unexplainable without more context.

[GChristensen]

It seems, that the correct name of manifest is scrapyard_helper.json, scrapyard_backend.json is incorrect name. That may be the source of the error. It is necessary to rename scrapyard_backend.json obtained from the installation of 2.1 to scrapyard_helper.json since it is Brave, anyway. I'll fix the installer.

[NicolasWladyslaw]

I can't find any script named scrapyard-backend.sh in /home/nicolas/.local/bin. There is however a Python 3 script named scrapyard_backend. There's also a Python 3 script named scrapyard_helper. pip3 list doesn't list the scrapyard-backend package. Am I supposed to just wait for the new installer now ?

[GChristensen]

pip3 list doesn't list the scrapyard-backend package.

That's strange. Although if scrapyard_backend script produces no errors, all is fine (it should be interrupted with ctrl+c, though).

Am I supposed to just wait for the new installer now ?

If scrapyard_backend script produces no errors, all what is necessary to rename scrapyard_backend.json obtained from the installation of 2.1 to scrapyard_helper.json and place it into Brave native messaging hosts folder. Please check if the host path in the json file points to scrapyard_backend script.

[NicolasWladyslaw]

It's working now. Thanks a lot for your help ! Great extension by the way.