search

ArchiveTeam / warrior-dockerfile

A Dockerfile for the ArchiveTeam Warrior
306 stars 57 forks source link

ModuleNotFoundError: No module named 'zstandard' #42

Closed yoursunny closed 3 years ago

yoursunny commented 3 years ago

I have official image version:

REPOSITORY                       TAG       IMAGE ID       CREATED       SIZE
archiveteam/warrior-dockerfile   latest    d01da7bdfc90   2 years ago   330MB

Many projects are failing to install due to this error:

2020-12-31 18:32:11,339 - seesaw.warrior - DEBUG - Select project mercurial
2020-12-31 18:32:11,340 - seesaw.warrior - DEBUG - Start selected project mercurial (reinstall=False)
2020-12-31 18:32:11,340 - seesaw.warrior - DEBUG - Install project mercurial
2020-12-31 18:32:11,341 - seesaw.warrior - DEBUG - git clone
2020-12-31 18:32:12,083 - seesaw.warrior - DEBUG - git operation: Cloning into '/home/warrior/projects/mercurial'...

2020-12-31 18:32:12,083 - seesaw.warrior - DEBUG - Install complete Cloning into '/home/warrior/projects/mercurial'...

2020-12-31 18:32:12,085 - seesaw.warrior - DEBUG - Result of the install process: True
2020-12-31 18:32:12,085 - seesaw.warrior - DEBUG - Clone project mercurial /home/warrior/projects/mercurial
2020-12-31 18:32:12,090 - seesaw.warrior - DEBUG - Cloning version 97da0f2
2020-12-31 18:32:12,161 - seesaw.warrior - DEBUG - Load pipeline /data/data/projects/mercurial-97da0f2/pipeline.py
2020-12-31 18:32:12,162 - seesaw.warrior - DEBUG - Pipeline has been read. Begin ConfigValue collection
2020-12-31 18:32:12,162 - seesaw.warrior - DEBUG - Executing pipeline
2020-12-31 18:32:12,165 - seesaw.warrior - ERROR - Error loading pipeline
Traceback (most recent call last):
  File "/home/warrior/warrior-code2/src/seesaw/seesaw/warrior.py", line 737, in start_selected_project
    pipeline_path, {"downloader": self.downloader})
  File "/home/warrior/warrior-code2/src/seesaw/seesaw/warrior.py", line 686, in load_pipeline
    exec(pipeline_str, local_context, global_context)
  File "<string>", line 29, in <module>
ModuleNotFoundError: No module named 'zstandard'
2020-12-31 18:32:12,166 - seesaw.warrior - WARNING - Project mercurial did not install correctly and we're ignoring this problem.

I tried building the image from source: (commit 956393e29b753cc8077608cd84514f5d88cd1700)

docker build -t archiveteam/warrior-dockerfile https://github.com/ArchiveTeam/warrior-dockerfile.git

But there are still similar errors:

fiergna commented 3 years ago

any news on this issue?

yoursunny commented 3 years ago

I found a workaround that enables some, but not all, projects.

  1. Select Warrior Extras Installer and wait for it to complete.

  2. Select "EU domains". It will fail to install.

  3. Enter the container with docker exec -it at-warrior /bin/bash and type the commands below.

    pip install requests warcio zstandard
pip3 install requests warcio zstandard
for D in /data/data/projects/*; do
  if ! [[ -x $D/wget-at ]]; then
    cp /opt/wget-at-1.20.3/bin/wget-at $D/wget-at
  fi
done

  4. Select "URLTeam 2", and then select "EU domains" again. It would install successfully this time.

The same procedure works for "Google Sites", "Flash Domains", and "GitHub". It doesn't work for "PasteBin".

fiergna commented 3 years ago

thanks for the quick reply. now i have another error:

2021-01-15 05:56:08,086 - seesaw.warrior - DEBUG - Executing pipeline 2021-01-15 05:56:08,133 - seesaw.warrior - ERROR - Error loading pipeline Traceback (most recent call last): File "/home/warrior/warrior-code2/src/seesaw/seesaw/warrior.py", line 737, in start_selected_project pipeline_path, {"downloader": self.downloader}) File "/home/warrior/warrior-code2/src/seesaw/seesaw/warrior.py", line 686, in load_pipeline exec(pipeline_str, local_context, global_context) File "<string>", line 53, in <module> Exception: No usable Wget+At found. 2021-01-15 05:56:08,133 - seesaw.warrior - WARNING - Project githubtest2 did not install correctly and we're ignoring this problem.

Edit copy command didn't work for me, hat to copy manually for D in /data/data/projects/*; do if ! [[ -x $D/wget-at ]]; then cp /opt/wget-at-1.20.3/bin/wget-at $D/wget-at fi

Freekers commented 3 years ago

This PR should fix these issues: https://github.com/ArchiveTeam/warrior-dockerfile/pull/44