search

Bartvelp / zim-converter

convert ZIM files, as found in the Kiwix WikiPedia library, to a SQLite database that is read by the WikiReader plugin of KOReader
15 stars 1 forks source link

Error in installing libzim #1

Closed Uwe-B closed 2 years ago

Uwe-B commented 2 years ago

Interesting project, tried to install it on 2 windows machines with the same error (win7 and win10):

pip install -r requirements.txt

Collecting libzim==2.0.0
  Using cached libzim-2.0.0.tar.gz (187 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ?─> [2 lines of output]
      [!] The libzim library cannot be found.
      Please verify it is correctly installed and can be found.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
?─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Any ideas how I can get this working?

Uwe-B commented 2 years ago

Edit: Guess libzim is using some precompiled libraries which are not provided for windows systems. So I probably have to give up.

Bartvelp commented 2 years ago

That is unfortunate, perhaps you need https://github.com/openzim/libzim but it seems not available for windows, you can use the docker image, that should work

You could also use Window Subsystem for Linux (WSL)

Uwe-B commented 2 years ago

Thanks, got it running via Docker and via WSL too. In Docker I wasn't able to get the created db file out of the image (probably me having not worked with Docker before) so I switched to WSL where everything was working fine. Now I started experimenting with larger zim files, since the smaller ones didn't fullfill my expectations but run around 17GB into an overflow error (but I guess this is another and completely different issue and the resulting db files are too big for the pocketbooks fat32 file system too): `multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "zim_converter.py", line 88, in process_range cursor.execute("INSERT OR REPLACE INTO articles VALUES(?, ?, ?)", [ OverflowError: BLOB longer than INT_MAX bytes """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "zim_converter.py", line 159, in convert_multithreaded(args, args.num_cores) File "zim_converter.py", line 119, in convert_multithreaded for task in results: File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next raise value OverflowError: BLOB longer than INT_MAX bytes`

Bartvelp commented 2 years ago

Good to hear you got it running, what file were you converting when the error occured? The converted file would also likely be too big for the fat32 filesystem of an Ereader SD card.

Uwe-B commented 2 years ago

That was the file: http://ftp.acc.umu.se/mirror/wikimedia.org/other/kiwix/zim/wikipedia/wikipedia_de_all_nopic_2022-08.zim Regarding filesize&Fat32: Yes I know, thinking about reactivating the old Run Kiwix on Pocketbooks project.

squromiv commented 8 months ago

it seems not available for windows

Still not, but we are waiting for. https://github.com/openzim/node-libzim/issues/113