Is it possible to run this on android?

[Mewto7]

Thanks for such an awesome software. I am trying to run this on android. Is that possible? I tried it on QPython, and it installed just fine by running pip3 install wikitools3. But I have no idea what to do now. Thanks again.

[yzqzss]

wikitools3 is not mediawiki-scraper, and we haven't published mediawiki-scraper to pypi yet.

Is it possible to run this on android?

IKN.

[elsiehupp]

Hi @Mewto7—a previous user asked about running mediawiki-scraper/wikiteam3 on Android (using Termux), and you can find the discussion here.

I don't know the specific differences between QPython and Termux, but if you figure out how to use QPython, we can add the instructions for that, as well.

Here's a lightly edited version of the Termux instructions from the linked conversion:

On Android you can use Termux.

Termux uses Debian's apt package manager, but it uses a wrapper with the name pkg. One complication is that pip seems to be confused by the subtle differences behind the scenes, so you might get error messages saying you're missing various packages.

I installed Termux on an Android emulator on my laptop and ran through the steps myself, and installing these packages seems to make the rest of the installation work.

pkg update && pkg upgrade && pkg install git libxslt python

Note: When you're prompted to continue type Y, and when you're prompted to replace an existing configuration file type N.

At this point you should be able to continue with:

git clone https://github.com/mediawiki-client-tools/mediawiki-scraper.git

cd mediawiki-scraper

poetry update && poetry install && poetry build

pip install --force-reinstall dist/*.whl

Please let us know if you run into any issues!

(And we should probably add a blurb to the wikitools3 PyPI README explaining that it lacks a command-line interface, which can be found in mediawiki-scraper, as wikitools3 is a backend library, and mediawiki-scraper is a frontend that makes use of it.)

[yzqzss]

In fact, mediawiki-scarper replaced wikitools3 with mwclient. mediawiki-scarper now does not depend on wikitools3.

=_=...

[Mewto7]

wikitools3 is not mediawiki-scraper, and we haven't published mediawiki-scraper to pypi yet.

Is it possible to run this on android?

IKN.

My bad. Sorry about this. I thought that they were the same. I am not good at this.

[Mewto7]

Hi @Mewto7—a previous user asked about running mediawiki-scraper/wikiteam3 on Android (using Termux), and you can find the discussion here.

I don't know the specific differences between QPython and Termux, but if you figure out how to use QPython, we can add the instructions for that, as well.

Here's a lightly edited version of the Termux instructions from the linked conversion:

On Android you can use Termux. Termux uses Debian's apt package manager, but it uses a wrapper with the name pkg. One complication is that pip seems to be confused by the subtle differences behind the scenes, so you might get error messages saying you're missing various packages. I installed Termux on an Android emulator on my laptop and ran through the steps myself, and installing these packages seems to make the rest of the installation work.

pkg update && pkg upgrade && pkg install git libxslt python

Note: When you're prompted to continue type Y, and when you're prompted to replace an existing configuration file type N.

At this point you should be able to continue with:

git clone https://github.com/mediawiki-client-tools/mediawiki-scraper.git

cd mediawiki-scraper

poetry update && poetry install && poetry build

pip install --force-reinstall dist/*.whl

Please let us know if you run into any issues!

(And we should probably add a blurb to the wikitools3 PyPI README explaining that it lacks a command-line interface, which can be found in mediawiki-scraper, as wikitools3 is a backend library, and mediawiki-scraper is a frontend that makes use of it.)

Thank you so much for developing this software and the fast replay.

I love wikia and I want to make sure the hard work of everyone does not disappear. I will try it and let you know if I can make it work with QPython. Thanks again.

[Mewto7]

poetry update && poetry install && poetry build

pip install --force-reinstall dist/*.whl

I can not run these two commands.

The first one gives this error:

`~/mediawiki-scraper $ poetry update && poetry install && poetry build

No command poetry found, did you mean:

Command phear in package cavez-of-phear

Command gotty in package gotty

Command poly in package mesa-demos from the x11-repo repository

Command perl in package perl

Command pet in package pet

Command pinentry in package pinentry

Command pinentry in package pinentry-gtk from the x11-repo repository

Command poly in package polyml

Command povray in package povray

Command pterm in package putty from the x11-repo repository`

The second one this:

`~/mediawiki-scraper $ pip install --force-reinstall dist/*.whl

WARNING: Requirement 'dist/*.whl' looks like a filename, but the file does not exist

ERROR: *.whl is not a valid wheel filename.`

I am on Termux. Not sure what the problem is. Thanks for the help.

[yzqzss]

pip install poetry

[Mewto7]

pip install poetry

Hello there! Yes, I figured that would be it, so I already went to

https://pypi.org/project/poetry/

and I am running it. I am writing everything down so that I can post here. Thanks.

[Mewto7]

=============================DEBUG ASSISTANCE=============================

If you are seeing a compilation error please try the following steps to successfully install cryptography:

1) Upgrade to the latest pip and try again. This will fix errors for most users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip

2) Read https://cryptography.io/en/latest/installation/ for specific instructions for your platform.

3) Check our frequently asked questions for more information:

 https://cryptography.io/en/latest/faq/

4) Ensure you have a recent Rust toolchain installed: https://cryptography.io/en/latest/installation/#rust

Python: 3.11.3

platform: Linux-4.9.193-perf-g806aeec-armv7l-with-libc

pip: n/a

 setuptools: 67.7.2

setuptools_rust: 1.6.0

rustc: n/a

          =============================DEBUG ASSISTANCE=============================

error: can't find Rust compiler

 If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

To update pip, run:

pip install --upgrade pip

 and then retry package installation.

If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.

This package requires Rust >=1.48.0.

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

ERROR: Failed building wheel for cryptography

Successfully built dulwich msgpack rapidfuzz

Failed to build cryptography

ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects

Got that after running pip install poetry. Looks like it requires rust. Installing rust now.

Sorry about the formatting.

[Mewto7]

Building wheel for cryptography (pyproject.toml) ... 

Stuck there for half an hour.

Then this happened:

=============================DEBUG ASSISTANCE=============================      

  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --release -v --features 'pyo3/extension-module pyo3/abi3-py36' --crate-type cdylib --` failed with code 101                                        

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

ERROR: Failed building wheel for cryptography   Failed to build cryptography

ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects    

~ $ Y

No command Y found, did you mean:     

Command c in package c-script

Command [ in package coreutils         

Command k in package kona

Command o in package o-editor                     

Command q in package q-dns-client

Command X in package xorg-server from the x11-repo repository

Again, sorry for the formatting. It looks like the thread that you linked also mentions this issue. I am reading it throughly to see if I can fix it. Will keep this thread posted.

[elsiehupp]

IIRC if it's trying to compile cryptography from scratch it means something else is wrong. This support page explains it somewhat.

In fact, mediawiki-scraper replaced wikitools3 with mwclient. mediawiki-scarper now does not depend on wikitools3.

I just checked, and I now see that this is indeed the case. It looks like wikitools3 is only used in one of the secondary modules of mediawiki-scraper that isn't currently supported. (This shows how closely I'm paying attention lol.)

[Mewto7]

IIRC if it's trying to compile cryptography from scratch it means something else is wrong. This support page explains it somewhat.

In fact, mediawiki-scraper replaced wikitools3 with mwclient. mediawiki-scarper now does not depend on wikitools3.

I just checked, and I now see that this is indeed the case. It looks like wikitools3 is only used in one of the secondary modules of mediawiki-scraper that isn't currently supported. (This shows how closely I'm paying attention lol.)

Hello there! So this means that it can not be done yet? Thanks.

[elsiehupp]

IIRC if it's trying to compile cryptography from scratch it means something else is wrong. This support page explains it somewhat.

In fact, mediawiki-scraper replaced wikitools3 with mwclient. mediawiki-scarper now does not depend on wikitools3.

I just checked, and I now see that this is indeed the case. It looks like wikitools3 is only used in one of the secondary modules of mediawiki-scraper that isn't currently supported. (This shows how closely I'm paying attention lol.)

Hello there! So this means that it can not be done yet? Thanks.

Which "it" are you asking about?

Working with the person who opened the other issue, I was able to get wikiteam3 (now called mediawiki-scraper) working on Android, so it should be possible for you to scrape a MediaWiki wiki with it, if that's what you're looking to do.

As for wikitools3... it seems like you may have been confusing it with wikiteam3? You shouldn't need wikitools3 unless you're building a new Python project of your own (and using it as a dependency).

[Mewto7]

Working with the person who opened the other issue, I was able to get wikiteam3 (now called mediawiki-scraper) working on Android, so it should be possible for you to scrape a MediaWiki wiki with it, if that's what you're looking to do.

Hello there! I am talking about running mediawiki-scraper on Android. You managed to run it, so it clearly works.

The issue is that pip install poetry tells me that I need Rust and even after that it can not build cryptography. Termux is 17 months old on F-Droid, so I wonder if that is the issue. Thanks

[yzqzss]

Install poetry and build the .whl file on your PC, then copy it to Termux to install it.

---

PC: pip install poetry PC: cd <mediawiki-scraper cloned dir> PC: poetry install PC: poetry build ...copy dist/*.whl file to Termux... Termux: pip install <that.whl>

[Mewto7]

Hello. Thanks for the instructions. I get this when I run poetry install. Any suggestions ? Thanks

`F:\Soft\Wikiteam3\wikiteam3-python3>poetry install Installing dependencies from lock file

Because wikiteam3 depends on internetarchive (^3.1.0) which doesn't match any versions, version solving failed.`

Poetry build shows this

Building wikiteam3 (3.0.0)

• Building sdist INFO: Could not find files for the given pattern(s).
• Built wikiteam3-3.0.0.tar.gz
• Building wheel INFO: Could not find files for the given pattern(s).
• Built wikiteam3-3.0.0-py3-none-any.whl

[robkam]

Closing this, if it's still an issue please reopen.