Cant load cfscrape, python

[notojak]

rtorrent 0.9.7 ruTorrent 3.8

After upgrading to the latest version of rutorrent, plugin does not load during startup Log info:

[07.04.2019 08:44:38] WebUI started. [07.04.2019 08:44:39] cloudflare: Cant load cfscrape, python

[Micdu70]

Hello @notojak I really don't know if it's a nice idea to include "cloudflare" plugin... I think it must be/stay a 3rd-party plugin. But to get this plugin to work, you need to have 'python' and 'cfscrape' python module installed. And If you don't want/need to use this plugin, just disable it. :D

[notojak]

yep... Python is installed ...

And If you don't want/need to use this plugin, just disable it. :D

Yes I did: I turned it off ;d

[Micdu70]

For 'cfscrape' installation, it's better to have pip command (e.g.: Debian => you need to install python-pip package via apt) and do pip install cfscrape Worked for me. :)

[notojak]

good recipe, @Micdu70 thanks

[Novik]

I rewrited this stuff. Corresponding code placed to the plugin _cloudflare. This plugin works as middleware - it catch every GET request and check - do we have troubles with a cloudflare. And if yes, then tried to make scrape and repeat request with a new cookies. For correct work you must 1) install python (I checked with 2.7, but I think, 3.x will works too) 2) install cfscrape with a pip 3) set $pathToExternals['python'] in the plugin's configuration to the correct value. 4) set environment variable PYTHONPATH for your web-server to the correct value.

Right now we do not have any reason to use this plugin.

[lell186]

Hi @notojak ,

Apologies in advance for bringing this up again and I realise you are not 'tech support' but I am unfamiliar with apache/web configuration and am struggling to get this working so hoping you or someone else is able to provide assistance. I am having trouble with autodl-irssi and believe this plugin could be my solution.

I am using:

• Ubuntu 16.04 LTS
• Python 2.7.12
• Apache/2.4.18 (Ubuntu)

I have cfscrape (cfscrape==1.9.7) installed through pip and have set $pathToExternals['python'] in the plugin's config file to the correct path to python ('/usr/bin/python' in my case). The bit I am struggling with is your last step - setting the environment variable in apache2. Is this something that needs to be input in my virtualhost config? I have pasted my current default config below:

DocumentRoot /var/www/html
<Directory />
  Options FollowSymLinks
  AllowOverride All
</Directory>
<Directory /var/www/html/>
  Options -Indexes FollowSymLinks MultiViews
  AllowOverride None
  Order allow,deny
  allow from all
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
  AllowOverride None
  Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
  Order allow,deny
  Allow from all
</Directory>

ErrorLog ${APACHE_LOG_DIR}/error.log

# Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
LogLevel warn

CustomLog ${APACHE_LOG_DIR}/access.log combined

<Location /rutorrent>
  AuthType Digest
  AuthName "rutorrent"
  AuthDigestDomain /var/www/html/rutorrent/ http://localhost/rutorrent
  AuthDigestProvider file
  AuthUserFile /etc/apache2/.htpasswd
  Require valid-user
  SetEnv R_ENV "/var/www/html/rutorrent"
</Location>
</VirtualHost>

<VirtualHost *:443>
ServerAdmin webmaster@localhost
ServerName localhost:443

SSLEngine on
SSLCertificateFile /etc/apache2/ssl/apache.pem

DocumentRoot /var/www/html/
<Directory />
  Options FollowSymLinks
  AllowOverride All
</Directory>
<Directory /var/www/html/>
  Options -Indexes FollowSymLinks MultiViews
  AllowOverride None
  Order allow,deny
  allow from all
</Directory>

ErrorLog ${APACHE_LOG_DIR}/error.log

# Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
LogLevel warn

CustomLog ${APACHE_LOG_DIR}/access.log combined

<Location /rutorrent>
  AuthType Digest
  AuthName "rutorrent"
  AuthDigestDomain /var/www/html/rutorrent/ http://localhost/rutorrent
  AuthDigestProvider file
  AuthUserFile /etc/apache2/.htpasswd
  Require valid-user
  SetEnv R_ENV "/var/www/html/rutorrent"
</Location>
</VirtualHost>

Any help you are able to provide would be really appreciated.

Kind regards,

Lell186

[Novik]

setting the environment variable in apache2. Is this something that needs to be input in my virtualhost config?

Add line

export PYTHONPATH="${PATH}:/where-you-have-python-modules-path"

To the /etc/apache2/envvars. Or use corresponding SetEnv for the location /rutorrent.

For obtain modules path type python and use command help('cfscrape'). For example for my system:

$ python
>>> help('cfscrape')

Help on package cfscrape:

NAME
    cfscrape

FILE
    /usr/local/aws/lib/python2.7/site-packages/cfscrape/__init__.py
...

I.e. required path is /usr/local/aws/lib/python2.7/site-packages.

Do not forget to restart apache after changes.

[Novik]

Or simple disable this plugin. This plugin is not used now and developed for future usage.

[lell186]

Or simple disable this plugin. This plugin is not used now and developed for future usage.

Hi Novik,

Thank you for trying to help again on this, I really appreciate it - it's a plugin I would actually like to use to get autodl-irssi working (cloudflare is blocking requests). Unfortunately, I have not yet been able to get it to work even with your detailed guide. I have tried restarting apache2 and restarting the VM (after making changes).

My /etc/apache2/envvars includes a new line at the bottom (I have tested this with and without the ${PATH}: part):

# Python / CFscrape
export PYTHONPATH="${PATH}:/usr/local/lib/python2.7/dist-packages"

My Virtualhost configs (80 and 443) now include the following under <Location /rutorrent> (I only tried this after playing with envvars):

#Python / CFscrape
  SetEnv PYTHONPATH="${PATH}:/usr/local/lib/python2.7/dist-packages"

My _cloudflare plugin config file includes: $pathToExternals['python'] = '/usr/bin/python';

Do I need to ensure any apache mods are enabled / disabled such as mod_python and mod_wsgi?

Kind regards,

Lell186

[Novik]

You don't need something like mod_python or mod_wsgi. Python is used by plugin as command. Try to run in command line sudo -u www-data PYTHONPATH=${PATH}:/usr/local/lib/python2.7/dist-packages /usr/bin/python -c "import cfscrape" and check output.

[lell186]

Hi Novik,

Thank you very much for the help; I now have the plugin working - well, it doesn't have that error at the very least. My problem was that 'requests' was not in the same directory (although it was in another directory) and so cfscrape would not work. The plugin, however, does not seem to resolve my cloudflare issues and so I will continue investigating the issue I am having personally.

Thanks again for your patience and help and I hope that this thread helps someone else in the future should they wish to get the plugin working themselves.

Kind regards,

Lell186

[gri1n]

Have you considered disabling the plugin by default if it's only developed for future usage?

[dausruddin]

Just in-case anyone got here like I did, install cloudscraper instead of cfscrape

sudo pip install cloudscraper

Reference: https://github.com/Novik/ruTorrent/wiki/Plugin_cloudflare

[snowdream]

@Novik How to set the environment variable(PYTHONPATH) in nginx?