search

sergiotapia / magnetissimo

Web application that indexes all popular torrent sites, and saves it to the local database.
MIT License
3k stars 190 forks source link

Documentation is incomplete or erroneous #83

Closed swatgoss closed 6 years ago

swatgoss commented 7 years ago

Hi,

In the page https://github.com/sergiotapia/magnetissimo/wiki/Usage:-Debian-7

We can read the package "exlixir" instead of "elixir", that prevents apt-get from running the installation.

E: Unable to locate package exlixir

Then later on the "npm install" step there is a problem because there are no package.json in the root directory of magnetissimo repo, so the command cannot complete and run as expected.

npm WARN enoent ENOENT: no such file or directory, open '~/magnetissimo/package.json' npm WARN magnetissimo No description npm WARN magnetissimo No repository field. npm WARN magnetissimo No README data npm WARN magnetissimo No license field.

Plus on a freshly created Debian Wheezy/7 AMD64 LXC, there are multiple assumptions that are not really true (like the presence of wget and curl in the base image), also, why use screen instead of tmux ?

Also, with esl-erlang from an ubuntu xenial (16.04) based server :

Package esl-erlang is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source

E: Package 'esl-erlang' has no installation candidate

The main problem is running "npm install" inside the cloned git repo, the phoenix server gives error when started and i can't seem to make the command run properly ...

coline-carle commented 7 years ago

The instructions are REALLY outdated First you need elixir. Two solution:

"install phoenix framework" google keywords should probably give you all the instructions you need.

magnetissimo is nodejs free, don't bother with npm it's no more required

swatgoss commented 7 years ago

Created a Debian 7 (LXD container inside a Ubuntu 16.04 host) : magnetissimo@MAGNETISSIMO:~/magnetissimo$ uname -a Linux MAGNETISSIMO 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 GNU/Linux magnetissimo@MAGNETISSIMO:~/magnetissimo$ cat /etc/issue Debian GNU/Linux 7 \n \l

Installed the packages, created the database, ran the mix commands, the mix phoenix.server gives me large errors like : 

[error] GenServer #PID<0.399.0> terminating
** (WithClauseError) no with clause matching: {:ok, %HTTPoison.Response{body: "<script type=\"text/javascript\">\n//<![CDATA[\ntry{if (!window.CloudFlare) {var CloudFlare=[{verbose:0,p:0,byc:0,owlid:\"cf\",bag2:1,mirage2:0,oracle:0,paths:{cloudflare:\"/cdn-cgi/nexp/dok3v=9eecb7db59/\"},atok:\"0bdeee8c30b76c9ff91e299ee2d7f47d\",petok:\"1d41856261666650c443487a32d07592e82e0082-1502742500-1800\",adblock:1,betok:\"ca2851ca24d9e4414a030f92aea6ce62c3400980-1502742500-120\",zone:\"limetorrents.cc\",apps:{}}];!function(a,b){a=document.createElement(\"script\"),b=document.getElementsByTagName(\"script\")[0],a.async=!0,a.src=\"//ajax.cloudflare.com/cdn-cgi/nexp/dok3v=85b614c0f6/cloudflare.min.js\",b.parentNode.insertBefore(a,b)}()}}catch(e){};\n//]]>\n</script>\n<script>var l=\"https://www.lime\"; var l2=\"torrents.cc\";window.location=l+''+l2;</script>", headers: [{"Date", "Mon, 14 Aug 2017 20:28:20 GMT"}, {"Content-Type", "text/html"}, {"Transfer-Encoding", "chunked"}, {"Connection", "keep-alive"}, {"Set-Cookie", "__cfduid=d09e2d6fbede249eeefb6714c8e76bbbe1502742500; expires=Tue, 14-Aug-18 20:28:20 GMT; path=/; domain=.limetorrents.cc; HttpOnly"}, {"Expires", "Mon, 14 Aug 2017 20:28:18 GMT"}, {"Last-Modified", "Mon, 14 Aug 2017 20:28:18 GMT"}, {"Pragma", "no-cache"}, {"Cache-Control", "no-cache, must-revalidate"}, {"Test", "11"}, {"Vary", "Accept-Encoding"}, {"Server", "cloudflare-nginx"}, {"CF-RAY", "38e6a8f47eed3bcf-CDG"}], status_code: 503}}
    (magnetissimo) lib/crawler/helper.ex:15: Magnetissimo.Crawler.Helper.download/1
    (magnetissimo) lib/crawler/limetorrents.ex:35: Magnetissimo.Crawler.LimeTorrents.process/2
    (magnetissimo) lib/crawler/limetorrents.ex:25: Magnetissimo.Crawler.LimeTorrents.handle_info/2
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :work
State: {[page_link: "https://www.limetorrents.cc/browse-torrents/Games/date/10/", page_link: "https://www.limetorrents.cc/browse-torrents/Other/date/10/", page_link: 
[...]
"https://www.limetorrents.cc/browse-torrents/Movies/date/6/"]}
[info] Application magnetissimo exited: shutdown

Is it a normal behavior for the server to print that its shutting down after a few seconds of runtime ?

coline-carle commented 7 years ago

it is, see #84

swatgoss commented 7 years ago

Thanks, it looks ok now !

After git checkout feature/general-cleanup mix ecto.migrate

I can run the phoenix server and it outputs the scan results in the terminal like it is supposed to !

swatgoss commented 7 years ago

In the case someone can benefit from it, installing Magnetissimo in a Debian 7 LXD container (named MAGNETISSIMO) :

Inspired from Outdated installation Wiki and #84

(remote) Host system

On the host system you can run a terminal emulator via :

sudo lxc exec MAGNETISSIMO /bin/bash

It will give you an interactive shell as the ROOT user of the container

Inside the container (or on a local, real, server)

System packages

In the LXD image repository, the Debian 7 system may not have wget or nano installed beforehand so we need to install it by running : (you can also use vim instead of nano)

apt-get update
apt-get install wget nano

Then we can fetch and install the erlang package :

wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
dpkg -i erlang-solutions_1.0_all.deb

It is possible to use gdebi instead of dpkg to install .deb package

Now it is time to install the needed services/packages (you can replace tmux by screen)

apt-get update
apt-get install postgresql redis-server esl-erlang elixir git tmux

Database setup

We need to set up the database user now to allow the usage of a normal user or root

su - postgres
psql

Set up the database named torrents, user torrent with password CHANGE_ME It is strongly advised to set your own database name, user name and password ...

CREATE USER torrent WITH PASSWORD 'CHANGE_ME';
CREATE DATABASE torrents OWNER torrent;
ALTER USER torrent CREATEDB;
\q

User setup

It is preferable to create a non-privileged user to run the project and use it from then Even in a container it is a good idea to have a distinct user running your services/programs Obviously you can choose any name, but here we will use magnetissimo

adduser magnetissimo
su magnetissimo

As an unprivileged user from now on

Clone the code

It is time to get the actual program you came here to get !

84 states to use the branch feature/general-cleanup for its stability

git clone -b feature/general-cleanup https://github.com/sergiotapia/magnetissimo.git
cd magnetissimo

Get the dependencies for magnetissimo !

mix deps.get

Configuration of magnetissimo

In the file ~/config/dev.exs you can specify the database parameters with a text editor Be sure to use the exact same database name torrents, user name torrent and password CHANGE_ME that you used before as the postgres database user !

nano ./config/dev.exs

config :magnetissimo, Magnetissimo.Repo,
adapter: Ecto.Adapters.Postgres,
database: "torrents",
username: "torrent",
password: "CHANGE_ME",
hostname: "localhost",
pool_size: 10

Apply the configuration and finish the magnetissimo setup

mix ecto.create
mix ecto.migrate

mix ecto.migrate should be re-run if you modify the server or database configuration to apply/update it

Launch the magnetissimo server now !

mix phoenix.server

It will start to scan websites and add them to your own database. You can connect a web browser to the port 4000 of the container to view the web interface.

Keeping the server alive

To keep the magnetissimo server alive when you disconnect from your host server you can run it inside a tmux session.

tmux
mix phoenix.server

To cleanly detach from your session, press CONTROL+B then press D (for Detach) You will return to the container shell - your magnetissimo will still be running in the background. To enter (Attach to) the tmux session again, you just need to run

tmux a
coline-carle commented 7 years ago

That's nice ! here are corrections you could make

The rest seems fine to me and could probably be applied to a bare metal installation

swatgoss commented 7 years ago

There is virtually no difference between a Debian 7 LXD and a bare metal server (except the default packages installed, LXD image is VERY lightweight).

Thanks for the advices, if you want me to rewrite a tutorial for Debian 7 i'll be happy to execute.

Also, i'm looking for some infos about a post you made last year, my actual installation is quite slow, and seems to only index from 1 source site at a time. Is there a documentation about running multiple (different) site crawlers in parallel to speed up things on my dedicated server ?

coline-carle commented 7 years ago

I'm not the author of the project, it interest me a lot, but the author seems quite busy right now, I'm just waiting he has enough time to start to contribute.

Yes, I think it would be a great guide, but I can speak for the author.

Actually all the crawler are spawn in parallel, each one for each website that's one of the main characteristic of elixir / otp. The thing is it's that merely one source that is up to date and working.

I made a PR #80 to help to keep the crawlers up to date, because it's the thing that make it unusable right now.

skwerlman commented 7 years ago

As a note, the wiki is publicly editable, so you can make corrections to it yourself

or not... i guess it's been set to maintainers only? it'd be good to let community members help out with documentation imo

tchoutri commented 6 years ago

Thanks for the feedback :) I'm closing this issue (for now!) because we're going to rewrite the documentation to take Erlang releases into account. Thanks for using Magnetissimo :)