Newznab Package - Githubissues

G1zm0 commented 12 years ago

This is related to #227, to keep a log. And ask for help if needed.

Step 1 to install is setting al Prerequisites

Enable php and webstation in DSM.
Do NOT enable register_globals
add :/opt/share/pear to php open_basedir
ipkg install php-curl
ipkg install php-pear
edit php.ini
(set max php memory limit to 256), won't know if this will work.
add pear to openbase_dir (include_path = ".:/php/includes:/opt/share/pear")

Result:

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

G1zm0 commented 12 years ago

Step 2 create the database

set it to local host and username is root, password is empty. like this

Result:

so far so good ;-)

G1zm0 commented 12 years ago

Step 3 News Server Setup -they recommend to use the provider they have a deal with, but you can use your own. Picture

Succes

Still... fingers crossed

G1zm0 commented 12 years ago

Step 4 Save Settings

Click and Go!

Step 5 Setup Admin User

Picture Just fill the boxes And Succes

Step 6 NZB File Path Picture

And we're done!

Now to the Admin Hangout and configure the rest

G1zm0 commented 12 years ago

Next you will want to get the latest headers:

cd newznab/misc/update_scripts
php update_binaries.php

But i'll get this error:

PHP Warning:  require_once(PEAR.php): failed to open stream: No such file or directory in /volume1/web/www/lib/Net_NNTP/NNTP/Protocol/Client.php on line 73
PHP Fatal error:  require_once(): Failed opening required 'PEAR.php' (include_path='.:') in /volume1/web/www/lib/Net_NNTP/NNTP/Protocol/Client.php on line 73

Looking into it at the moment

G1zm0 commented 12 years ago

Fixed this to use the full path: /usr/bin/php update_binaries.php

Now working!

G1zm0 commented 12 years ago

@Diaoul: Looks like it is working. It's slow, but it works. I'm getting a lot of time-outs from my server:

Error 205: Unexpected response: 'Timeout after 30 seconds, articles 0, bytes 0, groups 1, posts 0, postbytes 0'

but repeating the script will eventually get everything.

Furthermore, I dit not set it up to be accessed from outside, but only to test if it works, and it does!

Tested with my local provider retention of 30 days :-)

G1zm0 commented 12 years ago

I'm now backfilling 6 groups up to 30 days.... If my DS211J could sweat, it would ;-)

Diaoul commented 12 years ago

It works, I never said I wouldn't. It's just slow as hell with many groups. I think that's OK if you want a few groups but if you need TV + Movies + Music + Games then go buy a real server :P Could you post some performances here? The update scripts are outputing some interesting informations :)

G1zm0 commented 12 years ago

DS211J> /usr/bin/php backfill.php
Processing alt.binaries.hdtv.x264
INFO: daytopost finding post for alt.binaries.hdtv.x264 30 days back.
Total Articles: 1283034110
 Upper: 2108292735
 Lower: 825258625
 Goal: Sun, 26 Aug 2012 21:30:24 +0200 (1346009424)
DEBUG: Searching for postdate
 Goaldate: 1346009424 (Sun, 26 Aug 2012 21:30:24 +0200)
 Firstdate:  (n/a)
 Lastdate: 1348601242 (Tue, 25 Sep 2012 21:27:22 +0200)
Start: 825258625
 End: 2108292735
 Interval: 641517055
Set interval to 320758528 articles.
Set interval to 160379264 articles.
Set interval to 80189632 articles.
Set interval to 40094816 articles.
New upperbound (2068197919) is 24.6 days old.
Set interval to 20047408 articles.
Set interval to 10023704 articles.
Set interval to 5011852 articles.
New upperbound (2063186067) is 27.4 days old.
Set interval to 2505926 articles.
New upperbound (2060680141) is 28.8 days old.
Set interval to 1252963 articles.
New upperbound (2059427178) is 29.9 days old.
Set interval to 626482 articles.
Set interval to 313241 articles.
Set interval to 156621 articles.
New upperbound (2059270557) is 30 days old.
Set interval to 78311 articles.
Determined to be article 2059270557 which is 30 days old (Sun, 26 Aug 2012 22:14:41 +0200)
Group alt.binaries.hdtv.x264: server has 825258625 - 2108292735, or ~15608 days.
Local first = 2097973153 (6 days).  Backfill target of 30days is post 2059270557.
Getting 20000 parts (38682596 in queue)
Received 19978 articles of 20000 requested, 0 blacklisted, 0 not binaries
Server did not return article numbers 2097953845,2097954126,2097955041,2097955529,2097956355,2097961029,2097962617,2097962742,2097964403,2097964539,2097964540,2097965516,2097965553,2097965563,2097966368,2097966900,2097967024,2097968146,2097969071,2097970374,2097971246,2097972567
179 new, 0 updated, 19,978 parts. 7.04 headers, 93.69 update, 100.73 range.
Getting 20000 parts (38662596 in queue)

G1zm0 commented 12 years ago

This is were the connection times out:

DS211J> /usr/bin/php update_binaries.php

newznab 0.2.3 Copyright (C) 2012 newznab.com

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

Updating: 6 groups - Using compression? No
Processing alt.binaries.hdtv.x264
Group alt.binaries.hdtv.x264 has 15,787 new parts.
First: 825258625 Last: 2108292735 Local last: 2108276948
Getting 15,787 parts (2108276949 to 2108292735) - 0 in queue
Received 15772 articles of 15787 requested, 0 blacklisted, 0 not binaries
Server did not return article numbers 2108277422,2108278422,2108278719,2108278844,2108283161,2108288648,2108289541,2108289917,2108290296,2108290437,2108292096,2108292443,2108292621,2108292671,2108292676
93 new, 38 updated, 15,772 parts. 6.83 headers, 72.73 update, 79.56 range.
Error 205: Unexpected response: 'Timeout after 30 seconds, articles 0, bytes 0, groups 1, posts 0, postbytes 0'
Returning from postdate
Group processed in 80.15 seconds

Processing alt.binaries.multimedia
Could not select group (bad name?): alt.binaries.multimedia
Processing alt.binaries.teevee
Could not select group (bad name?): alt.binaries.teevee
Processing alt.binaries.tv
Could not select group (bad name?): alt.binaries.tv
Processing alt.binaries.tvseries
Could not select group (bad name?): alt.binaries.tvseries
Processing alt.binaries.x264
Could not select group (bad name?): alt.binaries.x264
Updating completed in 80.72 seconds

Repeating this will result give a proper result. It is only a slow machine....

DS211J> /usr/bin/php update_binaries.php

newznab 0.2.3 Copyright (C) 2012 newznab.com

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

Updating: 6 groups - Using compression? No
Processing alt.binaries.hdtv.x264
Attempting to repair 15 parts...
-repairing 2108277422 to 2108277422
Server did not return article numbers 2108277422
-repairing 2108278422 to 2108278422
Server did not return article numbers 2108278422
-repairing 2108278719 to 2108278719
Server did not return article numbers 2108278719
-repairing 2108278844 to 2108278844
Server did not return article numbers 2108278844
-repairing 2108283161 to 2108283161
Server did not return article numbers 2108283161
-repairing 2108288648 to 2108288648
Server did not return article numbers 2108288648
-repairing 2108289541 to 2108289541
Server did not return article numbers 2108289541
-repairing 2108289917 to 2108289917
Server did not return article numbers 2108289917
-repairing 2108290296 to 2108290296
Server did not return article numbers 2108290296
-repairing 2108290437 to 2108290437
Server did not return article numbers 2108290437
-repairing 2108292096 to 2108292096
Server did not return article numbers 2108292096
-repairing 2108292443 to 2108292443
Server did not return article numbers 2108292443
-repairing 2108292621 to 2108292621
Server did not return article numbers 2108292621
-repairing 2108292671 to 2108292671
Server did not return article numbers 2108292671
-repairing 2108292676 to 2108292676
Server did not return article numbers 2108292676
0 parts repaired.
Group alt.binaries.hdtv.x264 has 9,680 new parts.
First: 825258625 Last: 2108302415 Local last: 2108292735
Getting 9,680 parts (2108292736 to 2108302415) - 0 in queue
Received 9676 articles of 9680 requested, 0 blacklisted, 0 not binaries
Server did not return article numbers 2108293187,2108293188,2108298003,2108298222
33 new, 33 updated, 9,676 parts. 3.99 headers, 43.96 update, 47.96 range.
Error 205: Unexpected response: 'Timeout after 30 seconds, articles 0, bytes 0, groups 1, posts 0, postbytes 0'
Returning from postdate
Group processed in 49.53 seconds

Processing alt.binaries.multimedia
Could not select group (bad name?): alt.binaries.multimedia
Processing alt.binaries.teevee
Could not select group (bad name?): alt.binaries.teevee
Processing alt.binaries.tv
Could not select group (bad name?): alt.binaries.tv
Processing alt.binaries.tvseries
Could not select group (bad name?): alt.binaries.tvseries
Processing alt.binaries.x264
Could not select group (bad name?): alt.binaries.x264
Updating completed in 49.74 seconds
DS211J> /usr/bin/php update_binaries.php

newznab 0.2.3 Copyright (C) 2012 newznab.com

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

Updating: 6 groups - Using compression? No
Processing alt.binaries.hdtv.x264
Attempting to repair 19 parts...
-repairing 2108277422 to 2108277422
Server did not return article numbers 2108277422
-repairing 2108278422 to 2108278422
Server did not return article numbers 2108278422
-repairing 2108278719 to 2108278719
Server did not return article numbers 2108278719
-repairing 2108278844 to 2108278844
Server did not return article numbers 2108278844
-repairing 2108283161 to 2108283161
Server did not return article numbers 2108283161
-repairing 2108288648 to 2108288648
Server did not return article numbers 2108288648
-repairing 2108289541 to 2108289541
Server did not return article numbers 2108289541
-repairing 2108289917 to 2108289917
Server did not return article numbers 2108289917
-repairing 2108290296 to 2108290296
Server did not return article numbers 2108290296
-repairing 2108290437 to 2108290437
Server did not return article numbers 2108290437
-repairing 2108292096 to 2108292096
Server did not return article numbers 2108292096
-repairing 2108292443 to 2108292443
Server did not return article numbers 2108292443
-repairing 2108292621 to 2108292621
Server did not return article numbers 2108292621
-repairing 2108292671 to 2108292671
Server did not return article numbers 2108292671
-repairing 2108292676 to 2108292676
Server did not return article numbers 2108292676
-repairing 2108293187 to 2108293188
Server did not return article numbers 2108293187,2108293188
-repairing 2108298003 to 2108298003
Server did not return article numbers 2108298003
-repairing 2108298222 to 2108298222
Server did not return article numbers 2108298222
0 parts repaired.
No new records for alt.binaries.hdtv.x264 (first 2108302416 last 2108302348 total -67) grouplast 2108302415

Processing alt.binaries.multimedia
Group alt.binaries.multimedia has 7,900 new parts.
First: 615918213 Last: 1036396306 Local last: 1036388406
Getting 7,900 parts (1036388407 to 1036396306) - 0 in queue
Received 7898 articles of 7900 requested, 0 blacklisted, 2 not binaries
Server did not return article numbers 1036392671,1036393834
150 new, 120 updated, 7,896 parts. 6.33 headers, 37.49 update, 43.82 range.
Error 205: Unexpected response: 'Timeout after 30 seconds, articles 0, bytes 0, groups 2, posts 0, postbytes 0'
Returning from postdate
Group processed in 44.10 seconds

Processing alt.binaries.teevee
Could not select group (bad name?): alt.binaries.teevee
Processing alt.binaries.tv
Could not select group (bad name?): alt.binaries.tv
Processing alt.binaries.tvseries
Could not select group (bad name?): alt.binaries.tvseries
Processing alt.binaries.x264
Could not select group (bad name?): alt.binaries.x264
Updating completed in 45.83 seconds
DS211J> /usr/bin/php update_binaries.php

newznab 0.2.3 Copyright (C) 2012 newznab.com

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

Updating: 6 groups - Using compression? No
Processing alt.binaries.hdtv.x264
Attempting to repair 19 parts...
-repairing 2108277422 to 2108277422
Server did not return article numbers 2108277422
-repairing 2108278422 to 2108278422
Server did not return article numbers 2108278422
-repairing 2108278719 to 2108278719
Server did not return article numbers 2108278719
-repairing 2108278844 to 2108278844
Server did not return article numbers 2108278844
-repairing 2108283161 to 2108283161
Server did not return article numbers 2108283161
-repairing 2108288648 to 2108288648
Server did not return article numbers 2108288648
-repairing 2108289541 to 2108289541
Server did not return article numbers 2108289541
-repairing 2108289917 to 2108289917
Server did not return article numbers 2108289917
-repairing 2108290296 to 2108290296
Server did not return article numbers 2108290296
-repairing 2108290437 to 2108290437
Server did not return article numbers 2108290437
-repairing 2108292096 to 2108292096
Server did not return article numbers 2108292096
-repairing 2108292443 to 2108292443
Server did not return article numbers 2108292443
-repairing 2108292621 to 2108292621
Server did not return article numbers 2108292621
-repairing 2108292671 to 2108292671
Server did not return article numbers 2108292671
-repairing 2108292676 to 2108292676
Server did not return article numbers 2108292676
-repairing 2108293187 to 2108293188
Server did not return article numbers 2108293187,2108293188
-repairing 2108298003 to 2108298003
Server did not return article numbers 2108298003
-repairing 2108298222 to 2108298222
Server did not return article numbers 2108298222
0 parts repaired.
Group alt.binaries.hdtv.x264 has 17 new parts.
First: 825258625 Last: 2108302432 Local last: 2108302415
Getting 17 parts (2108302416 to 2108302432) - 0 in queue
Received 17 articles of 17 requested, 0 blacklisted, 0 not binaries
1 new, 6 updated, 17 parts. 0.21 headers, 0.10 update, 0.31 range.
Group processed in 2.00 seconds

Processing alt.binaries.multimedia
Attempting to repair 2 parts...
-repairing 1036392671 to 1036392671
Server did not return article numbers 1036392671
-repairing 1036393834 to 1036393834
Server did not return article numbers 1036393834
0 parts repaired.
Group alt.binaries.multimedia has 22 new parts.
First: 615918213 Last: 1036396328 Local last: 1036396306
Getting 22 parts (1036396307 to 1036396328) - 0 in queue
Received 22 articles of 22 requested, 0 blacklisted, 0 not binaries
0 new, 15 updated, 22 parts. 0.03 headers, 0.14 update, 0.17 range.
Group processed in 0.39 seconds

Processing alt.binaries.teevee
Could not select group (bad name?): alt.binaries.teevee
Processing alt.binaries.tv
Attempting to repair 3 parts...
-repairing 491453758 to 491453758
Server did not return article numbers 491453758
-repairing 491456333 to 491456333
Server did not return article numbers 491456333
-repairing 491457955 to 491457955
Server did not return article numbers 491457955
0 parts repaired.
Group alt.binaries.tv has 5,240 new parts.
First: 369767829 Last: 491465884 Local last: 491460644
Getting 5,240 parts (491460645 to 491465884) - 0 in queue
Received 5240 articles of 5240 requested, 0 blacklisted, 0 not binaries
39 new, 27 updated, 5,240 parts. 2.49 headers, 23.98 update, 26.47 range.
Group processed in 27.02 seconds

Processing alt.binaries.tvseries
No new records for alt.binaries.tvseries (first 98978441 last 98978440 total 0) grouplast 98978440

Processing alt.binaries.x264
Could not select group (bad name?): alt.binaries.x264
Updating completed in 29.81 seconds

G1zm0 commented 12 years ago

update_releases.php is to big to post. But is pretty fast

G1zm0 commented 12 years ago

Got it running for a time now, reworked the cronscript.

Picture

Diaoul commented 12 years ago

What changes did you do to the update scripts ?

G1zm0 commented 12 years ago

I replaced all the php commands to the full path and addded a rm -f command to remove the PID. but as I'm looking now it is still not working. I use this script:

#!/bin/sh
# do not forget to change NEWZNAB.. vars =)

set -e

export NEWZNAB_PATH="/volume1/web/misc/update_scripts"
export NEWZNAB_BINUP="update_binaries.php"
export NEWZNAB_RELUP="update_releases.php"
export NEWZNAB_SLEEP_TIME="10" # in seconds . 10sec is good for 100s of groups. 600sec might be a good start for fewer.
export NEWZNAB_PID_PATH="/volume1/web/" # don't forget the trailing slash . need r/w on it

export PATH="${PATH}:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin"
PIDFILE="newznab_binup.pid"

case "$1" in
  start)
    [ -f ${NEWZNAB_PID_PATH}${PIDFILE} ] && { echo "$0 is already running."; false; }
        echo -n "Starting Newznab binaries update"
        cd ${NEWZNAB_PATH}
        (while (true);do cd ${NEWZNAB_PATH} && /usr/bin/php ${NEWZNAB_BINUP}  2>&1 > /dev/null && /usr/bin/php ${NEWZNAB_RELUP}  2>&1 > /dev/null ; sleep ${NEWZNAB_SLEEP_TIME} ;done) &
        PID=`echo $!`
        echo $PID > ${NEWZNAB_PID_PATH}${PIDFILE}
        ;;
  stop)
        echo -n "Stopping Newznab binaries update"
        kill -9 `cat ${NEWZNAB_PID_PATH}${PIDFILE}`
        rm -f ${NEWZNAB_PID_PATH}${PIDFILE}
        ;;

  *)
        echo "Usage: $0 [start|stop]"
        exit 1
esac

After the script is done, the PID is still there and will not restart with my cronjob. stopping will not help, because it stops the script because there is no process to kill.

Maybe you have a suggestion??

Diaoul commented 12 years ago

What is the condition to exit the infinite loop? Because I don't see how the PID could even be written to the file.

G1zm0 commented 12 years ago

There is none, it keeps looping according tot the script. There is a PID-file written, I see it. But if you manually stop the script, it is not removed. maybe that's the problem....the PID is not written to the file???

G1zm0 commented 12 years ago

BTW, the script is part of the package you get if you download it. I'm not a real expert on this...

G1zm0 commented 12 years ago

makes it:

#!/bin/sh
# do not forget to change NEWZNAB.. vars =)

set -e

export NEWZNAB_PATH="/volume1/web/misc/update_scripts"
export NEWZNAB_BINUP="update_binaries.php"
export NEWZNAB_RELUP="update_releases.php"
export NEWZNAB_SLEEP_TIME="120" # in seconds . 10sec is good for 100s of groups. 600sec might be a good start for fewer.
export NEWZNAB_PID_PATH="/volume1/web/" # don't forget the trailing slash . need r/w on it

export PATH="${PATH}:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin"
PIDFILE="newznab_binup.pid"

case "$1" in
  start)
    [ -f ${NEWZNAB_PID_PATH}${PIDFILE} ] && { echo "$0 is already running."; false; }
        echo -n "Starting Newznab binaries update"
        PID=`echo $!`
        echo $PID > ${NEWZNAB_PID_PATH}${PIDFILE}
        cd ${NEWZNAB_PATH}
        (while (true);do cd ${NEWZNAB_PATH} && /usr/bin/php ${NEWZNAB_BINUP}  2>&1 > /dev/null && /usr/bin/php ${NEWZNAB_RELUP}  2>&1 > /dev/null ; sleep ${NEWZNAB_SLEEP_TIME} ;done) &
        ;;
  stop)
        echo -n "Stopping Newznab binaries update"
        kill -9 `cat ${NEWZNAB_PID_PATH}${PIDFILE}`
        rm -f ${NEWZNAB_PID_PATH}${PIDFILE}
        ;;

  *)
        echo "Usage: $0 [start|stop]"
        exit 1
esac

I'll have a look. Let you know tomorrow. Thanks

G1zm0 commented 12 years ago

BTW Working on this....I'm now up to 23 groups. This NAS is so slow, that you should not try it on a machine under 256MB RAM. You'll get time-outs and it will not work properly. If you make a spk, people will submit issues caused by ther slow machines. Maybe a restriction in arch or a disclaimer???

Diaoul commented 12 years ago

I tried on a Raspberry Pi which has a slow CPU too and didn't get any timeouts. Maybe use a different usenet provider? I heard Astraweb does the job with Newznab

G1zm0 commented 12 years ago

Hmmm that can be the cause, will have a try....

It is a great package though Cool :-)

G1zm0 commented 12 years ago

Tried again with another provider Did a reset of the DS211J and started from scratch. Worked much better! Here is the first output. This is a better performance indicator

DS211J> /usr/bin/php update_binaries.php

newznab 0.2.3 Copyright (C) 2012 newznab.com

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

Updating: 2 groups - Using compression? No
Processing alt.binaries.boneless
Group alt.binaries.boneless has 50,001 new parts.
First: 1579845 Last: 7289539780 Local last: 0
New group starting with 50000 messages worth.
Getting 20,001 parts (7289489780 to 7289509780) - 30,000 in queue
Received 20001 articles of 20001 requested, 0 blacklisted, 0 not binaries
500 bin adds...
512 new, 0 updated, 19,996 parts. 8.14 headers, 103.17 update, 111.34 range.
Getting 20,001 parts (7289509781 to 7289529781) - 9,999 in queue
Received 20001 articles of 20001 requested, 0 blacklisted, 0 not binaries
198 new, 361 updated, 20,000 parts. 9.25 headers, 102.45 update, 111.74 range.
Getting 9,999 parts (7289529782 to 7289539780) - 0 in queue
Received 9999 articles of 9999 requested, 0 blacklisted, 0 not binaries
105 new, 388 updated, 9,998 parts. 3.61 headers, 52.06 update, 55.67 range.
Group processed in 281.19 seconds

Processing alt.binaries.x264
Group alt.binaries.x264 has 50,001 new parts.
First: 70601095 Last: 88534109 Local last: 0
New group starting with 50000 messages worth.
Getting 20,001 parts (88484109 to 88504109) - 30,000 in queue
Received 20000 articles of 20001 requested, 0 blacklisted, 0 not binaries
Server did not return article numbers 88502626
201 new, 0 updated, 19,999 parts. 9.70 headers, 98.41 update, 108.13 range.
Getting 20,001 parts (88504110 to 88524110) - 9,999 in queue
Received 20001 articles of 20001 requested, 0 blacklisted, 0 not binaries
236 new, 1 updated, 20,001 parts. 7.16 headers, 98.71 update, 105.88 range.
Getting 9,999 parts (88524111 to 88534109) - 0 in queue
Received 9936 articles of 9999 requested, 0 blacklisted, 0 not binaries
Server did not return article numbers 88530855,88530856,88530857,88530858,88530859,88530860,88532505,88532506,88532507,88532508,88532509,88532510,88532511,88532512,88532513,88532514,88532515,88532516,88532517,88532518,88532519,88532520,88532521,88532522,88532523,88532524,88532525,88532526,88532527,88532528,88532529,88532530,88532531,88532532,88532533,88532534,88532535,88532536,88532537,88532538,88532539,88532540,88532541,88532542,88532543,88532544,88532545,88532546,88532547,88532548,88532549,88532550,88532551,88532552,88532553,88532554,88532555,88532556,88532557,88532558,88532559,88532560,88532561
325 new, 2 updated, 9,936 parts. 5.34 headers, 51.60 update, 56.94 range.
Group processed in 273.20 seconds

Updating completed in 554.70 seconds
DS211J>

G1zm0 commented 12 years ago

This is working much better, but is heavier for the NAS: Load

Diaoul commented 12 years ago

You bet

thatroots commented 12 years ago

I can help you out and test it on my 1512+.. just need some help on installation.

G1zm0 commented 12 years ago

How can I be of any assistance?

thatroots commented 12 years ago

It's all good, I managed to install , I just had an issue were I had php installed in /etc while installing pear, i was editing the wrong php.ini ipkg remove php -force-dependance , and it's all good.

I managed to install all of it and everything seem to work fine....load seem find...for the moment @G1zm0 Let me know if you need anything. @Diaoul Un grand must ..:) qui manque dans ta collection de spk,

G1zm0 commented 12 years ago

I'm wondering how long it takes to update the binairies. For a few groups that is. To get an idea about the performance.

thatroots commented 12 years ago

Here is a log of my first update:

php update_binaries.php

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Updating: 3 groups - Using compression? No Processing alt.binaries.boneless Group alt.binaries.boneless has 50,001 new parts. First: 1310406581 Last: 1430415431 Local last: 0 New group starting with 50000 messages worth. Getting 20,001 parts (1430365431 to 1430385431) - 30,000 in queue Received 20001 articles of 20001 requested, 0 blacklisted, 0 not binaries 500 bin adds... 512 new, 0 updated, 20,001 parts. 5.03 headers, 28.84 update, 33.87 range. Getting 20,001 parts (1430385432 to 1430405432) - 9,999 in queue Received 19778 articles of 20001 requested, 0 blacklisted, 0 not binaries Server did not return article numbers 1430403586,1430403589,1430403595(..........................)1430404344 123 new, 317 updated, 19,778 parts. 4.03 headers, 28.36 update, 32.39 range. Getting 9,999 parts (1430405433 to 1430415431) - 0 in queue Received 8181 articles of 9999 requested, 0 blacklisted, 0 not binaries Server did not return article numbers 1430408215,(............),1430415399,1430415420,1430415426,1430415427 100 new, 176 updated, 8,181 parts. 1.76 headers, 11.77 update, 13.53 range. Group processed in 80.60 seconds

Processing alt.binaries.movies.divx Group alt.binaries.movies.divx has 50,001 new parts. First: 260665631 Last: 371810873 Local last: 0 New group starting with 50000 messages worth. Getting 20,001 parts (371760873 to 371780873) - 30,000 in queue Received 20001 articles of 20001 requested, 0 blacklisted, 1 not binaries 216 new, 0 updated, 20,000 parts. 4.13 headers, 29.31 update, 33.44 range. Getting 20,001 parts (371780874 to 371800874) - 9,999 in queue Received 19989 articles of 20001 requested, 0 blacklisted, 1 not binaries Server did not return article numbers 371785354,371790742,371790743,371790744,371790745,371790746,371790747,371790748,371793791,371796560,371798180,371799544 213 new, 8 updated, 19,988 parts. 4.31 headers, 28.96 update, 33.27 range. Getting 9,999 parts (371800875 to 371810873) - 0 in queue Received 9928 articles of 9999 requested, 0 blacklisted, 0 not binaries Server did not return article numbers 371802145,371810711,(......),371810868,371810869,371810870,371810871,371810872 121 new, 8 updated, 9,928 parts. 1.15 headers, 14.67 update, 15.82 range. Group processed in 83.30 seconds

Processing alt.binaries.multimedia Group alt.binaries.multimedia has 50,001 new parts. First: 941392384 Last: 1061395904 Local last: 0 New group starting with 50000 messages worth. Getting 20,001 parts (1061345904 to 1061365904) - 30,000 in queue Received 20000 articles of 20001 requested, 0 blacklisted, 5 not binaries Server did not return article numbers 1061345951 360 new, 0 updated, 19,994 parts. 2.17 headers, 57.42 update, 59.59 range. Getting 20,001 parts (1061365905 to 1061385905) - 9,999 in queue Received 20001 articles of 20001 requested, 0 blacklisted, 3 not binaries 432 new, 29 updated, 19,998 parts. 2.60 headers, 29.49 update, 32.09 range. Getting 9,999 parts (1061385906 to 1061395904) - 0 in queue Received 9436 articles of 9999 requested, 0 blacklisted, 2 not binaries Server did not return article numbers 1061393139(....),1061395903 106 new, 39 updated, 9,434 parts. 1.11 headers, 14.26 update, 15.37 range. Group processed in 107.80 seconds

Updating completed in 272.26 seconds

Do you know if there is a way of scheduling updates?

G1zm0 commented 12 years ago

In the folder /misc/update_scripts/cron_scripts there is a script called newznab.sh This enters a loop to execute update_binairies.php and update_releases.php and than take a short break/pause

thatroots commented 12 years ago

my bad , I hadn't read the whole exchange...I'll set it up and let you know how it goes, what kind of scheduling did u use?

Diaoul commented 12 years ago

Anyways, 30 sec range is fine for 20k parts. I had 110 with my Raspberry Pi

thatroots commented 12 years ago

I ended up just changing the sleep time and NEWZNAB_PATH, the rest seemed ok

Did you add the script to crontab? @reboot?

thatroots commented 12 years ago

added also ,

rm -f ${NEWZNAB_PID_PATH}${PIDFILE}

had same issue as G1zm0

thatroots commented 12 years ago

I let it run during the night....no luck :(...the script doesn't seem to work...any ideas?

G1zm0 commented 12 years ago

If fixed this by timing the time it took to update. Than added this to the crontab. Starting update_binairies.php every 20 min and the update_releases.php 5 minutes before that.

But the problem is the script. If the process is stopped for some reason there is no process to kill and the PID-file is never removed.

I've had a friend update the script yesterday, but haven't had the time to test it.

#!/bin/sh
# do not forget to change NEWZNAB.. vars =)

set -e

export NEWZNAB_PATH="/usr/local/www/newznab/misc/update_scripts"
export NEWZNAB_BINUP="update_binaries.php"
export NEWZNAB_RELUP="update_releases.php"
export NEWZNAB_SLEEP_TIME="600" # in seconds . 10sec is good for 100s of groups. 600sec might be a good start for fewer.
export NEWZNAB_PID_PATH="/var/run/" # don't forget the trailing slash . need r/w on it

export PATH="${PATH}:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin"
PIDFILE="newznab_binup.pid"

case "$1" in
  start)
        if [ -f "${NEWZNAB_PID_PATH}${PIDFILE}" ]; then
            if (ps -p `cat "${NEWZNAB_PID_PATH}${PIDFILE}"` &> /dev/null); then
                echo "Process is already running!"
                exit 1
            else
                rm -f "${NEWZNAB_PID_PATH}${PIDFILE}"
            fi;
        fi;

        echo -n "Starting Newznab binaries update"
        cd ${NEWZNAB_PATH}
        (while (true);do cd ${NEWZNAB_PATH} && php ${NEWZNAB_BINUP}  2>&1 > /dev/null && php ${NEWZNAB_RELUP}  2>&1 > /dev/null ; sleep ${NEWZNAB_SLEEP_TIME} ;done) &
        PID=`echo $!`
        echo $PID > ${NEWZNAB_PID_PATH}${PIDFILE}
        ;;
  stop)
        echo -n "Stopping Newznab binaries update"
        kill -9 `cat ${NEWZNAB_PID_PATH}${PIDFILE}`
        ;;

  *)
        echo "Usage: $0 [start|stop]"
        exit 1
esac

If you start this every hour it should work. If it is already running -> no actions. If it stopped for some reason it will restart

thatroots commented 12 years ago

@Diaoul , Can we expect an spk for this, or should we keep going through G1zmo procedure? Thanks for all the efforts.

Diaoul commented 12 years ago

When i'll have some time I may work on this

thatroots commented 12 years ago

Thanks!!

theCheek commented 11 years ago

@G1zm0 Interesting work. Wanting to set this up on my 2411+ with 2GB memory. Would like to add my experience but slightly new to this. Still stuck on step 1: "add pear to openbase_dir (include_path = ".:/php/includes:/opt/share/pear")"

What do you mean by this? Is this in the Synology GUI or via the commandline? Thanks for your assistance

G1zm0 commented 11 years ago

It is part of the "edit php.ini" There is a section where "include_path" is specified. Not in the GUI, but in the commandline

G1zm0 commented 11 years ago

This "SPK-under-investigation" is becoming more interesting since the closing of more and more index sites.

theCheek commented 11 years ago

@G1zm0 thanks for the response. Ok so in my php.ini i found:

; open_basedir =

So i just remove the comment, and add the above so it reads:

open_basedir = include_path = ".:/php/includes:/opt/share/pear

is this correct? doesn't look right but then what do i know? :)

thatroots commented 11 years ago

@G1zm0 I second that

@theCheek without an spk , you will need to use php installed using ipkg to be able to use pear. so the php.ini is only editable through command line.

But we are waiting for @Diaoul to make us a clean spk without any manual config.

Diaoul commented 11 years ago

Don't wait, make it yourself ;) I have no time for a new big SPK. Maybe in a month or so.

theCheek commented 11 years ago

thanks @thatroots

yeh I figured i'd give it a go following @G1zm0 concise instructions. So far so good I have managed to get to the admin hangout following the install. Will keep you posted on performance of the DS2411+

Yep following nzbmatrix and newzbin closure..this will become a very popular project so an spk would be great. If you can point me in the right direction of a guide on how to do this, then let me know and i will have a bash myself.

G1zm0 commented 11 years ago

@thatroots, @Diaoul I must agree with Diaoul. It is an open source. Diaoul is aready spending a lot of time on this Read the https://github.com/SynoCommunity/spksrc README.rst for guidance to make your own SPK.

@theCheek, I'm looking forward to your experiences

thatroots commented 11 years ago

no problem, I'll start working on it.

@Diaoul I might bother you with a few questions Maybe @G1zm0 can help me out

Diaoul commented 11 years ago

I don't think that's possible to make a SPK out of that as it requires php-pear that's not available. I suggest you ask Synology for that extension and then we'll try to make a SPK for this.

JorenRapini commented 11 years ago

Hey everyone. I've got everything working on my DS411+ except for a cron job. Originally I tried editing /etc/crontab and adding update_binaries/update_releases to it to run on alternating hours, but it didn't seem to be working and once I rebooted my diskstation all of my changes to crontab were erased and set back to default synology settings.

I see that you're using g1zmo's newznab.sh script, but where and how are you adding that to the cron? Thanks all!

On a side note in regards to performance, I'm trying to pull in about 35 groups right now. Once it gets to about 30, I receive the "Could not select group (bad name?)" error even though my cpu/memory usage is only about 25%, so I'm going to see if reducing the amount of days of headers to grab will help next, maybe to 0.5 down from 1.5.

SynoCommunity / spksrc

Newznab Package #249