Source plugin cannot find torrent

[jerryfreak]

Please complete the following tasks.

• [X] Web browser cache cleared
• [X] Copied everything if upgrading to v4.0 or master
• [X] Web browser, ruTorrent and PHP version provided

Tell us about your environment

browser (multiple) rutorrent 3.10 PHP 7.4 ubuntu 22.04 installed acording to arakasi72s script after installing PHP 7.4

returns "Source torrent for this download not found"

Give us detailed steps to reproduce the error

right click torrent > "Get .torrent"

Tell us the error message

Source torrent for this download not found

Provide the php error log contents (if applicable)

No response

[jerryfreak]

upon inspection, the source plugin version installed with the rtisnt script was 3.10

i tried copying the 4.0 files instead, but instead of giving me error, nothing at all happened when 'get .torrent' was attempted.

do i need to update another plugin as well to get 4.0 to work? wrt source/action.php, i'm not sure what the "send" added to lines 5, 19, and 61 are doing, nor what the "FileUtil" added to line 48 is doing (sorry, noob here)

wrt source/init.js, line 38 went from .load(function() to .on('load', function()

these are the only changes i see

i feel like this is hrder than it should be. whats the easy way to update plugins?

[stickz]

@jerryfreak You must copy and replace the entire ruTorrent installation to successfully upgrade to version 4.0. We made significant improvements to the web interference. The changes are not backwards compatible.

[jerryfreak]

thanks it may be time to upgrade

i was ultimtely able to solve the "get torrent" issue, it seemed to be a php issue, rebuilding OS from scratch worked

that said it may be time to upgrade to 4.0, as my rutorrent 3.10 just broke. was working fine at 6000 torrents but i added a few more hundred and now i get "bad response from server: (500 [error,list])".. although rtorrent is still chugging along.

how do i max out my memory usage. ive got 32GB but only 5GB is in use. i set it to 16000MB in .rtorrent.rc but i feel i need to increase it somewhere else in php configuration?

[stickz]

You don't want to use this feature in rTorrent if you have 32GB of ram. It's better to set it to 3GB and let the operating system do the heavy lifting. The Linux Caching mechanism is much more efficient than memory mapping on rTorrent.

I have a better script for you if you want speed gains. Reinstalls customized software versions optimized for speed. cd /etc/rtinst sudo wget https://raw.githubusercontent.com/stickz/rtinst/master/scripts/rtupdate sudo rtupdate

Revert any time with the following commands. Restore rtupdate back to default and run again. cd /etc/rtinst sudo wget https://raw.githubusercontent.com/arakasi72/rtinst/master/scripts/rtupdate sudo rtupdate

Also, add these optimizations to sysctl.conf. BBR congestion control, tcp fast-open and socket improvements. sudo nano /etc/sysctl.conf

# Some network settings
net.ipv4.tcp_fin_timeout = 20
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_congestion_control=bbr

Then save your changes. sudo sysctl -p

I hope this helps. These changes are highly stable. It's not required to use my whole script to run them.

[jerryfreak]

thanks, is that for 3.10 or 4.0?

i tried to replace 3.10 with 4.0 on a different test build. different machine entirely. getting same 500 error,list, but also says warning xmlrpc call has failed

i installed it with a script and really dont know which customized files to copy from the old install, thanks sorry for the noob questions

[stickz]

This is for the software supporting rTorrent on your operating system. It doesn't matter which version of ruTorrent.

You can run sudo rutupgrade and forcefully upgrade to master for v4.0. Run this command afterwards to fix the one plugin. git clone -q https://github.com/swizzin/autodl-rutorrent.git /var/www/rutorrent/plugins/autodl-irssi

[jerryfreak]

that rutupgrade broke the 3.10 in the same fashion

[26.12.2022 08:00:55] Bad response from server: (500 [error,list]) Warning: XMLRPC call is failed.
[26.12.2022 08:00:55] Error downloading files. Make sure autodl-irssi is started and configured properly (eg. password, port number): Could not get files listing: AJAX error, status: error, Exception info: Internal Server Error

attempting to fix autodl didnt work either (not too concerned i dont use it)

fatal: destination path '/var/www/rutorrent/plugins/autodl-irssi' already exists and is not an empty directory.

[stickz]

Do sudo rm -r /var/www/rutorrent/plugins/autodl-irssi first. Afterwards, run git clone again to resolve the problem.

git clone -q https://github.com/swizzin/autodl-rutorrent.git /var/www/rutorrent/plugins/autodl-irssi

[jerryfreak]

yeah i figured as much, changed the permissions to save autodl-irssi-old and ran the command- everything worked on the tester machine so i decided to give the main machine a try

although im completely confused. the tester machine says rutorrent4.0 in the tab but looks identical to 3.10 and all plugins are listed as 3.10 versions. lists 4.0 as the version when i run sudo rutupgrade

the main machine also lists 4.0 as the version when i run sudo rutupgrade, but the ui is totally borked, there is no list of torrents and the normally lower status bar is stuck at the top. cant connect to rtorrent at all.

logging in from a different machine on the network reveals an error which disappears almost immediately

restoring rutupgrade gets me backtgood ui (alas still stuck with the 500,list error)

not sure what else to do except manually go in and stop all torrents in terminal

[stickz]

You need to clear your web browser cache after upgrading to version 4.0. This step is critical for proper rendering!

[jerryfreak]

got it- thanks for your help had to rollback, fortunately i backed up my session folder yesterday and took it from 6900 torrents back to 6100 and its loading again (now under rutorrent 4.0, without errors). any other ideas on optimizing? - i expected it to maybe start to get glitchy not go from 100% functional to zero at some random number of torrents. 99.9% of these torrents are idle at a given time so im hoping this dedicated box can handle it.

[stickz]

If you're using my version of rtupdate it includes a custom software package for significant optimization. The only way you can further optimize is to upgrade to GCC 12. If you have a software raid that helps as well.

sudo apt install gcc-12 g++-12 Then run gcc --version. Once 12, do sudo rtupdate.

You can run my version of the rt daemon as well, if you want to load ruTorrent into memory. That will speed things up. rtupdate (my version) installed vmtouch for that. It takes about 30mb of ram.

rt stop cd /etc/rtinst sudo wget https://raw.githubusercontent.com/stickz/rtinst/master/scripts/rt rt start

These are more micro-optimizations though. It's going to be razor fast with my custom software solution and sysctl tweaks.

[jerryfreak]

sorry to get off-subject but this is the only help resource i can find

rutorrent crashes like clockwork when i get close to 6400 torrents. 6390 works 100%. add a few more and the whole thing falls apart

ive tested over and over, deleting then adding different torrents

any ideas what would cause this hard limit?

config.php says its supposed to be logging to /tmp/rutorrent_errors.log, but no such file exists

[stickz]

Yes it is a race condition with trackers. I patched that problem with a custom software update.

Did you successfully run my rtupdate script?

[jerryfreak]

i did but it looks like it punted

heres what it output

`$ sudo rtupdate

Checking File Repositories
xmlrpc-c: OK
rtorrent: OK
rtorrent 0.9.8 detected
`

rtorrent is still running, imagine it needs to be shut down. need to figure out a way to easily kill or not autostart the process on boot. whenever i stop using rt stop it tries to kill - 9 and fails and borks my session, sometimes sparking massive hash checks

i have another OS i can boot into and access the root partition, whats the easy way to toggle it off? just rename .rtorrent.rc so it is "missing"? I doubt i want to run that script from another instance

[jerryfreak]

ive tried your rtupdate script with rtorrent stopped. same output above seems to be hanging up at line 140 im not seeing any message about installing libtorrent or anything afterwards

same results running script as root

same results running on my tester 22.04 build

i tried sudo rtorrent -h on the tester build to see if there were any options for logging. somehow that made the script run.

output:

$ sudo rtupdate -h

Checking File Repositories
xmlrpc-c: OK
rtorrent: OK

Installing libtorrent-0.13.8/rtorrent-0.9.8
Fetching source files
######################################################################### 100.0%
######################################################################### 100.0%
Installing xmlrpc
Installing libtorrent
Installing rtorrent
Restarting rtorrent for <user>

not sure if it fully updated. i dont see "Installing libundns" which should be in the output above. I also see no metnions of vmtouch, c-ares, etc.

is there a logfile?

[stickz]

Nether of these runs are working properly for you. Try this command instead to run it directly from GitHub. sudo bash -c "$(wget --no-check-certificate -qO - https://raw.githubusercontent.com/stickz/rtinst/master/scripts/rtupdate)"

[jerryfreak]

script seemed to run ok rebooted still reliably fatally crashes at 6390 something torrents

[stickz]

Can try stop rtorrent with rt stop? Then open htop? Afterwards run rt start. I want to know if the CPU usage is maxed out during the startup process. This was the bottleneck I fixed where the thread time outs after 5 minutes.

[jerryfreak]

"restarting" was uneventful. i say "restarting" because it returned "no session lock file", and only displayed 214 of the 5600 torrents in there i started htop as soon as possible upon rebooting, it was spikey but not pinning any of (what i beleive) are the 8 cores numbered 1-8 above mem and swap?

is there a way to start it with a script upon bootup so i can see a few more seconds?

[stickz]

jerryfreak, if the CPU usage is not pinned at 100%, that is great. It means you successfully installed my custom software solution and it working properly. You will need to install the gdb, if you would like to debug this crash. sudo apt install gdb

1) Stop torrent with the rt command. You will need to manually launch it with the command gdb rtorrent.

2) After it's launched, you will see a terminal window. Type run to run your rtorrent program.

3) Once it crashes, type bt to display a back trace. Expand your terminal window all the way to see as much information as possible. Take a screenshot and send it to me. I require the information at the TOP of the stack trace, not the BOTTOM.

There's a 50% chance we'll be able to fix this. But at the very least, it will display helpful information about the crash.

[jerryfreak]

thanks for your help. i migrated 1000 torrents out to another client so it will be a few more days before i get back near 6400. i need a better way to stop rtorrent that lets it shut down properly. the kill command from stop rt always messes up the session

[jerryfreak]

ok ive got my machine up to 6300 torrents again, i have a bunch more to load, i cloned my drive of torrent data, and am sticking it in a fresh build with 22.04>arakasi script>rutupgrade>your script before i load any torrents

not afraid to break this one since its a backup. this machine only has a lowly celeron J4125 and 8GB of ram, though. its a bit sluggish with rutorrent connecting, but refreshing it does work and then reliably it crashed when approaching 6400 torrents, with the same list error

so i restored the session with 6300 torrents, and stopped and attempted to run under gdb, which really doesnt work. it only loads 188 of the 6300 torrents (188 is repeatable and exact) and there are errors in terminal like

(10:13:18) EE1BC62A253EFC0F7CBD57DFDCECCBE3F8CD9F32->file_list: Failed to prepar
e file '/xxxx.flac': Could not open file: Too many open files

and [Detaching after fork from child process xxxx]

i see the exact same behavior when running rtorrent in terminal absent gdb

killing terminal window and doing rt start yields same 188 torrents and rutorrent reports this error in the log

[04.01.2023 10:20:04] rTorrent user can't access 'id' program. Some functionality will be unavailable.

rebooting restores the 6300 torrent session

what are our other debugging options? seems to be a configuration issue as both machines reliably crash near 6400 torrents

[stickz]

I need to see the backtrace from gdb. When it stops please retrieve using the bt command. It's not a post to "work" using the GNU debugger program. This program provides information about "why" it's not working. These errors won't cause that happen.

[jerryfreak]

debugging in this instance is not useful as i cant reproduce the problem im attempting to solve

rtorrent doesnt behave anything like the normal process when i run it in a terminal window ( either through gdb run or even rtorrent or rt start commands). im too much of a novice to understand the difference in rtorrent running upon startup (as a service?) vs in a terminal window in regard to permissions and other differences

in the 6400+ torrent problem rtorrent keeps working fine but rutorrents ability to list torrents suddenly breaks

running rtorrent in a terminal (with or without gdb) results in in only loading a few hundred torrents on either machine (repeatably 182 on teh new machine, and 243 on the main machine... despite using identical session directories,), but rtorrent or rutorrent is not crashing in this case. i dont have the ability to manually try to add 6200 torrents to this instance without really messing up my session

if there was a way to log it doing normal startup and running, that would be more useful? im not aware of being able to run rtorrent in a terminal window when its already running - it treats it as a separate process and returns an error that the socket is already in use

[jerryfreak]

perhaps i should try flood instead of rutorrent to help isolate the problem between rtorrent and rutorrent?

can flood and rutorrent be installed concurrently?

[stickz]

No, the problem is rTorrent. Flood is even more complicated to run and won't resolve your issue. It's very easy to use gdb.

Stop the rtorrent program using rt stop then:

Step 1: gdb rtorrent Step 2: type run in the gdb shell. Step 3: wait for program to crash. Go back to gdb shell. Step 4: Then type bt in the gdb shell and paste the output here.

This is a computer science issue. If you don't want to debug the program, I can't help you. I'm very busy trying to release an rutorrent update. If we don't make progress on this issue soon, I'm going to have to close the issue report.

[jerryfreak]

did my best.

starting under gdb only loaded 200 or so of the 6300 torrents from the session

so i loaded the 600 or so i had, and then started dropping the (97% unloaded) torrent files from the session directory.

rtorrent itself crashed around 3300 torrents and yielded the following

#4  0x00007ffff73c0bc6 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#5  0x00007ffff73c0cf7 in BIO_write ()
   from /lib/x86_64-linux-gnu/libcrypto.so.3
#6  0x00007ffff7779e5e in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#7  0x00007ffff777c69b in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#8  0x00007ffff775032f in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#9  0x00007ffff775afb5 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#10 0x00007ffff7f10c88 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#11 0x00007ffff7efeb56 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#12 0x00007ffff7ef4d5a in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#13 0x00007ffff7ed8d98 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#14 0x00007ffff7edd758 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#15 0x00007ffff7edf781 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#16 0x00007ffff7edf95f in curl_multi_socket_action ()
   from /lib/x86_64-linux-gnu/libcurl.so.4
#17 0x00005555555f2ea1 in core::CurlStack::receive_action(core::CurlSocket*, int) ()
--Type <RET> for more, q to quit, c to continue without paging--c
#18 0x00005555555f3076 in core::CurlStack::receive_timeout() ()
#19 0x00005555555974d8 in client_perform() ()
#20 0x00007ffff7def04f in torrent::thread_base::event_loop(torrent::thread_base*) () from /lib/libtorrent.so.21
#21 0x00005555555945a3 in main ()
(gdb) Quit
(gdb) 

so i was able to crash it (perhaps by adding torrents too fast - CPU was above 70% where normally it is well below 20%), but none of this behavior is anything like the problem im trying to troubleshoot, perhaps you'll find it useful

[stickz]

Okay thank you, this is what I require. I just need you to scroll upwards. #1, #2 and #3 are missing. These lines are critical. It looks like a curl crash which is easily fixable. I just need to see why curl is crashing. The other details are not relevant.

[jerryfreak]

so i tried again and was patient. after the run above i rebooted, from a cold boot rutorrent was giving me the "500 list" error because obviously there were more than 6393 torrents in the session

so i killed rtorrent and ran under gdb.

was back to 300 or so torrents and first started adding them a few hundred at a time taking breaks when cpu would get above 50%. once above 6300 i added them 10 at a time and above 6380 one or two at a time.

per usual, rutorrent lost connection exactly when adding torrent number 6394

rtorrent never crashed, i was able to add hundreds and hundreds more torrents and see them disappearing from the watch folder and see rtorrent processing them like normal in the terminal window. not sure i can debug, how do i stop it? as of now its still running and ill probably let it run until i hear from you (although its bedtime here)

i tried to take screenshots of the terminal as i was about to drop the critical torrent but i saw nothing at all unusual in the rtorrent/terminal side, only rutorrent dropping connection and returning the usual Bad response from server: (500 [error,list])

as far as lines 1,2 and 3 from the crash above, not sure they are accessible. i scrolled up as far as i could, and unless they are logged, were probably lost in the reboot. i saw the ine numbering and tried to get there but i think it was inaccessible due to the pagination

not sure i could reproduce that crash, but i could try.

[stickz]

I'm spinning up a cloud server right now to test a change I just made to the script. https://github.com/stickz/rtinst/commit/710b3873dbfca6e70c7ccbf97cd2f2e440f5eafb. The curl version doesn't match the operating system version on Ubuntu 22.04 LTS. Dependencies could be causing that issue.

[jerryfreak]

cool

my main machine that behaves the same way is 20.04, FWIW

[stickz]

Okay the changes passed. Here's a shortcut to update curl. rt stop. Run these commands. then rt start. ONLY do this for Ubuntu 22.04 LTS. This WILL crash if you do it on Ubuntu 20.04 LTS.

cd ~ wget https://github.com/curl/curl/releases/download/curl-7_85_0/curl-7.85.0.tar.gz tar -xzf curl-7.85.0.tar.gz rm curl-7.85.0.tar.gz

cd curl* ./configure --prefix=/usr --with-openssl --enable-ares --disable-static make -j$(nproc) CFLAGS="-O3 -fipa-pta -march=native -fexcess-precision=fast -freciprocal-math -flto" sudo make -s install

[jerryfreak]

its maybe best to close this ticket out. since the source plugin issue was resolved in the first few posts and weve been talking about random stuff since then.

while the curl patch may fix that random crash i saw in testing, the limit of rutorrent not being able to display list of torrents when rtorrent is running above 6393 is still unaddressed.

i have seen it in all of the following conditions: -20.04LTS -22.04LTS

-the current rtorrent code at rtinst v1.84 -your modified rtorrent code as above

-rutorrent 3.10 from current rtinst -rutorrent 4.0 from the master build as of a few days ago

not sure where to open ticket. you say its an rtorrent issue, not rutorrent, but rutorrent is malfunctioning and doesnt crash or produce any error messages. should i open the ticket under rutrrent? rtorrent? rtinst?

perhaps for now i need to try a build on 16.04 or 18.04 with rutorrent 3.09? i find it hard to believe that nobody has ever reported this limit before in the history of the codebase... when searching online the only answer is "well it should be practically unlinited with proper resources", yet i see the exact same limit,, repeatable to the single digit number of torrents on different machines with wildly different resources, with different versions of ubuntu/rtorrent/rutorrent

it seems the only workaround would be a multiuser environment with multiple instances (or different clients like qbit/deluge/transmission, none of which offer the functionality of rutorrent in regard to grouping, organizing, and preparing blocks of torrents for other machines and/or cold storage, which is my use case

[stickz]

@jerryfreak Please open a new issue report if the ruTorrent web client won't display torrents when the count is higher than 6393. This is a bug we could probably fix. Include screenshots using the snipping tool. You can bur sensitive information if required.

I would like to know if you're using the rpc plugin or the httprpc plugin. This could potentially have a big impact on this issue.

[jerryfreak]

ok working on it now

when did 4.0 go stable? how can i check if installs done in the last week are 4.0 stable or beta3? for now i put beta 3 in the bug report but it really has been seen on every version ive tried