search

1265578519 / OpenTracker

一个linux中开源和免费的BitTorrent Tracker
http://bbs.itzmx.com/thread-18214-1-1.html
76 stars 17 forks source link

kernel messages debug #4

Closed 1265578519 closed 1 year ago

1265578519 commented 1 year ago

运行十年了头一次崩溃，异常原因，猜测是硬件问题？ opentracker[13006]: segfault at 7f6031449430 ip 0000000000405f3b sp 00007f60320b00f0 error 4 in opentracker[400000+14000]

1265578519 commented 7 months ago

我在stats?mode=everything里面找到了相关显示，看来是新的代码故意限制compact=0？ 不过禁止访问次数只有37，不算很高，看来主流bt客户端都支持默认compact=1

      <count code="400 Invalid Parameter (compact=0)">37</count>
erdgeist commented 7 months ago

我在stats?mode=everything里面找到了相关显示，看来是新的代码故意限制compact=0？ 不过禁止访问次数只有37，不算很高，看来主流bt客户端都支持默认compact=1

      <count code="400 Invalid Parameter (compact=0)">37</count>

opentracker never supported compact=0. That's why no bittorrent client seems to require it. I do not understand why I would make the reply less compact just to allow it to be compressed better. Compression can only remove the parts I add when being less compact.

erdgeist commented 7 months ago

ot_fullscrape.o: In function fullscrape_make_gzip': ot_fullscrape.c:(.text+0x56e): undefined reference todeflatePending'

I replaced deflatePending with a more generic way to reallocate a normal sized new buffer, just like during the mail fullscrape loop.

1265578519 commented 7 months ago

opentracker never supported compact=0.

是的！我刚刚验证了历史版本的代码，确实不支持compact=0，而不是新代码发生了改动 可能我和另一个NexusPHP tracker记错了

1265578519 commented 7 months ago

I replaced deflatePending with a more generic way to reallocate a normal sized new buffer, just like during the mail fullscrape loop.

现在可以正常安装了，不过依旧有2个警告

[root@tracker opentracker]# make
cc -c -o opentracker.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE opentracker.c
cc -c -o trackerlogic.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE trackerlogic.c
cc -c -o scan_urlencoded_query.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE scan_urlencoded_query.c
cc -c -o ot_mutex.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_mutex.c
cc -c -o ot_stats.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_stats.c
cc -c -o ot_vector.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_vector.c
cc -c -o ot_clean.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_clean.c
cc -c -o ot_udp.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_udp.c
cc -c -o ot_iovec.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_iovec.c
cc -c -o ot_fullscrape.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_fullscrape.c
cc -c -o ot_accesslist.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_accesslist.c
cc -c -o ot_http.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_http.c
ot_http.c: In function ‘http_sendiovecdata’:
ot_http.c:168:7: warning: implicit declaration of function ‘asprintf’ [-Wimplicit-function-declaration]
       header_size = asprintf(&header, "HTTP/1.0 200 OK\r\nContent-Type: text/plain\r\n%sContent-Length: %zd\r\n\r\n", encoding, size);
       ^
cc -c -o ot_livesync.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_livesync.c
cc -c -o ot_rijndael.o -I../libowfat -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_rijndael.c
cc -o opentracker opentracker.o trackerlogic.o scan_urlencoded_query.o ot_mutex.o ot_stats.o ot_vector.o ot_clean.o ot_udp.o ot_iovec.o ot_fullscrape.o ot_accesslist.o ot_http.o ot_livesync.o ot_rijndael.o -L../libowfat -lowfat -pthread -lz
strip opentracker
cc -c -o opentracker.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE opentracker.c
cc -c -o trackerlogic.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE trackerlogic.c
cc -c -o scan_urlencoded_query.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE scan_urlencoded_query.c
cc -c -o ot_mutex.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_mutex.c
cc -c -o ot_stats.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_stats.c
cc -c -o ot_vector.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_vector.c
cc -c -o ot_clean.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_clean.c
cc -c -o ot_udp.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_udp.c
cc -c -o ot_iovec.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_iovec.c
cc -c -o ot_fullscrape.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_fullscrape.c
cc -c -o ot_accesslist.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_accesslist.c
cc -c -o ot_http.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_http.c
ot_http.c: In function ‘http_sendiovecdata’:
ot_http.c:168:7: warning: implicit declaration of function ‘asprintf’ [-Wimplicit-function-declaration]
       header_size = asprintf(&header, "HTTP/1.0 200 OK\r\nContent-Type: text/plain\r\n%sContent-Length: %zd\r\n\r\n", encoding, size);
       ^
cc -c -o ot_livesync.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_livesync.c
cc -c -o ot_rijndael.debug.o -I../libowfat -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_rijndael.c
cc -o opentracker.debug opentracker.debug.o trackerlogic.debug.o scan_urlencoded_query.debug.o ot_mutex.debug.o ot_stats.debug.o ot_vector.debug.o ot_clean.debug.o ot_udp.debug.o ot_iovec.debug.o ot_fullscrape.debug.o ot_accesslist.debug.o ot_http.debug.o ot_livesync.debug.o ot_rijndael.debug.o -L../libowfat -lowfat -pthread -lz
[root@tracker opentracker]# ./opentracker
^C[root@tracker opentracker]#
1265578519 commented 7 months ago

现在正在测试full scrape http://49.12.76.8:2710/scrape

在浏览器上访问得到了错误，在BT客户端上正常工作 image

image

image

1265578519 commented 7 months ago

这一版本代码，我一旦关闭服务器xshell窗口，opentracker程序就立即被自动退出结束了，我的启动命令没有变化

cd /home/OpenTracker-master;cd opentracker;./opentracker -f opentracker.conf.sample -p 8080 -P 8080 -p 6961 -P 6961 -p 2710 -P 2710 &
erdgeist commented 7 months ago

这一版本代码，我一旦关闭服务器xshell窗口，opentracker程序就立即被自动退出结束了，我的启动命令没有变化

cd /home/OpenTracker-master;cd opentracker;./opentracker -f opentracker.conf.sample -p 8080 -P 8080 -p 6961 -P 6961 -p 2710 -P 2710 &

Fixed in last commit.

erdgeist commented 7 months ago

在浏览器上访问得到了错误，在BT客户端上正常工作

Hmm, what's the problem there? I could download the full scrape without problems. Which browser causes that problem?

1265578519 commented 7 months ago

Hmm, what's the problem there? I could download the full scrape without problems. Which browser causes that problem?

在Firefox 上可以正常打开，在Microsoft Edge上会提示错误 image

1265578519 commented 6 months ago

可以增加一个最大peer数量吗？就和电驴ed2k服务器一样，超过多少人数后就无法登录到tracker服务器中 比如设置100个用户 如果第101个用户访问tracker服务器的时候，只会返回现有的100个peer列表，不会把自己的ip地址写入进tracker服务器中，可以同时在min interval右侧多返回一个参数max peer = 1，代表服务器当前队列人数已满

1265578519 commented 6 months ago

服务器最大人数设置，应当放入配置文件中使用 -f 启动进程 如果tracker服务器使用512MB内存，那么最大服务人数应当为2000K ≈ 3000K人

erdgeist commented 6 months ago

Hmm, what's the problem there? I could download the full scrape without problems. Which browser causes that problem?

在Firefox 上可以正常打开，在Microsoft Edge上会提示错误 ![image](https://private-user-i

Fixed in

https://erdgeist.org/gitweb/opentracker/commit/?id=ff03fd74496620e884bdf9bfec47afaf7e4e4d1a

erdgeist commented 6 months ago

可以增加一个最大peer数量吗？就和电驴ed2k服务器一样，超过多少人数后就无法登录到tracker服务器中 比如设置100个用户 如果第101个用户访问tracker服务器的时候，只会返回现有的100个peer列表，不会把自己的ip地址写入进tracker服务器中，可以同时在min interval右侧多返回一个参数max peer = 1，代表服务器当前队列人数已满

服务器最大人数设置，应当放入配置文件中使用 -f 启动进程 如果tracker服务器使用512MB内存，那么最大服务人数应当为2000K ≈ 3000K人

I do not understand. Maybe it is better if you first state the problem that you want to solve. And only then it makes sense to suggest specific solutions.

Is it correct that you want to somehow limit the memory footprint of the tracker by only allowing a maximum amount of peers to announce?

If so, you want to limit this amount per torrent? If you do that, other peers announcing themselves on other torrents will still use up more memory.

Or do you want to limit the total amount of peers on the tracker? This is very expensive, because I do not keep a centralized count of all peers over all torrents. Also that would mean that one torrent with many peers will prevent other torrents to ever be announced.

If it is less memory consumption you're after, you should consider tuning OT_PEER_TIMEOUT so that peers get removed sooner, but then you would probably need to also modify OT_CLIENT_REQUEST_INTERVAL and OT_CLIENT_REQUEST_VARIATION accordingly.

Overall, limiting the amount of peers to announce sounds rather unfair and would lead to some unforeseen cases where only bad or bad faith announcers can prevent others from successfully announcing.

Without a proper problem description and some well thought through analysis of potential solutions, it's hard to come up with a good proposal.

1265578519 commented 6 months ago

同时减少数值设置 OT_CLIENT_REQUEST_INTERVAL 和 OT_PEER_TIMEOUT 是可以降低内存使用的，但是会带来频繁的请求，引起更高的CPU占用率和网络流量 如果单独只降低 OT_PEER_TIMEOUT 的数值，虽然可以在不增加CPU使用率的情况下降低进程内存大小，但是那是不合理的，这种做法会导致过早的删除，用户请求tracker服务器后找不到当前其它可用peer，如果要修改那么两个数值必须同时修改 所以看来正确的解决办法是升级内存，从512MB升级为1GB，更改当前套餐计划2.6美元→3.8美元

image

image

image

image

1265578519 commented 6 months ago

Fixed in

https://erdgeist.org/gitweb/opentracker/commit/?id=ff03fd74496620e884bdf9bfec47afaf7e4e4d1a

现在chrome浏览器正常工作 image

1265578519 commented 6 months ago

现在我在 Firefox 上得到了异常显示，没有弹出下载，我猜应该是头部设置的原因 我看到输出内容为：Content-Type: text/plain 正确的值应该是：Content-Type: application/octet-stream image

1265578519 commented 6 months ago

https://erdgeist.org/gitweb/opentracker/commit/?id=b56e648b5e96f33db6d1cab9b43d613579837b04 这个提交导致stats页面变成了下载。。。 stats页面应当使用text/plain

erdgeist commented 6 months ago

I hope that https://erdgeist.org/gitweb/opentracker/commit/?id=160ba08074827f0ddecaf611dd9f9b15593be9c1 fixes all content-type issues.

If not, I will need to manually adapt them per request type.

1265578519 commented 6 months ago

https://github.com/1265578519/OpenTracker/issues/4#issuecomment-2057400870 ot_http.c 的两个警告不修复一下吗？

[root@tracker opentracker]# make
cc -c -o opentracker.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE opentracker.c
cc -c -o trackerlogic.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE trackerlogic.c
cc -c -o scan_urlencoded_query.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE scan_urlencoded_query.c
cc -c -o ot_mutex.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_mutex.c
cc -c -o ot_stats.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_stats.c
cc -c -o ot_vector.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_vector.c
cc -c -o ot_clean.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_clean.c
cc -c -o ot_udp.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_udp.c
cc -c -o ot_iovec.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_iovec.c
cc -c -o ot_fullscrape.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_fullscrape.c
cc -c -o ot_accesslist.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_accesslist.c
cc -c -o ot_http.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_http.c
ot_http.c: In function ‘http_sendiovecdata’:
ot_http.c:170:7: warning: implicit declaration of function ‘asprintf’ [-Wimplicit-function-declaration]
       header_size = asprintf(&header, "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n%sContent-Length: %zd\r\n\r\n", encoding, size);
       ^
cc -c -o ot_livesync.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_livesync.c
cc -c -o ot_rijndael.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -O3 -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_rijndael.c
cc -o opentracker opentracker.o trackerlogic.o scan_urlencoded_query.o ot_mutex.o ot_stats.o ot_vector.o ot_clean.o ot_udp.o ot_iovec.o ot_fullscrape.o ot_accesslist.o ot_http.o ot_livesync.o ot_rijndael.o -L../libowfat -lowfat -pthread -lz
strip opentracker
cc -c -o opentracker.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE opentracker.c
cc -c -o trackerlogic.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE trackerlogic.c
cc -c -o scan_urlencoded_query.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE scan_urlencoded_query.c
cc -c -o ot_mutex.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_mutex.c
cc -c -o ot_stats.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_stats.c
cc -c -o ot_vector.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_vector.c
cc -c -o ot_clean.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_clean.c
cc -c -o ot_udp.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_udp.c
cc -c -o ot_iovec.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_iovec.c
cc -c -o ot_fullscrape.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_fullscrape.c
cc -c -o ot_accesslist.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_accesslist.c
cc -c -o ot_http.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_http.c
ot_http.c: In function ‘http_sendiovecdata’:
ot_http.c:170:7: warning: implicit declaration of function ‘asprintf’ [-Wimplicit-function-declaration]
       header_size = asprintf(&header, "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n%sContent-Length: %zd\r\n\r\n", encoding, size);
       ^
cc -c -o ot_livesync.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_livesync.c
cc -c -o ot_rijndael.debug.o -I../libowfat -DGIT_VERSION=492da4a57e3ceac30709a9f932a3552eef2d2898 -Wall -pipe -pthread -Wextra  -D_DEBUG -g -ggdb  -DWANT_COMPRESSION_GZIP -DWANT_COMPRESSION_GZIP_ALWAYS -DWANT_IP_FROM_PROXY -DWANT_FULLSCRAPE -DWANT_NO_AUTO_FREE ot_rijndael.c
cc -o opentracker.debug opentracker.debug.o trackerlogic.debug.o scan_urlencoded_query.debug.o ot_mutex.debug.o ot_stats.debug.o ot_vector.debug.o ot_clean.debug.o ot_udp.debug.o ot_iovec.debug.o ot_fullscrape.debug.o ot_accesslist.debug.o ot_http.debug.o ot_livesync.debug.o ot_rijndael.debug.o -L../libowfat -lowfat -pthread -lz
[root@tracker opentracker]#
erdgeist commented 6 months ago

ot_http.c 的两个警告不修复一下吗？

I tried to fix it in https://erdgeist.org/gitweb/opentracker/commit/?id=a6c47663482479fbad3153a7a9d4dc9f07e11688 where according to

https://man7.org/linux/man-pages/man3/vasprintf.3.html

I was supposed to declare #define _GNU_SOURCE. I do not know why you still see a warning. The function is obviously there, because the linker does not throw an error. Under no OS I have here I can reproduce the warning.

1265578519 commented 6 months ago

https://erdgeist.org/gitweb/opentracker/commit/?id=e8d385f17607bbe86204251062039690dfc3979a 这个提交，测试已经消除了2个ot_http.c警告

对于zstd压缩，我没办法测试，视乎是少了 zstd.h 文件，我不知道较新的zstd压缩是否支持我使用的操作系统，brotli可以肯定是支持的，因为我在我的操作系统上成功使用过brotli

ot_fullscrape.c:18:18: fatal error: zstd.h: No such file or directory
 #include <zstd.h>
                  ^
compilation terminated.
make: *** [ot_fullscrape.o] Error 1
[root@tracker opentracker]#
1265578519 commented 6 months ago 
yum -y install libzstd-devel brotli-devel

Installed: libzstd-devel.x86_64 0:1.5.5-1.el7 brotli-devel.x86_64 0:1.0.9-10.el7 Complete!

我使用安装命令成功安装了libzstd-devel，现在获得了不同的错误

ot_fullscrape.o: In function `fullscrape_make_zstd':
ot_fullscrape.c:(.text+0x29): undefined reference to `ZSTD_createCCtx'
ot_fullscrape.c:(.text+0x5f): undefined reference to `ZSTD_CCtx_setParameter'
ot_fullscrape.c:(.text+0x194): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x1fd): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x23f): undefined reference to `ZSTD_freeCCtx'
ot_fullscrape.c:(.text+0x452): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x4c3): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x4fc): undefined reference to `ZSTD_freeCCtx'
ot_fullscrape.c:(.text+0x5b1): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x5c8): undefined reference to `ZSTD_freeCCtx'
ot_fullscrape.c:(.text+0x5f0): undefined reference to `ZSTD_freeCCtx'
collect2: error: ld returned 1 exit status
make: *** [opentracker] Error 1
[root@tracker opentracker]#
erdgeist commented 6 months ago

https://erdgeist.org/gitweb/opentracker/commit/?id=e8d385f17607bbe86204251062039690dfc3979a 这个提交，测试已经消除了2个ot_http.c警告

对于zstd压缩，我没办法测试，视乎是少了 zstd.h 文件，我不知道较新的zstd压缩是否支持我使用的操作系统，brotli可以肯定是支持的，因为我在我的操作系统上成功使用过brotli

ot_fullscrape.c:18:18: fatal error: zstd.h: No such file or directory
 #include <zstd.h>
                  ^
compilation terminated.
make: *** [ot_fullscrape.o] Error 1
[root@tracker opentracker]#

Can you try something like

yum install libzstd-devel

erdgeist commented 6 months ago 
yum -y install libzstd-devel brotli-devel

Installed: libzstd-devel.x86_64 0:1.5.5-1.el7 brotli-devel.x86_64 0:1.0.9-10.el7 Complete!

我使用安装命令成功安装了libzstd-devel，现在获得了不同的错误

ot_fullscrape.o: In function `fullscrape_make_zstd':
ot_fullscrape.c:(.text+0x29): undefined reference to `ZSTD_createCCtx'
ot_fullscrape.c:(.text+0x5f): undefined reference to `ZSTD_CCtx_setParameter'
ot_fullscrape.c:(.text+0x194): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x1fd): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x23f): undefined reference to `ZSTD_freeCCtx'
ot_fullscrape.c:(.text+0x452): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x4c3): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x4fc): undefined reference to `ZSTD_freeCCtx'
ot_fullscrape.c:(.text+0x5b1): undefined reference to `ZSTD_compressStream2'
ot_fullscrape.c:(.text+0x5c8): undefined reference to `ZSTD_freeCCtx'
ot_fullscrape.c:(.text+0x5f0): undefined reference to `ZSTD_freeCCtx'
collect2: error: ld returned 1 exit status
make: *** [opentracker] Error 1
[root@tracker opentracker]#

This is what https://erdgeist.org/gitweb/opentracker/tree/Makefile#n33 is for:

LDFLAGS+=-lzstd adds zstd to the linker options.

1265578519 commented 6 months ago

image 我同时 LDFLAGS+=-lzstd 启用后，现在安装过程没有发生错误了，正常安装成功

由于chrome对http请求不会携带br, zstd头部，浏览器只在https上携带br, zstd 我使用浏览器插件ModHeader来赋予自定义请求头部gzip, deflate, br, zstd https://chrome.google.com/webstore/detail/idgpnmonknjnojddfkpgkljpfnnfcklj

现在scrape成功输出Content-Encoding: zstd image

在未使用自定义头部的时候，访问获得ERR_CONTENT_DECODING_FAILED 错误，我相信这是浏览器对http请求的兼容性问题，因为修改自定义请求头部为gzip, deflate, br, zstd，访问正常 image

我觉得可以增加brotli，，，主流BT客户端现在不支持zstd，会返回错误Unsupported tracker response format. image

erdgeist commented 6 months ago

The -DWANT_COMPRESSION_ZSTD_ALWAYS will enforce zstd, even if the browser has not offered in its Accept-Encoding-Header. This is okay behaviour for gzip compression, but probably not for zstd, because only Chrome supports zstd at the moment.

I am currently working on https support. Browsers do only offer modern compression over https, so we can actually use brotli and zstd. For now, over plain http, it just does not work on any other Browser than some chrome variants.

erdgeist commented 6 months ago

I also re-modelled the Makefiles so that features that need to be enabled by giving compiler and linker flags can be enabled and disabled together be commenting out the include Makefile.gzip line

1265578519 commented 6 months ago

我觉得之前那种Makefile方式更好，经过我的测试，，设置成这样，可以分别响应不同的压缩编码类型 只要把ALWAYS这个开关禁用就可以实现最好的自适应效果，你现在改动了Makefile文件为组合方式，我不知道怎么修改了 image

image

#FEATURES+=-DWANT_ACCESSLIST_BLACK
#FEATURES+=-DWANT_ACCESSLIST_WHITE
#FEATURES+=-DWANT_DYNAMIC_ACCESSLIST

#FEATURES+=-DWANT_SYNC_LIVE
#FEATURES+=-DWANT_IP_FROM_QUERY_STRING
FEATURES+=-DWANT_COMPRESSION_GZIP
#FEATURES+=-DWANT_COMPRESSION_GZIP_ALWAYS

FEATURES+=-DWANT_COMPRESSION_ZSTD
#FEATURES+=-DWANT_COMPRESSION_ZSTD_ALWAYS
LDFLAGS+=-lzstd

修改Makefile为这样后，同时工作zstd与gzip

erdgeist commented 6 months ago

我觉得之前那种Makefile方式更好，经过我的测试，，设置成这样，可以分别响应不同的压缩编码类型 只要把ALWAYS这个开关禁用就可以实现最好的自适应效果，你现在改动了Makefile文件为组合方式，我不知道怎么修改了 image

image

#FEATURES+=-DWANT_ACCESSLIST_BLACK
#FEATURES+=-DWANT_ACCESSLIST_WHITE
#FEATURES+=-DWANT_DYNAMIC_ACCESSLIST

#FEATURES+=-DWANT_SYNC_LIVE
#FEATURES+=-DWANT_IP_FROM_QUERY_STRING
FEATURES+=-DWANT_COMPRESSION_GZIP
#FEATURES+=-DWANT_COMPRESSION_GZIP_ALWAYS

FEATURES+=-DWANT_COMPRESSION_ZSTD
#FEATURES+=-DWANT_COMPRESSION_ZSTD_ALWAYS
LDFLAGS+=-lzstd

修改Makefile为这样后，同时工作zstd与gzip

Okay, now I made the forced gzip disabled by default.

1265578519 commented 6 months ago

我觉得可以正式版1.0发布了，我直接和BT客户端开发者反馈支持zstd，这时候就不需要服务端支持brotli了 https tracker使用加密，会带来很大的系统性能损耗，他只适合更多核心的高性能cpu，如果要支持https我认为更应该去解决tcp多线程，不然他只工作在一个CPU核心上，不像udp可以使用配置文件 listen.udp.workers 8

erdgeist commented 6 months ago

Yes, I know about the performance hit. I am also in discussions with the libowfat's author of getting multi threading support in libowfat for tcp sockets.

But my plan is not to have all connections to be https (although smaller trackers can just enable this, if they choose to), but to have an extra port where the /stats and /scrape (fullscrapes) can be retrieved over https and where you can disable fullscrapes to be fetched over plain http. This will then limit the impact of the https overhead to the handful of stats and fullscrapes, where it should not actually matter.

I also have code to detect if on a port plain http or https is being sent, so I could even allow the stats and fullscrapes to be fetched from the standard tracker URL, just with https:// manually added in front of the URL.

erdgeist commented 6 months ago

我觉得可以正式版1.0发布了

No, I can't release version 1.0 yet. I need to wait for libowfat 0.34 to be released, because opentracker now uses features from the CVS.

1265578519 commented 6 months ago

好吧，在我使用的操作系统中只能安装libowfat 0.32或者更低版本的libowfat 0.31，使用libowfat 0.33只会获得无法安装的错误，我相信很多人和我一样无法成功安装高版本的libowfat，这可能也是我不关心libowfat 0.34发布的原因

1265578519 commented 6 months ago

我想在stats页面新增一个新功能，可以统计最热门的下载者数量最多的种子 stats?mode=top100 Top 100 torrents by leechers:

使用 peers - seeds 应该可以得到这个leechers top100列表

erdgeist commented 6 months ago

Like this? https://erdgeist.org/gitweb/opentracker/commit/?id=37f5b2403bd76a785c05afc843572f464410d457

I can not test it at the moment, I am in the country side behind bad internet. :)

1265578519 commented 6 months ago

Like this? https://erdgeist.org/gitweb/opentracker/commit/?id=37f5b2403bd76a785c05afc843572f464410d457

I can not test it at the moment, I am in the country side behind bad internet. :)

测试结果，工作不正常 image

erdgeist commented 6 months ago

Thanks. I forgot to initialise the list.

Should work with https://erdgeist.org/gitweb/opentracker/commit/ot_stats.c?id=59c1c30107ccba4c633ae7ca11c3249c88719e83

1265578519 commented 6 months ago

Thanks. I forgot to initialise the list.

Should work with https://erdgeist.org/gitweb/opentracker/commit/ot_stats.c?id=59c1c30107ccba4c633ae7ca11c3249c88719e83

现在工作正常！ image

1265578519 commented 4 months ago

我有一个新的优化性能想法

给不同peers总数赋予不同的值 seeds + leechers = peers peers＜50

define OT_CLIENT_REQUEST_INTERVAL (60 * 30)

peers＞50 并且＜200

define OT_CLIENT_REQUEST_INTERVAL2 (60 * 60)

peers＞200

define OT_CLIENT_REQUEST_INTERVAL3 (60 * 120)

以上三个函数都要生效VARIATION进行随机值，可能要赋予新的函数2和3

define OT_CLIENT_REQUEST_INTERVAL_RANDOM

define OT_CLIENT_REQUEST_INTERVAL_RANDOM2

define OT_CLIENT_REQUEST_INTERVAL_RANDOM3

OT_CLIENT_REQUEST_INTERVAL2 大概是这样，创建新的一行，我不知道我的写法是否会错误，因为还要加入peers检测，等你来更新来测试一下

define peers2 (50)

define peers3 (200)

define OT_CLIENT_REQUEST_INTERVAL2 (60 * 60)

define OT_CLIENT_REQUEST_INTERVAL_RANDOM2

(OT_CLIENT_REQUEST_INTERVAL2 - OT_CLIENT_REQUEST_VARIATION / 2 + (int)(nrand48(ws->rand48_state) % OT_CLIENT_REQUEST_VARIATION)) if ($peers > $peers2 && $peers2 <= $peers3)

这项功能我想可以节省较多的服务器性能，较多peers的种子还拥有PEX做用户交换，不需要频繁和tracker通讯 image