single connection pool for posting/checking

[sjpotter]

Currently, it seems, that nyuu forces users to create 2 connection pools for posting and checking. In my first attempt at using it, I specified -n 45 and -k 1 (I'm limited to 50 connections). However, 45 posters overwhelmed the 1 checker and it got rate limited because of that. I then moved to -n 40 and -k 5 and that seemed to work without any rate limiting. However, it seems weird to force it that way. why not be able to specify -n 45 -k -1 to have a single connection pool and just use the normal "poster" connections to check whenever there are entries in the check queue.

From my example, it would seem that the majority of connections would still be used for posting, without the user having to manage / waste connections they have.

[animetosho]

Thanks for the suggestion!

Yeah, I've been considering this for a while. It would remove the check connections option and checking would be done on connections from the pool.
I wanted to integrate this with better multi-server management (since Nyuu would need to handle the case of posting and checking on different servers anyway), but have been thinking of toning down my initial plans.

The other idea I was considering was request pipelining, since that should be possible with STAT commands (support has been implemented in the NNTP layer, just need to design it into the upper layers).

[prdp19]

That would be really great! I use -S --ssl-ciphers ECDHE-RSA-AES128-SHA -k 20 --check-tries 5 --check-post-tries 5 -n 30 --connect-retries 5 --post-retries 5 --post-retry-delay 15 --error-teardown -e --use-lazy-connect

My limit is also 50 connections. Unfortunately the --check-connections slows down the process (just towards the end on). Very often the upload is already done but it still checks the uploads.

It would be nice if this could be sped up. What is the best limit to set the -k to?

Do you have any general recommendations when using NVMe?

With larger files I often/always get: [INFO] Upload speed has been throttled due to delays in generating articles to submit

[animetosho]

Very often the upload is already done but it still checks the uploads.

By default, --check-delay is 5 seconds and --check-retry-delay is 30 seconds. If posts are taking a while to appear on the server, checking can get delayed for quite some time after all articles have been submitted.
You can try tweaking those times, but otherwise, the system is more or less working as intended.

Maybe this suggestion to abandon checking once enough articles have been found, could be a solution.

What is the best limit to set the -k to?

You just need enough to avoid throttling - you'll get a Check queue is now full - upload speed will be throttled to compensate or similar notice if it ever occurs.
20 is likely overkill. Maybe start with 2, and see how it goes with that.

With larger files I often/always get:

Maybe try using a larger --post-queue-size, e.g. --post-queue-size=60.

[prdp19]

Thank you for your feedback. I will try it out.

So it is normal that the check (after finished upload) can take a few seconds (up to a few minutes), correct?

I seem to be doing well with -k 5 and -n 45.

--post-queue-size=60 I'll try it out!

I have set --connection-threads 3 - does that fit, should I leave this option out or expand it?

Powerful and stable tool, thank you for sharing your skills! You have my respect.

[animetosho]

So it is normal that the check (after finished upload) can take a few seconds (up to a few minutes), correct?

Depends on the server. Check requires the post to show up; if the server isn't showing signs of the post existing, it'll wait some time.

I have set --connection-threads 3 - does that fit, should I leave this option out or expand it?

The option is mostly useful if you have high bandwidth but a CPU without AES acceleration (which I can't imagine to be terribly common).
It might also be useful if you're turning for >=10Gbps or similar throughput.
But generally, if you're not seeing a CPU bottleneck, it likely doesn't do anything useful.