Nyuu seems to fail on an intermittent connection

[RollingStar]

My connection has been dropping all day. I managed to upload two 200 MB files that were within a subdirectory. But Nyuu got stuck on part 3 for 7+ hours, plenty of time to upload 200 MB on my connection.

I think it is because it must have lost a connection while uploading, and been unable to reconnect. I'm on Windows 10 x64 with Powershell.

[animetosho]

Hmm, that's unusual. Would you happen to have a copy of the log/output and the command line and/or config used?

If you're actually going to run it again, could you also please add --progress stderr --progress http:localhost:8888 to the command, and when it gets stuck, visit http://localhost:8888/ in your web browser and copy+paste the output as well?

With default settings, Nyuu should log a warning if it detects that the connection dropped out. If not (e.g. hard connection drop) the default timeout should pick up something and cause a connection re-connect, which should also get logged as a warning. If it's not doing that, then there's a bug somewhere.

[animetosho]

Out of interest, have you ever encountered this any more?

[RollingStar]

No, but I'm on a different connection that is more reliable. Not only was the Internet connection worse for me a few months ago, but it was over a powerline adapter, so there were two points of failure.

If you have advice for where activity might be logged, I can try looking for a log file. I haven't seen one thus far.

[animetosho]

Output (info/warning messages etc) are written to the console - they should be displayed above the progress indicator line.
Thanks for the reply.

[RollingStar]

There was no error message to the best of my knowledge.

[animetosho]

Nyuu should always log something unless you've specifically told it not to.

By default, if there's no errors, it should display something like:

Uploading X article(s) from Y file(s) ...
Reading file file_A...
Reading file file_B...
All file(s) read...

I presume that, by 'no error message', you mean to say that the output looked something like that?

[RollingStar]

Yeah. From what I remember it just looked like it was still uploading part 3.

[sspilleman]

See below my output... (some lines removed, you get the idea :-)

Please also note, I have configured the connections to be 14, although my provider "says" wil my account tier, I should have 30 concurrent ones, so I also don't understand the code 502 errors

Uploading 6809 article(s) from 1 file(s) totalling 12.68 GiB
Reading file sweet.mp4...
NNTP connection error occurred: write EPIPE
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTP connection error occurred: Response timed out
NNTPError: Response timed out
Posted 848 article(s) in 00:11:18.072 (2442.59 KiB/s). Raw upload: 20.93 MiB/s
NNTP connection failed: Unexpected response to auth pass (code: 502): Too many connections., reconnecting after 10 second(s)...
Process did not terminate cleanly; active handles: WriteStream (2), Timer

[animetosho]

Thanks for posting that, but, from the output, it doesn't appear that Nyuu failed. Or did it?
Nyuu will report issues that it encounters, but it seems like it's managed to work around them in this case. Though there does seem to be quite a number of problems that did come up.

so I also don't understand the code 502 errors

I can only speculate, but maybe your provider somehow doesn't realise dead connections as not being used.
The response time outs you're getting is interesting. Basically the server didn't respond to Nyuu's request, so Nyuu forcefully tears down the connection and starts a fresh one (as the state of the NNTP protocol is undefined at that point). Perhaps the provider takes a while to notice the disconnect and counts these non-existent connections towards the number of total connections you have open.

[sspilleman]

Unfortunately the posting did fail. As you can see from the last lines, it managed to post 848 articles but it is supposed to post 6809 articles (as you can see from the first line, before exiting ungracefully with the error: Process did not terminate cleanly; active handles: WriteStream (2), Timer. The line "NNTPError: Response timed out" (before last) is printed in red.... looks like some generic exception handler. The previous ones ("NNTP connection error occurred: Response timed out") look like warnings (yellowish) and are the ones that appear recoverable......

Your explanation on the 502 errors is indeed the same that I was thinking of as well. I do have the option "keep-alive" set, could that have contributed to this issue as well?

[animetosho]

Oh, oops, I missed that part, thanks for pointing that out.

By the default, Nyuu will give up after trying for a while. Red text does indicate an error (you guessed right!) and Nyuu is meant to halt when it encounters one. Use the --skip-errors option to continue despite errors.

This error isn't an issue with Nyuu, it basically means that, despite Nyuu's attempts, it wasn't able to submit the article to the server. In other words, "post, server didn't reply, ok, try again, post, server still didn't reply, try again ... post, server didn't reply, ok I give up -> error" (controlled by the --on-post-timeout flag).

I do have the option "keep-alive" set

I'd leave that unset.

So all in all, I'd try adding --skip-errors all to your command, and maybe reducing the number of connections to see if that helps. Just note that skipping errors may mean that you'll get incomplete postings.

[sspilleman]

skip-errors isn't an option (for me).... I need postings to be 200% complete ;-). Im working on a fuse nzb filesytem... put some nzbs in a folder and open the files directly from a mountpoint.... I don't wanna par/unrar, so 100% complete is crucial (as otherwise files would be corrupt)

You can find it here nzbfs. Would be nice to get your feedback 2 :-)

btw. I've tried everything now! (increase timeouts, significantly up the number or retries, etc.... I cant get one single posting to complete for some reason.... the all fail (with a red error msg at the end, articles posted less than needed) :-( Please see below the current version of my config-sander.js.... Maybe I'm overlooking something here....

/**** Nyuu options/config file ****/
// This file contains all the default options for Nyuu
// You can customize these options to avoid having to specify them on the command line, however it is recommended that you not edit this file
// Instead, copy this file elsewhere and use the `--config` option to get Nyuu to use this copy. Missing options there will sourced from this file
// WARNING: this file is not maintained for backwards compatibility; this means that you'll need to re-set all custom options every time Nyuu is upgraded!

module.exports = {
    /** Upload Server Options **/
    servers: [{
        // connection options - see the following pages for full documentation
        // non-SSL: https://nodejs.org/api/net.html#net_socket_connect_options_connectlistener
        // SSL: https://nodejs.org/api/tls.html#tls_tls_connect_options_callback
        connect: { // connection options
            host: 'news-eu.newshosting.com',
            port: 563, // null => if server.secure, port=563, else, port=119
            // SSL options
            rejectUnauthorized: true,
        },
        secure: true, // set to true to use SSL
        user: 'xxxxxxxxxxx',
        password: 'xxxxxxxxxxx',
        // note that these times are specified in miliseconds
        timeout: 120000, // 60000ms = 1 minute
        connTimeout: 60000, // 30 seconds
        reconnectDelay: 10000, // 5 seconds
        connectRetries: 20,
        requestRetries: 20, // how many times to retry an interrupted request
        postRetries: 20, // how many times to retry if server returns 441 response to posted article
        keepAlive: false, // always reconnect on error, even if not needed
        onPostTimeout: null, // list of actions (strings) to take if server sends no response to a post; values can be 'retry', 'strip-hdr=X' and 'ignore'; if not set (null), defaults to ['retry','retry','retry'...] where the number of elements == requestRetries
        tcpKeepAlive: false, // false to disable, otherwise set a number for probe interval (in ms)
        postConnections: 8, // number of connections for posting
        checkConnections: 6, // number of connections used for checking
        // TODO: consider ability to reuse posting connections for checking?
        //ulConnReuse: false, // use uploading connections for post checks; only works if checking the same server as the one being uploaded to
    }],
    // multiple servers can be specified by adding elements to tbe above array, but note that:
    // - specifying options via the CLI may get confusing
    // - servers are currently selected randomly for posting/checking; Nyuu won't otherwise do anything special if you specify multiple servers (this includes falling over if a server is misbehaving)

    /** Post Check Options **/
    check: {
        delay: 10000, // (in ms) initial delay for performing check
        recheckDelay: 30000, // (in ms) delay retries by this amount of time; not used if tries<2
        tries: 20, // number of check attempts; should be 0 if not performing post checks
        group: '', // if set, will switch checking connections to this group; some servers seem to want one when STATing posts, otherwise they fail to show them; if set, should be a valid group you never post to, eg "bit.test"
        postRetries: 20, // maximum number of post retry attempts after a post check failure; set to 0 to never retry posting
        queueCache: null, // maximum number of cached posts in the post-check queue; if this number is exceeded, posts are dropped from cache if possible; if posts cannot be dropped from cache, this value acts like queueBuffer and will pause uploading when full. Caching is only useful if posts need to be re-posted due to a failure condition, in which case, uncached posts need to be re-generated off disk; default 5 or min(connections*8,100) if unseekable streams are used
        queueBuffer: 1000, // maximum number of posts in the post-check queue; if this number is exceeded, uploading is paused until the queue is emptied below this size
    },

    skipErrors: [], // list of errors to skip; can be set to true to imply all errors; valid options are 
    maxPostErrors: 0, // if > 0, maximum number of failed articles to allow before aborting
    useLazyConnect: false, // if true, will only create connections when needed, rather than pre-emptively doing so

    /** Post/Article Options **/
    articleSize: 1000000, // in bytes, must be a multiple of 2
    bytesPerLine: 128, // in bytes, note: as per yEnc specifications, it's possible to exceed this number

    postDate: null, // if set, override timestamps used for Message-ID header, Date header and NZB timestamps
    keepMessageId: false, // if true, don't randomize Message-ID header every time the post is submitted
    comment: '[PRiVATE]', // subject pre-comment
    comment2: 'pssdaolpu', // subject post-comment
    groupFiles: false, // group "similar" files (based on filename) together into sub-collections, similar to how usenet indexers would do it; only affects the file counter in the subject line

    // if any of the following are functions, they'll be called with args(filenum, filenumtotal, filename, size, part, parts, chunkSize)
    postHeaders: {
        // required headers; do NOT set Message-ID as this is auto-generated
        Subject: null, // if null, a default Subject is used
        From: 'xxxxxxxxxxx', // 'A Poster <a.poster@example.com>'
        Newsgroups: 'xxxxxxxxxxx', // comma seperated list
        Date: null, // if null, value is auto-generated from when post is first generated
        Path: '',
        // optional headers
        //Organization: '',
        'User-Agent': 'Nyuu',
        // nice list of headers: https://www.cotse.net/privacy/newsgroup_header.htm or http://www.cs.tut.fi/~jkorpela/headers.html
    },
    // postHeaders can also, itself, be a function, in which case, it is called with (name, size, num, numTotal) as arguments, and must return an object like the above

    /** NZB Options **/
    nzb: {
        writeTo: null, // supply a writable stream (or function which returns one) or filename for NZB output
        writeOpts: { // for details, https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options
            //mode: 0666,
            flags: 'w', // default wx, change to 'w' to overwrite file if it exists
            defaultEncoding: 'utf-8',
            encoding: 'utf-8',
        },
        overrides: {
            // here you can override values for NZB <file> entries
            // if unset, will use the NNTP header values from the first segment of the file
            // can be set to a function, which will be called with args(header_value, filenum, filenumtotal, filename, size, parts)
            subject: null, // Subject header
            poster: null, // From header
            date: null, // timestamp when post was generated (note: will be interpreted as a Javascript date)
            groups: null // Newsgroups header
        },
        minify: false,
        compression: '', // can be 'gzip', 'zlib', 'deflate', 'xz' or '' (none)
        compressOpts: {}, // options for zlib, see https://nodejs.org/api/zlib.html#zlib_class_options
        metaData: {
            // eg:
            // password: 'mysecret',
        },
    },

    /** Input Stream Copy/Tee Options **/
    inputCopy: null, // a writable stream to copy the input to, or a function (see example below)
    /* this example excludes PAR2 files from being copied
    inputCopy: function(filename, filesize) {
        if(!filename.match(/\.par2$/i))
            return fs.createWriteStream(filename + '.pipe');
    }
    */
    copyQueueBuffer: 4, // number of article-sized chunks to buffer to copied streams

    /** Tuning Options **/
    useBufferPool: true, // self manage article buffers rather than rely on GC's management; also improves performance of writing to buffers
    headerAllocSize: 4096, // amount of buffer space to allocate for post headers, only used if useBufferPool is true
    diskReqSize: null, // chunk size when reading from disk; default = Math.ceil(1048576/articleSize)*articleSize
    diskBufferSize: 1, // number of chunks to buffer
    articleQueueBuffer: null, // number of buffered articles; default is numConnections

    /** Other Options **/
    subdirs: 'skip', // can be 'skip' or 'keep'; note that it affects directly passed directories too
    // if above setting is 'keep', filenames will be transformed according to the following setting
    // the default is to keep the filename component only, which essentially flattens all files into a single directory
    // this is similar to how other clients handle folders
    // you can also return false from this function to skip specific files
    subdirNameTransform: function (fileName, pathName, fullPath) {
        return fileName;
    },
    // another example: include path, seperated by dashes (e.g. "MyFolder - SubFolder - SomeFile.txt")
    // subdirNameTransform: function(fileName, pathName, fullPath) { return pathName.replace(/\//g, ' - ') + fileName; },
    dumpPostLoc: '', // dump all failed articles to this location (the Message-ID will be appended to this, so if you want to store in a directory, end this with a trailing slash); only useful for debugging
    isFullConfig: true // leave here to indicate that this is a full config file, as opposed to the simplified config file
};

[animetosho]

skip-errors isn't an option (for me).... I need postings to be 200% complete

Ultimately, if your server doesn't want to accept your posts, there's not a whole lot you can really do. Check the on-post-timeout option (or onPostTimeout in the config) - I've found that sometimes stripping the User-Agent header helps get things through. You can also try setting it to ignore and let the post check pick it up if it's missing (this may help if it actually got posted, but the server didn't tell us so).

Another option, if you believe that posts getting rejected is just a temporary thing, is to use skip-errors, dump out the failed posts somewhere (--dump-failed-posts option), then attempt reposting them at a later time, using the --input-raw-posts option. It may also help to 'repost' these to a different server.

You do have some odd settings in there:

• checkConnections: generally little reason to set this above 1
• articleSize: 1M bytes is rather large, are you sure the server can handle this fine? Maybe try setting this back to the default

Would be nice to get your feedback 2

Unfortunately it's not something I'd use so can't really give any good feedback. Interesting idea I suppose. I did notice that yEnc is done in Javascript, which is very slow.
Otherwise good luck on your project.

[sspilleman]

Thanks for the response! I'll try to work with your suggestions and see if I'm getting better results.

I have been testing newsmangler with article sizes up to 3Mb which works fine.. (that program works most of the times btw, but I sometime have has some/very few articles missing and that program doesn't support post checks, hence I'm trying to get this one working....

[animetosho]

If it is the issue, I can't quite think of any reason why one program would work with large posts and another wouldn't. Are you sure that propagation works fine as well? I was under the impression that most news servers had a limit of around 1 million bytes, as in, for the output article size (input, which is what most newsposters refer to for article size, needs to be smaller due to yEnc overheads).

Any particular reason why you need such large articles anyway? For seeking purposes on a filesystem, one would think that smaller is better.

[RollingStar]

Nyuu has gone through so many revisions that I now doubt I'll ever be able to reproduce this, if it even was an error with Nyuu. I'm fine with closing this once @sspilleman is satisfied.

[animetosho]

Thanks for the comment @RollingStar