Nyuu doesn't include the closing </nzb> tag in nzb file until the whole process ends

[df-a]

Hello there,

So if you let nyuu run in a batch, it will not include the closing tag in nzb until the very end of the process and all nzb files are processed and generated. So if anything happens and the upload is interrupted like a power failure or a system halt, the resulting nzb file(s) cannot be imported into any nzb program, because they all lack the tag in the file. Once you manually include it per file, all files will then work fine.

Can you please make it that the moment an nzb is finished generating as it moves to the next one, it includes the closing tag on the finished file automatically and not all the way until every other nzb file is done?

Awesome, awesome app btw @animetosho. You were not kidding when you said Nyuu is designed primarily to be fast/efficient, reliable and customizable, exposing all the interesting bits and pieces. Congrats on making an amazing piece of tech.

[animetosho]

Thanks for the suggestion and compliment.

Would this be using the multi-NZB feature via config.js?
If so, currently the function is called whilst it runs through the files, so it doesn't know when the NZB is going to end until everything has. Perhaps it could gather all the mappings up at the beginning to know what files each NZB has up front, to know when it's ended - I can investigate this.

Another way to do it is to just have separate invocations to Nyuu for each NZB you want to create, since this is being done from a script anyway.

[df-a]

Not exactly. I did it with the list method. Could you illustrate the multi-NZB feature via config.js with a concrete example please? Because I don't really understand the comments in config.js about it.

I feed nyuu lists of links. For every list it makes a separate nzb file. So, say I have the following lists:

1.lst
2.lst
3.lst
4.lst
5.lst
...

# example file 1.lst
path/to/file/1
path/to/file/2
...
path/to/file/40
path/to/file/41

Nyuu takes them and will create the nzb files (correctly) as such

1.nzb
2.nzb
3.nzb
4.nzb
5.nzb
...

It creates these files sequentially, one by one, as it is uploading. However, during creation, if you open any of these files you will find: (in this specific example, say I open file 2.nzb while file 4.nzb is being generated, so files 1.nzb 2.nzb and 3.nzb are already finished, it will look like this)

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE nzb PUBLIC "-//newzBin//DTD NZB 1.1//EN" "http://www.newzbin.com/DTD/nzb/nzb-1.1.dtd">
<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
    ...
            </segments>
    </file>

These nzb's can be imported into nzb clients (I only tested nzbget), but they will show as "failed" and have "0 bytes".

When every single upload is finished, then I open the nzb file and they look as follows:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE nzb PUBLIC "-//newzBin//DTD NZB 1.1//EN" "http://www.newzbin.com/DTD/nzb/nzb-1.1.dtd">
<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
    ...
            </segments>
    </file>
</nzb>

These nzbs can then be used as normal in any nzb aware client. The previous ones only lack the closing nzb tag. If anything happens during the upload that interrupts it, you will have to take the fully uploaded nzbs (that will have no closing tag) and manually append the closing nzb tag to them if you want to use them. You have to manually discern which nzb corresponds to a fileset that has been uploaded correctly (fully) and which does/hasn't.

A solution would be to create the nzb file empty with the closing nzb tag in place as such

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE nzb PUBLIC "-//newzBin//DTD NZB 1.1//EN" "http://www.newzbin.com/DTD/nzb/nzb-1.1.dtd">
<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
    ...
</nzb>

And then proceed to add the nzb stuff to it as nyuu gets uploaded post data.

[animetosho]

Ah, tokens in the NZB output name - I forgot that I added that feature. Thanks for pointing that out.
The function feature in config.js is meant to be an advanced feature - behind the scenes, if you use tokens in the NZB name, it uses that functionality to generate multiple NZB files.

Alright, I'll take a look into seeing how to fix this.

[df-a]

If it is of any help, I invoke a single nyuu instance, pass it a single config.json, I also have some edits in config.js, and serve this invocation a few lists with files which are then separately converted to nzbs.

[animetosho]

Hopefully the above change does what you were looking for. Feedback otherwise welcome.

[df-a]

Good day to you Animetosho and Happy Thanksgiving if you observe that holiday,

Thanks a lot for trying to help with the commit. I tested out the new commit and I am back with the results. It actually introduced a bug. Previously I could not find a bug, but now I think I see it.

OP was about an unfinished nzb file. As we all know, a standard nzb file only has two nzb tags, a single opening <nzb> and a single closing </nzb>, and all the data is sandwiched in between. When something happens during the upload and interrupts it, like network disconnection, a blackout, anything, Nyuu would not include the closing nzb tag of any nzb file it is currently generating until the very end of the process when you get this message:

Example

[INFO] NNTP connection lost
[INFO] Process complete. Report follows:
         Elapsed time: 01:04:24.607
         Total articles: 20202 (13.27 GiB)
         Articles read: 20202 (100.00%)
         Articles posted: 20202 (100.00%)
         Articles checked: 20202 (100.00%)
         Errors skipped: 0 across 0 article(s)
         Upload Rate (network|real): 4550.13 KiB/s | 3864.93 KiB/s

After this message, it will include the tag each file needs to be recognized by an NZB reader, in my case NZBGet. Then and only can can you import the generated nzbs without any problem. If you try to import any of the generated nzb files before nyuu gives this particular message, they will result in failure. You will only see "failure" in nzbget.

However, with https://github.com/animetosho/Nyuu/commit/71788cd7de4468583a1f6aa480fc4cde37b24e7c, nyuu sometimes includes multiple closing nzb tags in any given file. I have only observed this behaviour while testing out that commit, not the previous master.

I have added a few examples of the problem ones to gists.

This is an example of the original problem. An unfinished nzb if the nyuu process got interrupted in any way. This nzb, while actually being complete, cannot be imported into NZBGet. If I add the singular closing </nzb> tag to the end of it, I can import it normally.

As you can see, it has no closing </nzb tag>. The contents themselves are fine though, all you have to do is add the closing tag (Assuming it's not this file that got interrupted meaning not every post was uploaded) and you can download the file.

This is an example of the new bug I've seen. Notice multiple closing tags scattered around, except at the end of the file. The placing seems random.

In this one, it follows every file with an </nzb>. But, correctly, there is just one opening <nzb> tag in the whole file.

I delete every extra </nzb> tag manually, then add a single </nzb>, and I can download it again without problems. So the files have been uploaded correctly.

And on this one, it's all over the place. It may have been a bug as the zipping algorithm tried to make too many splits with this file. But it happened with larger files as well.

Again, I delete every extra </nzb> tag manually, then add a single </nzb>, and I can download it again without problems. So the files have been uploaded correctly.

These have been happening more lately, whereas previously it never happened. Only when interrupted it wouldn't finish the nzbs. These are from a 32 post test batch today, which was completed as usual, but 15 of the nzbs had one or more of these ailments despite having been completed. The other 17 were fine. That has never happened to me before while using nyuu. I've never had a completed batch of nyuu posts have "faulty" nzbs in them, prior to me testing https://github.com/animetosho/Nyuu/commit/71788cd7de4468583a1f6aa480fc4cde37b24e7c.

As this is a public forum I edited the examples somewhat as to make them unable to be completed, but if you need I could add them as they were originally.

If you need any more information, just LMK.

[df-a]

I tried again with a smaller 5 post batch twice, and both times all files were upload correctly. Which leads me to believe that the previous result was probably a fluke. I also changed the message IDs somewhat, so there are more variables that changed in the meantime leading to an unscientific approach. Sorry about that.

I'm off to test some more.

[animetosho]

Ooh that's bad - thanks a lot for the detailed information.

I found and fixed a bug, which hopefully addresses the issue. Let me know how it works for you.

[df-a]

It's working perfectly. The nzbs aren't all updated in one shot (at the very end) anymore, rather, once nyuu moves on to the next post and empties the buffer for current post, it will close the singular nzb file. Previously, all the files of a batch once it is finished would have the same modified time.

I have not seen the multiple closing-tags-in-the-same-file either anymore.

The result is that once the upload is interrupted the previous nzb files from posts in the same batch are usable (only the one that was interrupted isn't, obviously).

Dude, you are a rockstar. I'm continuing with testing. Thanks!!! And a happy further thanksgiving (if, like mine, your country doesn't celebrate it then it's a thanksgiving online :) )

[df-a]

Confirmed, #93 is fixed with 9e8e6f8. Thanks again!

[animetosho]

Thanks for confirming the fix!

Unfortunately I'm not based in the USA, but happy Thanksgiving to you!