Random Message IDs

[df-a]

Hello there animetosho,

I have been trying to manipulate message IDs in nyuu. Mainly, I want randomization with some strings and bits of strings in it. I have made a function for config.js, which should have the following outputs(examples)

YmrxHuESL02E0scsRm598MCbKyEP0QfSrj19@PEACHTREES
retH7MM38EGc5RzPNMRC2EEbxSnZ3@PEACHTREES
ITmpBHp1mmj46ES70eRoiuw2MDKCYnEJaJIDSJ08rM@PEACHTREES
u5t6eHqJcOd27E1M7kGRPLYBTMoaxlSE6YBQRSvLOlk@PEACHTREES
mwB7Hgf1s63EYbBwRt0X6MPQGmEJ6fLSUpr8@PEACHTREES
my9Hd7B68EVplR1XMMDQCEyt0SijU@PEACHTREES
rd5zv3H4SXL1u74E4lM65zRqQExaWM6xjCqkEgeUTl7S1wqDbd@PEACHTREES
CuLiNDHaP7b7w82EsAk94R0rRRjiMXsUZmXEv5JSElSOMUrXg@PEACHTREES
yftNHq6kE88EfRsqROb47MDgByExAtvSXleM@PEACHTREES
IzEHVJF94E3gYRCnYM8TXE3KESnXR@PEACHTREES
cKgRSHoAzwES00Etqg2zxRvCev9kMkbuxZ7EFRB5RwSULrCrq@PEACHTREES

As you can see, random string plus some constants. Notably no dashes, periods, nor any other special characters.

When I do an upload with nyuu while invoking this function, these are the results I get (again, a small sample but to show the results):

ONYaISsHq6SQ37668EoBOS5TrRddalbYNMGyed0w
s2c6KlsH0ZMntsm68EzZUlk4uR0ev0qCIMUzp2Mo6E5Xxgu6NSNMvq2ss@PEA
sAyH8qY68EACbRafWMlvREKcBSBOp@PEACHTREES
gyewHjRAk68E87bRyRIjM8UFkEMP8SWM8i@PEACHTREES-S
4A2WHo2sI68EXJgNRejMMM8pocECt3qS87zD@PEACHTREES
pX6wTHcGDVW68EMC75BRSsQLwMxp6AEEtV9F9SKKj9q@PEACHTREES-HTREES
zcsHQqy68E8SbRff8M2c5EfCpSPMg@PEACHTREES-z-PEACHTREES
8VckvdHMef6js68EvRvloKRw9qoEYMEvnFMpEytipO7SYc
U7RHpSm68EqgVRLWjMlgsEQ77SUJv@PEACHTREES-EgZ6BuxuSCrng4q3-PEACHTREES
bBZHwNd68EluRe2pMo4NEVlKStQ3@PEACHTREES-EG5IPC65SbReRPKW-PEACHTREES
60tQIuHv5Gr370EzUWoJLRmJ2VIYMRLoudUEPI3FRFSrvpRgo@PEACHTREES-
x6OMgpHkNU5UP71EQNuJZjRTBbvmSMmrjb0rE5cN
IUkMWlFHFQVSZOF72ESEQWtcQRf9CFbvVM3x9rTZjEuWYVNiYSSAAmiYS@PEA
SnDHDLL72EwKiRQQeMie6EPXSFel@PEACHTREES-
DfmHC1Q72E0WZRNPWMfQzERFStVw@PEACHTREES-4ZqSAmTsOJ-PEACHTREES
ntiNWaKHG4cp9RN72E3z8XDYTRSoTTC8TMndlPOP
ZNgtD7FHCtkFEbg72E3dYpTEvROAlNDrAMYYnTxnoEEEP0nWUSJ8MYIXS@PEACHTREES
Y6R7BjXHgcXSdI773Ei61WzY3RuJy4pqKMjv8K6y1EmefMU
EvEHO3z73ElxrRMd3MK8yEdEYSgqe@PEACHTREES-AMSBpBotW-PEACHTREES
flDnylHlCVCg974ERcmmARmIjXfFMNiwM1OE3u9y4MSx8RwpC@PEACHTREES-CHTREES

A whole lots of dashes are introduced and I don't understand why. Less importantly but equally puzzling is how the string at the end gets garbled. The reason I want to do away with these dashes is that some news servers do see these as spam and drops them like it's hot.

I try with a simpler function, and now I get

Kdv1WN0cJVzc0bAbd54tqeJFL0rTNl5JByydX96wBZmZcvuy-
tFpuQOPK3lsCo7QxUo6N61v7eTcJOiYk9wWwopzGDTkCVE-cin
1QI6oPvmj7ZnUGBsCUqTP0G9e1J0LcpiBZqXxFabmcQgPSIkG
3bBTGRiD5edyvI7nkYA5514s6tiYV7sb7SWVBDmMFP6wf6-8jJnq
1bOipvEzySC0a58NXfwg5oCmYpalydF2q7rOUKM2BNW03neoy5c1Co
4igm4xbp9yCFzi8ur2nvOz7Kgabd05vOuVVr5vtFgbcTfq-U6WEQ
L8e38Iz2nu0TXxo5RxsNTVklOY3S6gCXzKAAA8mv0dnqq
1yy0N2rPf6FCVScTO0AdPV6IyuZVxzo54nyBNfAfOxVkQsx-r

Still adds random dashes to the output. I can't figure out why.

So, my questions are:

• Is there something special that I have to account for if I want to customize message IDs like this?
• How/why is the function output being changed into results that would not be the result of the function?
• What are the actual post variables that I have access to on the JS end (such as {fnamebase} etc, how should these be referenced from JS)?

[animetosho]

If all you want is random characters with a special suffix, the easiest way would be using the option --message-id '${rand(24)}@PEACHTREES', or something like that.

As for your function, I probably should document somewhere that it doesn't really like generators that return IDs of different lengths.
This is because once the article has been written, changing the length of the message ID would require rewriting the article, whereas if it's of the same length, it'll write the ID "in-place". The workaround Nyuu uses is to force the ID to always be the same length, which is what you're seeing.

You can either change the generator to always generate fixed length IDs, or use the --keep-message-id option to prevent it changing IDs.

What are the actual post variables that I have access to on the JS end (such as {fnamebase} etc, how should these be referenced from JS)?

It's mentioned in config.js:

// if any of the following are functions, they'll be called with args(filenum, filenumtotal, filename, size, part, parts)

[df-a]

Thank you for the information animetosho. I had the way from config.json already implemented, but it always gave strings of equal (fixed) length. I wanted to change it up and introduce random strings.

I tried with the --keep-message-id option, bit it would still be introducing dashes in the string. They do vary in length, but have dashes at random places and can also vary as to the placement of the required string (which may or may not be completely mangled). In my testing I have realized that these dashes make it so it doesn't fully propagate across the usenet; instead, some providers drop these.

I'll keep playing with this see what I can come up with.

Thanks again!

[animetosho]

I've tried your function, but haven't been able to reproduce your issue with --keep-message-id. Are you checking by looking at the outputted NZB?

If you know of servers which don't like dashes in the Message ID, I can change the default to use a different character. Do you know which servers have such issues?

[df-a]

Maybe I applied it incorrectly.

I would have to add the func to config.js and also add the --keep-message-id flag to true in conf.json, right? I did that and it kept doing the same as if I had not added the --keep-message-id flag at all, i.e. mangle the function output and introduce the string spliced all over the place and also dashes at random. Still random, just not exactly a possible output of the function as I'd have liked.

Do you know which servers have such issues?

Eweka for one drops everything. I think Usenet.farm as well, it will have a lot of failures despite picking up some of it. The funny thing is I see some nzbs that DO have a dash in the message ID and eweka still picks them up correctly, so this may be a bug from the originating server. I will be getting some spread out blocks on many more servers to keep testing, as we may have stumbled upon a bug that has to do with the propagation of the usenet itself, rather than nyuu.

Therefore, hang on with any changes for now, I'm still doing some testing with nyuu's current master build.

[animetosho]

Make sure you don't have '--' in the name if you're using the JSON config file (I doubt you have it there, but just in case). What you say is correct otherwise.
You can also set keepMessageId: true in config.js to change the default if you're not sure it's being set properly.

Thanks for the server examples. I've had no issues with posting to Usenet.farm with dashes in the message ID, so that's interesting to hear.

we may have stumbled upon a bug that has to do with the propagation of the usenet itself

Any propagation issue due to specific message IDs would technically be a bug in server code, as the NNTP spec requires servers to not interpret message IDs. Regardless, there's little users can do about it other than to work around them.

I'd have thought dashes wouldn't be a problem, but if there's a different character which incurs fewer issues/bugs overall, I have no issues with changing the default.

Thanks for the investigations - am interested to hear your future results.

[df-a]

Hi there @animetosho , I have kind of experimented quite a bit with this.

Whenever you introduce too much randomization, some servers will drop some/most/every packet(s). I assume it's some sort of spam protection. Notably Eweka does this. UsenetExpress backbones do not have the same "problem". I haven't been able to test farm.

But dashes are fine. This block works fine, for example

<segment bytes="756743" number="1">fuUrtlWxiWD8iXjL-Yc-mTID2RNUo8DKOrl</segment>
<segment bytes="756444" number="2">ZEuKkziDpUcvo1UfoT7rjGpBaG6xfUfS9T</segment>
<segment bytes="756752" number="3">sxumnXpT1sXBBu3BkD4QbgD958uSM4IJiR</segment>
<segment bytes="756535" number="4">b09wXs7ormd5lq2ckNPxT7Vi7sdKm8AGXe8</segment>
<segment bytes="756535" number="5">vOZq-twSOjKz4RNSsnQUjxD2G9lFmmFDmbM</segment>
<segment bytes="756335" number="6">2QGIMSU2sAPNUUErqW3OYUF6iJizrWVw3G</segment>
<segment bytes="756584" number="7">6-x4w1k3xINCX3y-UI4Nc5QWb19cIG1uqm</segment>
<segment bytes="756439" number="8">ZLelJl3yFJR6dSwRVFlM2bzSPi3wVlCMEpj</segment>
<segment bytes="756464" number="9">fobN5SLkgcTgzSRZmRYybV744EGyVN-Ogc-</segment>
<segment bytes="756676" number="10">30d4wDgnX6ACMe-kP2lyQ2R5EYYvVccvX</segment>
<segment bytes="756523" number="11">fIyJnvdkaTkyqfuMto2jfIfxFL-CcVFnZ</segment>

Everything works nicely.

[animetosho]

Alright, thanks for getting back with your results.