How to use to decode a segment

[javi11]

First of all, nice work here ;).

I´m trying to use the library to decode a yenc segment but I always end up with an transform.ErrShortSrc.

The segments that I get are encoded following this code:

`go buf := &bytes.Buffer{} buf.WriteString(fmt.Sprintf("From: %s\r\n", data.poster)) buf.WriteString(fmt.Sprintf("Newsgroups: %s\r\n", data.group)) buf.WriteString(fmt.Sprintf("Message-ID: <%s>\r\n", data.msgId)) buf.WriteString("X-Newsposter: xxxxx\r\n")

// Build subject
subj := fmt.Sprintf(
    "[%d/%d] - \"%s\" yEnc (%d/%d)",
    data.fileNum,
    data.fileTotal,
    data.fileName,
    data.partNum,
    data.partTotal,
)
buf.WriteString(fmt.Sprintf("Subject: %s\r\n\r\n", subj))

// yEnc begin line
// yEnc begin line
buf.WriteString(fmt.Sprintf(
    "=ybegin part=%d total=%d line=128 size=%d name=%s\r\n",
    data.partNum,
    data.partTotal,
    data.fileSize,
    data.fileName,
))
// yEnc part line
buf.WriteString(fmt.Sprintf("=ypart begin=%d end=%d\r\n", data.partBegin+1, data.partEnd))

// Encoded data
err := yenc.Encode(p, buf)
if err != nil {
    return nil, err
}
// yEnc end line
h := crc32.NewIEEE()
h.Write(p)
buf.WriteString(fmt.Sprintf("=yend size=%d part=%d pcrc32=%08X\r\n", data.partSize, data.partNum, h.Sum32()))

`

Then I just decode it by input the segment read from the Usenet provider. Like here https://github.com/javi11/usenet-drive/blob/dbb86388ad2648590442335512c8dd6f90b37212/internal/usenet/filereader/buffer.go#L289. Right now I´m using another implementation of yenc decoder.

Could you please guide on how to use the library?

Thanks

[mnightingale]

I agree I really should find some time to comment and document what this is and how to use it better.

I mostly put this together to I could learn some Go and play around with creating a NNTP downloader. I haven't made that public yet and I'm not very confident it'll ever get to a usable product.

Since my motivation was to try and create a high performance (> 5 Gbps) and efficient client, I found the existing naive yenc decoder implementations were a significant bottleneck. SABnzbd and NZBget solved this problem many years ago using code based on what became https://github.com/animetosho/rapidyenc so it seemed the ideal solution.

I'd suggest referring to the tests for how I'd use the decoder, for instance see what it's expecting the input/encoded data to look like https://github.com/mnightingale/rapidyenc/blob/master/decoder_test.go#L69 and calling decode https://github.com/mnightingale/rapidyenc/blob/master/decoder_test.go#L34-L45

You can also find a snippet from my client/download https://gist.github.com/mnightingale/0d917379b28a1bf0cad965bd4d34beb8 github.com/valyala/bytebufferpool is probably not necessary.

AcquireDecoder() allows me to reuse the decoder instance between requests, reducing allocations, but you need to return it with ReleaseDecoder to make it available for a new request - to make the required logic simpler you could just NewDecoder(defaultBufSize)

Then use it as an io.Reader and calling Meta() will return the state of the decode, name, crc, etc.

ReleaseDecoder() returns the decoder and resets its state.

It does expect the data to follow the NNTP protocol, so after the =yend line it would expect a .\r\n, I think that will be why you are getting a transform.ErrShortSrc.

If you are using a single instance or using NewDecoder() you'll need to call Reset() after each decode.

---

I do think this package should be considered unstable/alpha at the moment since the API may have breaking changes; the encode API is quite lacking since I primarily needed a decode method. The decode API I think is a bit odd with the choice to be an io.Reader necessitating a separate call to copy the state, it would probably be a good idea to use the Part struct approach.

In an ideal world there would be a way to use SIMD/intrinsics from Go and not have to use Cgo; it would be safer and potentially faster due to the penalty of calling Cgo. However, I've benchmarked decoding of 1MB blocks at >5GB/s so at this stage to me it's not worth the effort of trying to find a different way to do it.

---

Hopefully this will help you get it working, do let me know how you get on!

[javi11]

Thanks for the info, it is working now :). I was using textproto. dotReader which removed the \r\n from the content. Now I just simply pass the reader into the decoder and pum, works :)

I will be looking forward to see the encoding API more mature