Allow specialized markdown header for NNTP posts

[schveiguy]

If I post via NNTP, my messages do not show up with markdown enabled.

Thunderbird does not allow me to change the Content-type header to add the markup=markdown piece. If Dfeed could recognize a custom header in addition to the Content-type header, that would be useful. Then I can have Thunderbird add the header when I want it to show markdown.

[CyberShadow]

The problem is that I don't want to proliferate an ad-hoc standard. These messages are forever, and, I don't want to create the only software that understands them correctly.

The convention that DFeed currently uses is not "standardized" by any standards body, but at least it has precedent. There exists other software that understands markup=markdown. Perhaps, with enough momentum and grass-roots adoption, it may yet become an actual standard.

So, please, file this issue in the Thunderbird bugtracker instead.

I did spend some time seeing if it's possible to do this using a Thunderbind extension. Their extension mechanism is both highly limited (when using their explicitly documented APIs) and allows doing some very intrusive changes to the main software (when using the APIs that replace Thunderbird components). It may be possible, but it's not easy, and I couldn't figure it out in a few hours.

I'm going to close this because I really feel like I have a responsibility to ensure that the messages that DFeed creates or helps create are as close to standard as possible, to ensure that they remain properly viewable at any point in the future without having to modify each involved software with site-specific hacks.

[schveiguy]

OK, well, my current and future efforts here will be to post with markdown, but without dfeed showing the markdown. I guess people will have to live with that.

[CyberShadow]

Will you file the Thunderbird bug?

If you don't want to do it, let me know and I will do it for you.

I don't use Thunderbird for newsgroups, so my understanding of the flow of using it that way is only as far as has been described to me.

[schveiguy]

I don't know what to file? I don't expect "support markdown" to be a trivial thing for them.

Or did you mean to support altering Content-type?

[CyberShadow]

I think the best way forward in that regard would be for there to be a way to create a Thunderbird extension that allows authoring messages using Markdown. That might involve previewing the rendered result, but at the very least, Thunderbird should have a mechanism of allowing extensions to add format=flowed to the Content-Type header.

[schveiguy]

I don't intend to write a Thunderbird extension, so I have a hard time justifying asking for them to add support for it. You can file if you wish.

[schveiguy]

Just tried this out, it works when I'm posting to NG, but doesn't seem to alter or add any headers:

https://addons.thunderbird.net/en-US/thunderbird/addon/markdown-here-revival/

but I think it was intended for a different purpose (converting to HTML rather than sending markdown messages). However, it is good to see there is at least some markdown editor that exists.

[CyberShadow]

OK, here it is: https://bugzilla.mozilla.org/show_bug.cgi?id=1717221

[CyberShadow]

OK, here's an idea for a workaround.

@schveiguy Can you try running https://gist.github.com/ccb3813d5953e5e7f2c53fe43275f986 and pointing Thunderbird to localhost:4119 instead of nntp.digitalmars.com:119?

Let me know if you'd like me to host an instance online if you can't or don't want to run it on your PC.

[schveiguy]

Oh that is clever! I'll have to play with it, though it could be a bit smarter on the replacement (if I understand correctly, it replaces ANY instance of Content-Type: ... with the proper data?)

I can run it on an internal server.

[CyberShadow]

Thanks!

it replaces ANY instance of Content-Type: ... with the proper data?

Well, only starting on a new line. And only for outgoing data.

though it could be a bit smarter on the replacement

Cleverness breeds complexity, complexity breeds bugs, bugs breed unhappiness, etc. Or something. :)

Packet fragmentation is maybe a bigger concern.

[schveiguy]

Why the enforce line here: https://gist.github.com/CyberShadow/ccb3813d5953e5e7f2c53fe43275f986#file-proxy-d-L29

It doesn't seem to use args, but this seems to require at least one arg. I took it out.

[CyberShadow]

My bad, vestigial from another program.

[schveiguy]

Not working, but not sure why:

NNTPProxy: Proxy started
NNTPProxy: Accepted connection from 172.16.76.13:56156
NNTPProxy: Client disconnected: Socket error: Lookup error: getaddrinfo error: Name or service not known
NNTPProxy: Server disconnected: Socket error: Lookup error: getaddrinfo error: Name or service not known
NNTPProxy: Accepted connection from 172.16.76.13:56157
NNTPProxy: Client disconnected: Socket error: Lookup error: getaddrinfo error: Name or service not known
NNTPProxy: Server disconnected: Socket error: Lookup error: getaddrinfo error: Name or service not known
NNTPProxy: Accepted connection from 172.16.76.13:56158
NNTPProxy: Client disconnected: Socket error: Lookup error: getaddrinfo error: Name or service not known
NNTPProxy: Server disconnected: Socket error: Lookup error: getaddrinfo error: Name or service not known

I don't know your ae library, so I'm not sure where to look.

[CyberShadow]

My bad, it's news.digitalmars.com not nntp.digitalmars.com.

Sorry, I don't have Thunderbird on this PC, so I haven't tested it...

[schveiguy]

Seems to be doing something now. Only downside is, it has to reload the entire set of headers (and hasn't downloaded the content). I'll have to see if I can restore the content from backup, as I use the Thunderbird search feature all the time. Will let you know when I get to the point of posting a markdown test.

[CyberShadow]

You could hack around that as follows:

1. In your /etc/hosts point news.digitalmars.com to 127.0.0.1 (or the machine you're running the proxy on)
2. If the proxy is running on the same machine, replace news.digitalmars.com with its IP address
3. Change the port to 119 (though that may require running the proxy as super-user or using something like authbind).

[schveiguy]

I think it's too late. What I should have done is added a new server instead of replacing the host with the IP (for the test anyway). but I think I can figure out where Thunderbird stores its cache, I have hourly backups, and just restore that.

[schveiguy]

Hm... didn't work, but I think this time it actually is a bug in dfeed. I see the content type is text/plain; markup=markdown; charset=utf-8; format=flowed

Though I can fix the proxy here to do the right substitution.

https://forum.dlang.org/thread/saj0o9$10qb$1@digitalmars.com

[CyberShadow]

Oops! Please swap serverTranscoder and clientTranscoder.

(Confused myself with the terminology... the program is both the server which your client connects to and the client as it's connecting to the DigitalMars server...)

[schveiguy]

haha lol, so it didn't actually send that header, it rewrote the header I was reading from the server Will fix.

[schveiguy]

works! https://forum.dlang.org/post/saj16m$12h0$1@digitalmars.com

[schveiguy]

FYI, this was a weird issue. Not sure how to solve it, will run in debug mode for now: https://github.com/CyberShadow/ae/issues/51

[schveiguy]

You should announce this workaround on the forum. I know a few people who would use it.

[CyberShadow]

Okay. https://forum.dlang.org/post/vuphbxxrfuvyhyhfaxpg@forum.dlang.org

[rikkimax]

If this was to be registered on the dub registry, it would be as simple as dub run X.

[CyberShadow]

Can I register a gist?

[CyberShadow]

It's not letting me register a gist.

[rikkimax]

Can I register a gist?

No, it has to be a repository.

[gizmomogwai]

Works for me too! One question about the function serverTranscoder: Is Data always the full request, or could it be, that the data is arriving at the function split in half (so that e.g. the replaceFrom cannot be found)?

[CyberShadow]

Yes, hypothetically. I mentioned packet fragmentation. Fixing that properly requires implementing rudimentary understanding of the protocol.

[gizmomogwai]

Thanks for the explanation. As long as it works for me most of the time :)

[schveiguy]

I'm thinking I will make an iopipe-based tool that is on dub that does this. The idea is awesome, and with a buffering system, it can be made correct, even with the possibility of fragmentation. All that is necessary is to look for newlines, and then scan for the string Content-Type: after that. If that fails, just send as normal, otherwise, do the translation.

[CyberShadow]

If you're going to make a better tool, might as well make it only do the rewriting if the line is in the headers part of the message :)

[schveiguy]

OK, I think I can swing that. The protocol looks pretty simple, I can use a line splitter to judge where posts start.

[schveiguy]

Ugh, the protocol is simple, but complicated enough that I have to know all the commands (At least once that have multi-line commands).

Not only that, but std.io isn't mature enough. I have no way to shut down the write end of the socket. Plus async i/o is not there yet, making this quite cumbersome. Here's what I have so far, but it doesn't work right (I haven't even tested the markdown rewrite): https://github.com/schveiguy/nntp-md-adder

will continue any discussion there.

[CyberShadow]

Ah, sounds like a good use case.

I think this should fix the two issues in the ae way (also untested): https://gist.github.com/CyberShadow/ccb3813d5953e5e7f2c53fe43275f986/revisions#diff-c5a125f52e69e0304b779f2c8101c88ea28e98f6e593e399124cba2a00a5a1eb