Newlines are inserted after every tag, which messes up inline tags

[benborgers]

See example: https://secure.helpscout.net/conversation/2659440427/5007

[mary-ext]

Hmm, can I get examples on what's going on? if I understand the issue title correctly then this would've been an issue I think, yet it's not.

cc @benborgers @jmduke

[commonsguy]

Um, hi! 👋🏻 I'm the Buttondown customer who reported the issue. Buttondown's editor is not handling newlines as expected of a Markdown processor when the Markdown is pasted in from an external editor. Those of us pursuing a POSSE strategy are more likely to be doing that.

For example, this paragraph has a bunch of extraneous newlines in it. I am entering this in Sublime Text, and we see the newlines (see screenshot). If I paste this into the CommonMark spec dingus, the newlines are ignored. I get the same results if I paste it into Stack Overflow. And, I get the same results if I paste it into Babelmark, testing across 27 different Markdown implementations. But, if I paste this from the GitHub comment editor into Buttondown, Buttondown keeps the newlines, which is unexpected. Admittedly, GitHub also seems to be keeping the newlines, at least within the preview. 🧐

[commonsguy]

...and GitHub also kept those newlines in the end comment, which is also unexpected.

Anyway, it'd be cool if Buttondown supported some sort of "import" option where we could supply Markdown that was handled the way other Markdown processors handle newlines, that's all. Thanks for considering this!

[jmduke]

👋 Thank you for the elaboration!

@mary-ext This is really I think more of a backend problem. We introduced this behavior in https://github.com/buttondown/monorepo/pull/1278 to fix https://github.com/buttondown/roadmap/issues/2557 and https://github.com/buttondown/roadmap/issues/2997. I wonder if we can specialize the use of nl2br now that we have the buttondown-editor-mode sigil to only invoke it for Fancy Mode-borne emails, which should solve this.

[brtholomy]

In my case, something changed in the backend handling of your RSS integration. Namely I don't supply Markdown at all, just HTML you scrape for me. Previously it looked like you were passing it through untouched (maybe just sanitized) - now it's formatted with newlines after every tag, including the content of the tag which is problematic.

Looks to me like you're converting RSS HTML to MD and back again to HTML just for the sake of your editor, which I'd really prefer didn't happen. That's why I write in MD and use a static site generator with a solid MD engine.

Just a reliable pipe from RSS to your Postmark hook is all I want, por favor. It'd be nice if that pipe had nothing to do with the finicky nature of WYSIWYG.

Django templating in the RSS flow is one of your best features: messing it up with the concerns of a mini-IDE is not. A sane template language and the "send test email" feature is all I need, since that's the final word on how it'll render.

Consider decoupling the "live preview" flow from customers with the RSS setup? I'd prefer it.

[jmduke]

@brtholomy

So two things have been true for a while (let's say past year or so?):

1. The way the RSS → Email integration works, at a high level, is that the content of the RSS feed is dumped straight into an 'email' object that the editor itself consumes.
2. We don't automatically munge any of that content, but switching between modes (e.g. opening an RSS-borne email in the editor and then going into Fancy Mode [the WYSIWYG]) can mutate it in the way that you're describing.

Am I correct in understanding that even though you're always in Markdown mode, you're still seeing weirdness? If so, that actually could just be another symptom of this issue's root cause (this is again in the weeds a bit but we do "render" raw HTML in an email through the Markdown pipeline.)

All of which is to say:

Just a reliable pipe from RSS to your Postmark hook is all I want, por favor. It'd be nice if that pipe had nothing to do with the finicky nature of WYSIWYG.

Django templating in the RSS flow is one of your best features: messing it up with the concerns of a mini-IDE is not. A sane template language and the "send test email" feature is all I need, since that's the final word on how it'll render.

Agreed on both fronts, and deviance from these core flows I do consider a bug.

[brtholomy]

We don't automatically munge any of that content, but switching between modes (e.g. opening an RSS-borne email in the editor and then going into Fancy Mode [the WYSIWYG]) can mutate it in the way that you're describing.

That's good news at least. But note that I only started babysitting each email like this because of another bug - which I didn't bother reporting cuz I figured it'd go away, and it did. I think the footer was failing to import or something.

Am I correct in understanding that even though you're always in Markdown mode, you're still seeing weirdness?

Yeah it'll insert a space after an inline tag, like this:

[foo ](bar)

I could tell you'd been doing refactoring, so some regression happened.

Believe me, I don't envy the task of creating a WYSIWYG editor in a browser - sounds like a deepfried nightmare. Serious yakshaving.

But here's the lowdown from a customer like me: this kind of wonkiness in a pipeline I took the time to set up is perceived as a near dealbreaker, because it means I have to futz with a GUI my whole setup is designed around avoiding. I want to git push my latest post and walk away, all civilized like. Bugs like this make me think: if they didn't have regression tests to catch this one, what's next?

I want to stay with Buttondown. I don't want to run my own server. But I think you should prioritize the hacker ethos more, and ease back on the shiny features. Write more integration tests. Get your shit tight. My stack: Hugo + Netlify + Buttondown has the potential to be wildly popular, I'd think. I know it's probably a minority of your customers, but it's a vocal and influential minority. Please them with integrations they can count on as much as they count on github. You still got that DIY vibe that Substack and Ghost will never have. Combine that with a good rep on the API side and you're golden.

[jmduke]

@brtholomy 100%, and I appreciate your candor here. We do have a lot of tests, but you're absolutely right that this is and always will be our most important/valued use case.

[commonsguy]

At least for my scenario, the fix works great — many thanks!

[brtholomy]

@jmduke do you think I could get the kind of stability I'm looking for with the API? It should be fairly simple to trigger:

1. RSS parsing
2. the draft email
3. and the real broadcast

right? or, I could even assemble the email myself and just trigger #3. Probably even with one curl line.

Am I correct that your API doesn't change much, and should skirt around all complications with the editor?

[jmduke]

@brtholomy Yes, absolutely! All of that should be out-of-thebox (we even have the RSS stuff programmatically available, though that's almost certainly overkill for this use case.)