Markdown files from external editors lose formatting on save

[inthreedee]

Summary Markdown files that are created in external markdown and text apps will have their formatting completely overridden and/or lost when edited through the Nextcloud web interface. Newlines, tabs, and spaces are thrown out everywhere in the document--not just on the edited line--and lists will be changed to asterisks instead of dashes. See screenshots below.

Background I use external text editors for my markdown files and then sync them to my Nexctloud. I primarly use Atom for its markdown support but will also use basic text editors like Gedit on Linux or Notepad on Windows. That's the great thing about markdown files, after all, they're just plain text. When I edit these files through my Nextcloud web interface, the formatting of the entire file is destroyed.

Related: https://github.com/nextcloud/text/issues/390

To Reproduce Steps to reproduce the behavior:

1. Create a markdown file in an external editor such as Atom/Gedit or a basic text editor like Notepad
2. Add some spaces, tabs, newlines, lists, etc.
3. Open the markdown file in the Nextcloud web editor and notice the formatting differences
4. Make a minor edit to the file in the Nextcloud web interface
5. View the file again in your external editor and notice the formatted has been destroyed.

Expected behavior I expect the Text app to not completely destroy any formatting created by external apps. If any special formatting is unsupported, it should be left alone for maximum compatibility with external apps. If any formatting changes must be made based on the current edits, those changes should apply ONLY to the current edits and not the entire file.

Screenshots

How it looks in Gedit under Linux:

How it looks when opened in my Nextcloud web interface:

Make an edit through the web interface and now the file's formatting has been destroyed when viewed in Gedit:

Client details:

• OS: Arch Linux
• Browser: Firefox
• Version: 72.0.1
• Device: desktop

Server details

**Text app version:** 1.1.1 **Operating system:** Ubuntu 18.04.3 **Web server:** Apache **Database:** Mysql 5.7.29 **PHP version:** 7.3.13 **Nextcloud version:** 17.0.2

[newhinton]

If you want to ensure your markdown files are treated as code rather than rich text, please ask your provider to toggle the corresponding setting or do so on your own instance.

Just for clarification: I was under impression that markdown IS code and not rich text. Isn't this the whole point of this ticket, nextcloud treating .md-files like a 'proprietary' format which nc can control as if it owns it?

[PhrozenByte]

Please don't bring up the same arguments over and over again. Nobody benefits from :+1: comments. This is all known and the Pros and Cons of all thinkable solutions, including switching to a different file extension, have been extensively discussed before (scroll up and also note referenced issues) and decisions were made. It's not feasible.

Our goal is to improve Text's compatibility with other Markdown software - and as a developer of such "other Markdown software" that suffers heavily from Text's incompatibilities, I can tell you that I'm fine with that. Improvements have been made (including the ability to switch Text's default file extension, disable rich text editing by default, support for tables, checklists, YAML Front Matters, etc.) and are still being made. One usually doesn't have to remove Text any longer just not to break other Markdown software.

@juliushaertl Shall we lock this conversation?

So CommonMark is the de facto standard of markdown. I don't see why we'd want to bring the ambiguities back.

Just for the record, Text actually violates the CommonMark standard by breaking markup it doesn't understand. See https://github.com/nextcloud/text/issues/593#issuecomment-953009079

[Sieboldianus]

The post you linked also highlights the best current solution to the problem:

The only other solution right now is to remove Text.

(I think this should be the default, I've lived without text fine for a long time now)

[max-nextcloud]

The post you linked also highlights the best current solution to the problem:

The only other solution right now is to remove Text.

From Nextcloud 25 on you can run

./occ config:app:set --value false text rich_editing_enabled

Which will make text preserve all chars in markdown files.

If removing text still is the best solution I don't understand the problem and I really would like to. What's the benefit of removing text over disabling rich editing?

[Sieboldianus]

Many thanks for the response @max-nextcloud : This seems indeed like a fair solution. It was not that clear to me. Currently I am using Markdown Editor 2.3.6, which renders Markdown and preserves original Markdown formatting - it requires turning Text off, as otherwise Text would capture the file open event.

[timkgh]

From Nextcloud 25 on you can run

./occ config:app:set --value false text rich_editing_enabled

Which will make text preserve all chars in markdown files.

This is very useful! Can it be customized per file extension or mime type?

[bronson]

To avoid .md files being clobbered by surprise, does it make sense to have rich_editing_enabled default to false?

[timkgh]

To avoid .md files being clobbered by surprise, does it make sense to have rich_editing_enabled default to false?

Absolutely agree, should be the default for new installations. Data loss/corruption is not fun.

[max-nextcloud]

Just for the record, Text actually violates the CommonMark standard by breaking markup it doesn't understand. See #593 (comment)

What are the main pain points there? It's hard to tell for text if it's not understanding syntax or people are just using funny chars for syntax it does understand.

I'm happy to try and maintain syntax that is meaningful in other contexts. I understand that there are many say PicoCMS plugins that add their own syntax. Knowing text is breaking a commonly used plugin would help fix that. We're also happy to review pull requests in that regard, or even help writing them.

I'll play with caching the markdown source in each node and using that for serialization - but that would still only work for nodes that are not altered.

[bronson]

Here is my experience: https://github.com/nextcloud/text/issues/772 Tables and frontmatter are very common in Markdown even though they're not in CommonMark. I'd be queasy if any application thought it was OK to corrupt them.

So CommonMark is the de facto standard of markdown. I don't see why we'd want to bring the ambiguities back.

If I'm understanding you correctly, it sounds like you're saying that supporting CommonMark is ideal, and corrupting non-CommonMark files is regrettable but acceptable? If that's correct, which version of CommonMark will you target? Will it be acceptable to corrupt future versions of CommonMark?

As for disabling the Text app, doing that also disables all text editing in the NextCloud mobile apps. It sounds like many people find this to be a good solution, but I don't think that will work for me. My install has users that rely on mobile text editing.

[inthreedee]

It's hard to tell for text if it's not understanding syntax or people are just using funny chars for syntax it does understand.

There are picture examples in my original post of how even basic text-only formatting like indents, line spacing, carriage returns, and dashes used as bullet points are obliterated. Note, I have not yet updated to NC25 to see if any of this has changed.

[Sieboldianus]

My biggest problem was that NC removed linebreaks in my Markdown files. I use \n (LF) instead of the \n\r (CRLF) on Windows (because I primarily work from WSL/Linux in those files; in Windows I use Notepad++ which supports \n). Whenever NC "corrected" some Markdown line, it would remove these \n linebreaks, ending in a complete mess.

[max-nextcloud]

There are picture examples in my original post of how even basic text-only formatting like indents, line spacing, carriage returns, and dashes used as bullet points are obliterated. Note, I have not yet updated to NC25 to see if any of this has changed

Some of this has changed. Some remains.

But frankly. I don't understand why you are using markdown as a file format to write prose that uses spaces for indenting. Do you render that markdown with some engine? If you do - what does the output look like? If you don't - why don't you use a .txt file?

I played some with the markdown you provided. Or at least i tried to retype it:

My attempt at retyping the start of your markdown

```md # markdown is supported here this is a test line this is a second test line _tabbed italics line_ **tabbed bold line** spaces instead A list: - thing 1 - thing 2 'Twas brillig, and the slithy toves Did gyre and gimble in the wabe: All mimsy were the borogoves, And he mome raths outgrabe ```

If i paste it here in github this is what it looks like:

# markdown is supported here this is a test line this is a second test line _tabbed italics line_ **tabbed bold line** spaces instead A list: - thing 1 - thing 2 'Twas brillig, and the slithy toves Did gyre and gimble in the wabe: All mimsy were the borogoves, And he mome raths outgrabe

This is what it looks like in text:


Code after saving in text

```md # markdown is supported here this is a test line this is a second test line *tabbed italics line* **tabbed bold line** spaces instead A list: * thing 1 * thing 2 'Twas brillig, and the slithy toves Did gyre and gimble in the wabe: All mimsy were the borogoves, And he mome raths outgrabe ```

I think that's way better than the last time you tried... but you probably think it's still unacceptable to remove the leading whitespace and replace the - for a list with a *.

So far i've mostly been around programmers who cared about leading whitespace in tabs vs. 2 spaces vs 4 spaces etc. In those cases it's usually about having consistent whitespace across the doc. There are markdown linters that provide this and it would be fairly easy to include those in a configurable way. But keeping track of the whitespace of each line to preserve it seems odd to me. It would also lead to inconsistent formatting once someone added a new paragraph in text rather than gedit.

[newhinton]

But keeping track of the whitespace of each line to preserve it seems odd to me. It would also lead to inconsistent formatting once someone added a new paragraph in text rather than gedit.

As far as i understand this (and to be fair, i somewhat feel the same) it is about not changing things. Usually markdown-tools of any kind read a text-file, then convert the text to a nice view and display that. It's entirely one-way, and nextcloud changed that. markdown is often treated like code in a way, and that requires, by definition, that no unwanted changes can be made by tools.

Sure, preserving whitespace seems weird, but what if i use unprintable characters to denote some type of metadata? Text-editors will not change, edit, or remove those characters, but nextcloud now does (at least for whitespace, i dont know about other chars. The point still stands though). It doesn't really matter if its odd, it does not belong to nextcloud, and yet it chooses to assume control over that, and that rubs a lot of people the wrong way.

[inthreedee]

@max-nextcloud You're right, your recent tests are a bit better, but the changes happening there are still too destructive in my opinion.

@newhinton hits the nail on the head in the comment above:

It doesn't really matter if its odd, it does not belong to nextcloud, and yet it chooses to assume control over that, and that rubs a lor of people the wrong way.

When the text file's formatting is interpreted and overwritten according to new rules, we can't possibly account for every scenario where a user needs their original formatting preserved. It is clear from this thread that there are a lot of people with workflows that are broken by Nextcloud Text and trying to accommodate all of them with special handling seems an exercise in futility.

To answer your specific questions:

Do you render that markdown with some engine? If you do - what does the output look like? If you don't - why don't you use a .txt file?

I use markdown files mainly for notes. These notes end up containing all kinds of content, as notes usually do. I require basic formatting for these notes (hence .txt isn't ideal) and I require them to be portable across various devices, text and/or markdown readers, and web browsers. I chose the .md format because, since it is just text, is 100% future-proof, easily portable, and can always be read by a plain old text editor. To be clear, Nextcloud Text is the only application in all of this that I have run into that is causing problems. All other applications I've used read the file, interpret and display it how they want, but leave the formatting alone.

@asaage explained this succinctly above:

The whole point of markdown is, that you don't need a fancy editor and that after editing you can render it in other formats like html, pdf, odt... nextcloud-text / wysiwig is trying to go the other direction (html->md) with varying results.

The important part here is that the output on these devices and other apps doesn't necessarily matter in my workflow, as long as it's mostly consistent with some kind of markdown standard. It's the process of then overwriting the whole file with different text formatting that is too destructive.

I hope that clarifies my workflow a bit more. Let me know if you have other questions.

[daffydock]

Many thanks for the response @max-nextcloud : This seems indeed like a fair solution. It was not that clear to me. Currently I am using Markdown Editor 2.3.6, which renders Markdown and preserves original Markdown formatting - it requires turning Text off, as otherwise Text would capture the file open event.

This is what I have been doing. However, in my use case, I made the now mistake of adapting Collectives with others and as such, I cannot disable the Text app since without it, Collectives fails to launch. So, what do I do? Disable/Enable the Text app when I need the formatting to work properly and then enable Text when doing colab work? In fairness, I have to leave it on in case other users need to use the Collectives app when I am not around. Now I am left thinking of switching to unrelated apps to Nextcloud altogether if only to make the workflow go smoother and to avoid future issues, as it stands, which is going to bit of a pain. I cannot be the only one who has this same issue.

Do not get me wrong, I like the Collectives app. It has a lot of potential so it is unfortunate its use gets indirectly affected due to an issue that has to do with the environment it exists in, rather than an issue with the app itself.

[Sieboldianus]

@daffydock : I think you are right and NC is (still) on the wrong path, the Text app should get its own extension. The comment from @newhinton sums up the basic underlying issue fairly well:

Usually markdown-tools of any kind read a text-file, then convert the text to a nice view and display that. It's entirely one-way, and nextcloud changed that. markdown is often treated like code in a way, and that requires, by definition, that no unwanted changes can be made by tools.

Following the principle of least surprise, NC should a) stick to this basic nature of Markdown as a one way street or b) come up with its own format/extension (which may lean on Markdown or any other syntax).

[bronson]

Can anyone suggest a way to make sure the .md files in my Nextcloud won't get corrupted? If users on my instance experience the data loss that I did, it'll be a while before they trust Nextcloud again.

I'm currently running 24.0.6. I can't upgrade to 25 because I'm running Nextcloud AIO, and Nextcloud AIO intentionally trails upstream releases by weeks/months for stability.

I can't disable the Text app because users need to be able to edit files using the Nextcloud mobile apps. It looks like disabling the Text app removes the ability to edit files on mobile, even if you install the Plain Text editor.

Is there something else I can do, even if it's temporary, to make sure the files on my instance are safe?

[asaage]

Is there something else I can do, even if it's temporary, to make sure the files on my instance are safe?

find nextcloud_data_dir -type f -name *.md  -exec chmod ugo-w {} \;

🤷‍♂️ but you probably should not try that.

[bronson]

Heh, the old security-to-utility tradeoff. That does make them a lot safer!

Is there any way to still be able to edit the files (both web and mobile) and also prevent the Text App from wrecking the files?

[timkgh]

From Nextcloud 25 on you can run

./occ config:app:set --value false text rich_editing_enabled Which will make text preserve all chars in markdown files.

It appears that enabling this option in v25 results in the Readme.md file in folders being rendered as plain text now in the Files app.

[max-nextcloud]

From Nextcloud 25 on you can run ./occ config:app:set --value false text rich_editing_enabled Which will make text preserve all chars in markdown files.

It appears that enabling this option in v25 results in the Readme.md file in folders being rendered as plain text now in the Files app.

That's an interesting observation. My gut reaction was... "Of course they are - why would this not apply to them" - but I guess the interesting difference is showing file content (which by definition cannot change the formatting) vs. editing file content. So maybe it would be good if this only applied to editing.

On the other hand - if i share a file read only right now people can still watch me edit it. This only works because both parties use the same mode - either richtext or source code. So read only shares probably should still be affected by this. It's only the rich workspace preview that should show rendered content.

[bronson]

Last week I had user suffer data loss when they copied a Markdown file with a simple table into Nextcloud. I explained how they need to be very careful with Markdown files and walked them through reverting to a previous version.

Later, I was considering whether I should send an email to all my users describing what happens if you click on Markdown files, and I saw the About box:

And realized ... that's false. Nextcloud isn't a safe place for all your data.

It's a safe place for most of your files, but it's really dangerous for Markdown files. And that doesn't appear to be on a path to be fixed any time soon.

If I submitted a patch to make rich_editing_enabled default to false, do you think it would be accepted?

If not, then I suppose the slogan in the About box could be corrected?

Those are honest questions. I'm more than happy to submit those pull requests.

[max-nextcloud]

Last week I had user suffer data loss when they copied a Markdown file with a simple table into Nextcloud. I explained how they need to be very careful with Markdown files and walked them through reverting to a previous version.

Could you provide a markdown file with a similar table that shows the same behavior?

[max-nextcloud]

If I submitted a patch to make rich_editing_enabled default to false, do you think it would be accepted?

I assume you don't have it enabled locally, right? Otherwise it would have failed to fix the problem in question - and making it the default would not help.

Before proposing to make it the default - could you try it out locally? We could really use more feedback:

• Does it fix the problem?
• How does it work for your users?
• Does it create other problems like the one about showing the source in rich workspaces?

[max-nextcloud]

Preserve specific markdown syntax

I looked into the options to preserve markdown syntax some more. markdown-it already exposes some of the properties of the parsed code in it's data representation:

1. markdown property on <ul> and <li> tokens shows which char they used for marking the list.
2. markdown property for emphasis and strong emphasis can be * or _ and ** or __.
3. markdown property for <hr> collecting the chars used to create the hr.
4. content property for <p> will contain the entire content of the given paragraph but leading whitespace on the first line.

With 1., 2. and 3. it would be fairly easy to hand down the property and fix the corresponding changes applied to markdown files.

4. is a bit trickier because we need to ensure changing the content of the paragraph clears the content property.

The only thing left from the initially reported markdown would be leading whitespace on the first line of a paragraph. Tackling that seems a little harder but doable.

Options on the table

I understand that there's probably other syntax of markdown extensions out there we would still break. That's not acceptable so I don't like any of the options on the table so far:

• breaking markdown files syntax - even if it's rare and only on hardly used extensions is not a viable option as many of you have layed out.
• disabling markdown rendering for everyone on all .md files by default is not an option either. It's way too important to many of our users to put it behind a feature toggle.
• using a different extension would be similar to using a feature flag - it would break the exising .md files for people who expect them to contain richtext rather than code.

Possible next steps within text

I think there's a lot we can do - also on a shorter time frame. I'd be very happy to help with this if people want to tackle it:

• fix the easy things as listed above.
• allow picking the editing mode per file rather than per instance.
• detect if a file would be serialized differently and ask the user how they want to proceed:
  • open read only
  • open as code
  • open as rich text and accept the changes

I don't like the warning the user option - but currently i don't see any other way to ensure .md syntax only changes if the user is okay with it. I guess we could make the warning behavior the default and allow overwriting it with an instance wide setting.

[Sieboldianus]

Thanks Max for laying this out. I think you are on the right track. I just want to highlight:

there's probably other syntax of markdown extensions out there we would still break

There are so many Markdown extensions that you will inevitably break something unexpected. I am not a full developer, but just look at my list:

• mkdocs documentation, with many extensions, e.g. Python Markdown Extensions, e.g. Caret, Mark & Tilde, pymdownx.details, pymdownx.superfences, InlineHilite, Snippets, SmartSymbols etc.
• hugo static site genartor, again with a big list of Markdown plugins for special syntax handling and its own template engine; every Hugo theme basically adds custom Markdown syntax handling. You would have to check all of them to make sure you do not break something.
• jekyll-reveal slides - these will be mixed Markdown & HTML, I have all my 50+ Presentations that would need to be checked; not to mention the thousands of Jekyll CI pipelines for Static Sites that exist out there, with custom bibliography hooks added to Markdown, variable handling, citations styles etc.
• jupyter notebooks converted to Markdown files using Jupytext - I usually track my *.ipynb files in git using Markdown from jupytext. Modifying these can directly lead to breaking code

If you really want to stick with Markdown, maybe an option could be to add a header that specifies that the files was generated by Nextcloud and the user allows it to be edited, e.g.:

---
nextcloud_generated: True
allow_edit: True
---

btw. you may ask why I add mkdocs, hugo, jekyll, or notebook folders to Nextcloud, when I already have them in git? Often, I need to continue editing from home, so I need a current version of git repositories including all the local static files excluded through .gitignore (test data, .env files, output) - I let Nextcloud sync these files, so I always have a current view of my Repo's additonal data accessible online somewhere. I do exclude .git/ itself. I think this is not an uncommon way to use Nextcloud - but I may be wrong, in this case the documentation should be changed to highlight that Nextcloud is not recommended for developers.

[timkgh]

I agree, it sounds very difficult to try to preserve the format of md files saved by other tools. People have their own linters and formatters in VSCode, Obsidian, etc. to format md files just they way they want and there's no way Nextcloud can win this fight. There should be a mode where the file is read-only in Nextcloud, it does its best to just render it as HTML for viewing purposes only and optionally if you want to edit just edit in plain text. For example I use Obsidian and I just want my md files sync'ed to Nextcloud and view them occasionally in the web interface, best effort rendering in the Nextcloud web interface, but not edit them. Or if I must edit them in Nextcloud, just drop me in a plain text editor.

I also like the suggestion above about adding metadata to md files to allow edit in Nextcloud.

[dumblob]

allow picking the editing mode per file rather than per instance.

Whatever, I do not mind at all unless we keep the default behavior faulty.

I really want only 1 thing: defaults have to be absolutely 100% safe no matter what.

[timkgh]

I really want only 1 thing: defaults have to be absolutely 100% safe no matter what.

+1 Data loss for a file storage system is a BIG NO-NO.

[bronson]

Could you provide a markdown file with a similar table that shows the same behavior?

I can't provide the affected file but I just verified that the file from https://github.com/nextcloud/text/issues/772 still suffers data loss:

The whitespace is necessary for anyone not using a WYSIWYG editor. And frontmatter is still getting clobbered.

The Markdown file: Car.md

[bronson]

If I submitted a patch to make rich_editing_enabled default to false, do you think it would be accepted?

I assume you don't have it enabled locally, right? Otherwise it would have failed to fix the problem in question - and making it the default would not help.

Well, making it the default would stop Nextcloud from messing up Markdown files! That, above all, is my intent here. I simply want Nextcloud to stop corrupting user files. As long as the Nextcloud project sees some file corruption as acceptable, my life as a Nextcloud administrator is more difficult.

As for your question, I can't enable it yet. I'm on AIO, and the AIO release will remain on 24 for at least a few more weeks as they solve some migration issues.

Before proposing to make it the default - could you try it out locally?

Here's the thing: with Nextcloud I really REALLY try to stay with the defaults. Any time I start customizing my install, my users start tripping over odd bugs, and then I'm spending time on forums and reproducing and filing issues (I learned that lesson in a big way with file encryption).

So, here's my position: if you've decided that it's time to stop corrupting files, then I'm happy to help! Let's change the default and solve the bugs. I'm in.

But, if "safe Markdown files" continues to be a niche feature hidden behind an obscure global option, then no, I don't think I will change it. Too much risk. My users have been dealing with Markdown messups for 2 years now. One day we may get used to it.

[PhrozenByte]

@max-nextcloud: https://github.com/nextcloud/text/issues/593#issuecomment-1323244299

• detect if a file would be serialized differently and ask the user how they want to proceed:
  • open read only
  • open as code
  • open as rich text and accept the changes

As far as I remember this was in discussion before and implementing it wasn't as easy as one might think. However, I'm having a hard time right now finding the discussion and the reasoning why it didn't work :unamused: As far as I remember there was some issue with TipTap/ProseMirror randomly changing the resulting Markdown source, so that files appeared to be changed even though they weren't. @juliushaertl, do you remember this discussion? It was in the early days of the app...

Anyway, can you please open a separate issue for this @max-nextcloud? IMHO your idea is worth closer examination :+1:

[max-nextcloud]

I opened a separate issue to discuss the idea of querying the user how to proceed. I will also bring this up as a topic for a design review as I am not a designer and there are probably ways to make the user experience better.

At the same time i will start separating issues for the different problems reported here thus far. I will also start documenting how to debug and fix such issues.

There are quite a few of them that should be good first issues to become familiar with heart of the text code base. If any of you would be interested in helping with this please ping me. I'm happy to jump on a call to investigate issues together, discuss approaches and review pull requests.

• [x] #3478 - should be good to get started. I added some initial pointers.
• [x] #1297 - might be fixed by the one above. If not might be a good follow up.

[bronson]

Since this issue seems to have become the default for this problem, maybe the title could be generalized?

From:

Markdown files from external editors lose formatting on save

To something like:

Markdown files from external editors experience data loss on save

"Loss of formatting" doesn't really describe what's being talked about here.

[nimishavijay]

This long running discussion seems like an interesting question for a design review call! We can discuss this in the upcoming call on our design team public Talk channel on Tuesday 29 Nov at 14:00 CET. Cc @max-nextcloud and @jancborchardt who will be there and anyone else who is interested in dropping by :)

[jancborchardt]

So @juliushaertl and I briefly talked about this before cause a friend of mine also remarked this to me – and I fully agree. Most of the time I use Nextcloud Notes on mobile, or my desktop text editor, but sometimes when I use Text on web, some formatting gets busted which of course is not nice. Even if it’s just turning bullet points from - to *

As far as I understood @juliushaertl we can only fix this step by step individually, e.g.:

• https://github.com/nextcloud/text/issues/3478 as you mentioned @max-nextcloud
• And I also agree with preserving the whitespace in a table as @bronson mentioned.

What probably helps is to not have one very long issue about "formatting loss" but to make sure to open individual issues for those. @bronson @timkgh @dumblob @Sieboldianus @inthreedee could I ask you to open individual issues about the things you mention in this issue and the specific problems you encounter? Then we can better isolate, because some might be easier to fix and others much more difficult. :)

Thank you!

[Sieboldianus]

What probably helps is to not have one very long issue about "formatting loss" but to make sure to open individual issues for those. @bronson @timkgh @dumblob @Sieboldianus @inthreedee could I ask you to open individual issues about the things you mention in this issue and the specific problems you encounter? Then we can better isolate, because some might be easier to fix and others much more difficult. :)

Thank you!

I am sorry if this sounds rude, but I still believe you're on the wrong track trying to mitigate the issue by looking at individual issues. I would have to be omniscient to know all the different problems with todays or future's Markdown files of different apps of different users to be able to list all the potential issues. Since I use Markdown as code, any modification (even if it is just a single dot, or a single space character - e.g. Python) would introduce a bug and render the code useless. This is not acceptable and I will not enable the text app, unless I can trust that there is a 100% way to prevent this. I also don't use the wysiwyg feature and I have no need for it (because, well, Markdown was designed for a different purpose, to work well without an editor). So atm the text-app is disabled and it will stay that as long as you try to fix individual issues, instead of changing the whole strategy of dealing with Markdown files.

[susnux]

Since I use Markdown as code, any modification (even if it is just a single dot, or a single space character - e.g. Python) would introduce a bug and render the code useless. This is not acceptable and I will not enable the text app

To me this sounds like using MS Word as an IDE.

The text app is "advertised" as an simple, visual, collaborative text editor, not as a code editor. I think most users will use it for taking notes or as the nextcloud alternative for etherpads.

So if you are really looking for only a markdown editor, then nextcloud markdown might be the better alternative.

---

For the most of the discussion here I think the it is going in a circle and is going to be emotional. The most complaints are that the source is modified, even if the rendered result is the same. I think this should be communicated more to the users, which I think is now tracked by #3477 so I fully agree with Jan about reporting the other issues you encountered as separate issues

[newhinton]

The text app is "advertised" as an simple, visual, collaborative text editor, not as a code editor.

The issue is that markdown IS code. It is a markup language that follows rules that are not standarized. That means that editors have to treat it like code, otherwise stuff breaks.

This discussion circles around, yes. Arguments are repeated on both sides, but to most users this discussion feels that the main "bug" is not adressed at all. The only thing that was suggested is to "fix symptoms" aka fixing some syntax issues that popularly arise.

It does nothing to solve the actual issue, nextcloud treating markdown NOT as code. As long as that is the case, people will (rightfully) complain.

[Sieboldianus]

To me this sounds like using MS Word as an IDE.

The text app is "advertised" as an simple, visual, collaborative text editor, not as a code editor. I think most users will use it for taking notes or as the nextcloud alternative for etherpads.

I am not looking for a markdown editor at all. I am looking for protecting my existing Markdown files, edited elsewhere (Jupytext, Hugo, mkdocs etc.), to not be modified by Nextcloud, when I simply want to view them (no edit). I can live with plain-text view.

[PhrozenByte]

I am sorry if this sounds rude, but I still believe you're on the wrong track trying to mitigate the issue by looking at individual issues. I would have to be omniscient to know all the different problems with todays or future's Markdown files of different apps of different users to be able to list all the potential issues. Since I use Markdown as code, any modification (even if it is just a single dot, or a single space character - e.g. Python) would introduce a bug and render the code useless. This is not acceptable and I will not enable the text app, unless I can trust that there is a 100% way to prevent this.

I don't want to sound rude either, but you're expecting Text to square the circle. Text is a WYSIWYG editor, thus it can't - by definition - guarantee that no modifications happen. Text can only try to reduce the number of modifications - and that's the reason why Text can only do this step-by-step. This issue is about identifying and reducing the number of modifications.

I also don't use the wysiwyg feature and I have no need for it (because, well, Markdown was designed for a different purpose, to work well without an editor). So atm the text-app is disabled and it will stay that as long as you try to fix individual issues, instead of changing the whole strategy of dealing with Markdown files.

I'm a bit confused. If you don't need Text, if you've already disabled Text, and if you don't want to change that, what is your issue? You can still use Plain text editor, or Markdown editor, or disable Text's rich text editing. Nobody forces you to use Text's rich text editing if it doesn't fit your needs?

---

So, I want to ask you people once again not to repeat the same arguments and claims over and over again. You made your point. You brought up indeed totally valid points, but considering the overall picture different decisions were made. Maybe different decisions would have been made in retrospective, but it is what it is. Please consider our code of conduct. Please don't get me wrong: I rather want to thank you for your great input! :+1: Any additional input about specific compatibility issues is very welcome! :heart: I just want to ask you to please leave the debate on principles.

[inthreedee]

So, to quickly summarize things on a high level here to make sure everyone's voice is heard and we're all on the same page. What's happened is that:

1. The Text team has chosen and committed to a particular set of technologies in the app to achieve their goals for the app.
2. A set of users is saying that this way of doing things breaks existing files and workflows. They say that fixing most of those breakages is insufficient and leaves the app harmful, leading to data loss/corruption.

Where we are now after all the discussion here:

3. The Text team wants to avoid drastic overhauls or changes to the app. Weighing the downsides of that vs the downsides of point 2 above, they currently believe that the major changes required to fix point 2 to be worse.
4. This set of users feels the opposite and that not prioritizing data integrity is worse for users.

The tension in the discussion here is a result of the irreconcilable nature of points 3 and 4. At the moment, the Text team is saying, essentially, "We know but we can't fix it".

(edited to update language based on feedback below)

[bronson]

Yes, that sounds correct. Great summary.

I feel like you’re using minimizing language though, and that’s been a source of a lot of the confusion around this issue.

I’d write the first sentence to point 2 to say: “Some users have mentioned that the set of technologies the Text team has chosen reliably corrupts existing Markdown files.”

And point 4 would be: “As long as the text team prioritizes application features above the integrity of existing Markdown files, users will continue to experience data loss.”

(edited to say thanks for the updates :-)

[inthreedee]

@bronson Good points, I updated the language in the summary. Trying to be fair and accurate.

[max-nextcloud]

My impression is that we are making progress in this discussion as soon as we avoid either-or-thinking.

I think any solution we come up with has to:

• Avoid data loss / changing the .md syntax without consent.
• Allow users who never heard of markdown to use the editor to format text with the default settings.

I proposed a design call because I think good design can take different concerns into consideration and come up with a solution that provides for one AND the other.

My assumption thus far is that people not familiar with markdown will not be able to use a markdown editor that exposes markdown syntax to them. I will also try and challenge that assumption. Which markdown based editor would you recommend for people who have not heard of markdown before?

Right now my thinking is that fixing the main encoding issues will actually make a solution that prevents data loss entirely a much easier sale because it will make interventions to prevent data loss happen less often.

[newhinton]

@max-nextcloud I dont think the issue is with the editor itself, at least for me i can say i actually do like the ui and the idea very much!

The best case would be to not touch .md files at all. As somewhere suggested nextcloud could use it's own ending, or maybe even .txt. That would solve everything at once, except that this would probably a breaking change.

However, there are some ways this could be implemented, at least partly:

• For new files, use .txt as a file ending.
• For existing .md-files, show a warning before a change is persisted, or before opening the file. In that dialog, a conversion could be added so that it is stored as a .txt and the .md is deleted.

This could also be implemented with a custom ending.

The (obvious) drawbacks:

• There is now a warning for existing files. To the user, probably not clear why, because it only has technical reasons.
• There is quite a bit code to maintain just for that

The pros:

• Markdown files are not accidentially broken. Breakages only occur deliberatly.
• The "ownership" of the files are exclusively to nextcloud. Any changes to markdownsyntax or otherwise can be made without concern of other applications.
• There is probably a lot of code less to maintain to fix eventual formatting issues that exist right now

That idea is a policy-decision that needs to be made by the dev's.

[bronson]

How did Tuesday's meeting go? Does anyone have a feel for how much time until Markdown files will be safe in Nextcloud again?

[max-nextcloud]

How did Tuesday's meeting go?

My impression was that it went pretty well. I think we were 7 people discussing. One hour was definitely way too little time to come to a final conclusion.

Preserve syntax by opening files in code editing mode

For me the main takeaway was a better design for the situation where syntax would change. Instead of warning the user we just open the document in code editing mode and have an info, why there's no wysiwyg functionality. It would be good to allow switching to wysiwyg - but that would not be required for a start from my point of view.

I updated https://github.com/nextcloud/text/issues/3477 accordingly.

Discussion about introducing per user setting

For some time we discussed that being able to open some documents in code editing mode and others in wysiwyg might even allow for a per user setting to choose the preferred mode. However that might be confusing:

1. Alex creates a document in wysiwyg mode.
2. Berta defaults to code editing mode and opens the doc later, changes it to syntax text would change.
3. Alex comes back and has no more wysiwyg mode.

Of course this can also happen when Berta uses a different editor all together. However introducing a per user setting seems like asking for this kind of trouble. If 2. happens while the session of 1. is still around Berta might end up in wysiwyg mode despite her per user setting.

We also discussed locking the file and disallowing collaboration when in code editing mode - to avoid such confusion. But once non-text markdown syntax is used that could still lead to confusion as illustrated above.

So right now it looks like the mode will only be chosen based on the document.

Individual Issues

We also discussed individual issues a bunch. Not much new there. We want to fix them. Formatting tables nicely seems to be an obvious one.

Does anyone have a feel for how much time until Markdown files will be safe in Nextcloud again?

It's definitely to big a change to be fixed in or backported to 25. Whether it can land in 26 will also depend on help from the community. We haven't seen any interest in this from customers. So that limits the amount of resources we can throw at it from the company side.

[bronson]

I'm glad the meeting went well. You covered a lot of ground.

As you might imagine, I'm surprised where you ended up. It sounds like Nextcloud will continue to be an unsafe palce for Markdown files while unkown developers fix some difficult problems. This feels like the sort of thing that could take a year or more to solve reliably, doesn't it?

Earlier in this thread, people proposed solutions that fix this 100% that can be implemented right now. I would be able to put together either of these patches myself if they would be welcomed:

1. Use a different file extension for the collaborative WYSIWYG editor (.mdc has been suggested) and open existing .md files in the code editor.
2. Set rich_editing_enabled default to false. (because file integrity should not be opt-in)

Then Nextcloud would be safe for Markdown files again while the WYSIWYG editor continues to improve. Is there any chance I could help here? I'd really like Nextcloud to be a safe place for user files again.