Convert html to markdown

mhicauber commented 2 years ago

I was using the ckeditor plugin for redmine (as I was using a really old redmine version, that didn't have any rich text capabilities at the time. Now that I've switch to last redmine version, I'd like to stick to native markdown syntax.

So i want to transform all this html stored content to markdown.

I've used following config, but I'm having an error

Any idea what might be wrong ?

convcfg='[{
  "from_formatting": "html",
  "to_formatting": "common_mark",
  "converters": [
    ["Ws", "http://localhost:9999", { "method": "POST" }]
  ]
}]'
rake reformat:convert to_formatting=common_mark converters_json="$convcfg"  dryrun=1 RAILS_ENV=production

Running with setup:
{:converters_json=>
  "[\n" +
  "  {\n" +
  "    \"from_formatting\": \"html\",\n" +
  "    \"to_formatting\": \"common_mark\",\n" +
  "    \"converters\": [\n" +
  "      [\n" +
  "        \"Ws\",\n" +
  "        \"http://localhost:9999\",\n" +
  "        {\n" +
  "          \"method\": \"POST\"\n" +
  "        }\n" +
  "      ]\n" +
  "    ]\n" +
  "  }\n" +
  "]",
 :to_formatting=>"common_mark",
 :workers=>1,
 :dryrun=>true,
 :from_formatting=>"(based on your Setting.text_formatting: CKEditor)"}
All 1/1 workers started.
Settings
Comment: converting 0/0 non-blank [:content] occurences of 0 total
Document: converting 111/111 non-blank [:description] occurences of 336 total
rake aborted!
No converter found for Document#5
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/converters/configured_converters.rb:14:in `convert'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:238:in `convert'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:227:in `block (2 levels) in migrate_spec'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:226:in `each'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:226:in `map'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:226:in `block in migrate_spec'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:54:in `block in pluck_each'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:39:in `each'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:39:in `pluck_each'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:223:in `migrate_spec'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:160:in `block in migrate_objects'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:159:in `each'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:159:in `migrate_objects'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:128:in `do_migrate'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:115:in `block (3 levels) in call'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/context.rb:40:in `with_cached_projects'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:114:in `block (2 levels) in call'
/srv/redmine/app/models/mailer.rb:634:in `with_deliveries'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:113:in `block in call'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:112:in `call'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/convert_redmine.rb:107:in `call'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/invoker.rb:36:in `convert_redmine'
/srv/redmine/plugins/redmine_reformat/lib/redmine_reformat/invoker.rb:24:in `run'
/srv/redmine/plugins/redmine_reformat/lib/tasks/reformat.rake:13:in `block (2 levels) in <top (required)>'
Tasks: TOP => reformat:convert
(See full trace by running task with --trace)

martincizek commented 2 years ago

Based on the log, I guess your current formatting is called CKEditor, not html.

Also:

Make sure you have the common_mark format installed (it's still not released, see #32424), we have also released Docker images with this patch, links are in the ticket). Or you can use either the old markdown for now.
I'd recommend to fine-tune the external service manually on the rich text sources taken from your Redmine. It's almost certain it won't work as you expect on the first try.

rtyshyk commented 2 years ago

Hi @mhicauber

Did you have success with CKEditor migration to markdown? I have a similar issue.

martincizek commented 2 years ago

Did you have success with CKEditor migration to markdown? I have a similar issue.

The issue in this ticket was the misconfigured from_formatting, it should have been CKEditor, not html.

rtyshyk commented 2 years ago

Hi @mhicauber Yes, I see that. But it is more about the migration in general, like service was used for migration, I mean what is inside of "http://localhost:9999".

As I understand there is no defaukt pipeline to convert html/ckeditor to markdown.

sylvainfaivre commented 2 years ago

I would also like to know how this plugin could be used to convert from CKEditor's html format to either Textile or Markdown format.

I tried using the method provided by redmine_ckeditor :

rails redmine_ckeditor:migrate RAILS_ENV=production FROM=html TO=textile

But it gives poor results with textile, and really awful results with markdown.

rtyshyk commented 2 years ago

I did a straightforward script that converted everything in DB using pandoc from HTML to markdown + remove all HTML attributes using Lua filter, for example, images still have it.

the result is more or less good, except for tables which broken, but we have no many of them.

conversation to commonmark is much better as it is partly support HTML, so tables is fine., but redmine_visual_editor plugging does not support commonmark yet, so I use redcarpet.

sylvainfaivre commented 2 years ago

Thanks for the hint @rtyshyk. Would you care to share your conversion script ?

Using the command line I shared above to convert from html to Textile, some formatting was kept : <strong> <em> <h2> <img> <ul>

but some was lost :

everything using <span style=""> for text color or other properties
quotes, hyphens and underscores were changed to html entities
some tables were converted OK, but some were replaced by html code

rtyshyk commented 2 years ago

It is php app, I was expecting to use it as a webserver for redmine_format, and do something complicated... but then just decided to do changes directly in db and keep in simple...

To convert it use docker run --env DATABASE_URL='mysql://user:password@mysql:3306/redmine?serverVersion=8.0' rtyshyk/redmine-html-convert php -d memory_limit=1024M bin/console app:convert

To see sources docker run rtyshyk/redmine-html-convert cat /app/src/Command/ConvertCommand.php

sylvainfaivre commented 2 years ago

It seems that your server doesn't allow unauthenticated access :

$ docker run registry.711media.de/711media-internal/redmine/converter cat /app/src/Command/ConvertCommand.php
Unable to find image 'registry.711media.de/711media-internal/redmine/converter:latest' locally
docker: Error response from daemon: Head "https://registry.711media.de/v2/711media-internal/redmine/converter/manifests/latest": denied: access forbidden.

If the code is not too long, maybe you could post it here, as it is relevant to this issue's topic.

rtyshyk commented 2 years ago

@sylvainfaivre I updated comment above with proper image name https://github.com/orchitech/redmine_reformat/issues/10#issuecomment-1143576979

orchitech / redmine_reformat

Convert html to markdown #10