double bracket parsing

[ccoupe]

Github (known as GFM in kramdown) has this double [[this is here ]] link syntax which github renders to http:/this.site/wiki/this-is-here.md Kramdown seems to ignore this in its parsing and we (ebook) really want it if we hope to create TOC and sidebar menus. Ideas @BackOrder , @passenger94 ??

[IanTrudel]

The best approach would be to extend the parser to handle the double brackets. It would allow you to both convert and build TOC at the same time. You may however consider a simple pattern matching and then conversion to a more standard [this is here](http://this.site/wiki/this-is-here.md) and then feed Kramdown.

Take a look at Libraries extending the functionality of kramdown.

Would you point out a reference to Github documentation about this double brackets link feature?

[ccoupe]

I haven't found a reference to the double bracket style yet but it clearly works at github and I must have learned once. Currently the kramdown parse renders it as text which would need special casing everywhere text is allowed (following list items in case) but couple be used anywhere.

[IanTrudel]

Do you mean that double brackets could occur in multiple, or any, Kramdown convert methods? Or just in convert_text method?

My understanding of Kramdown HTML converter convert_root is used for that purpose. As you can see, it builds the TOC right there. convert method might also be a candidate for a gsub on the double brackets.

[ccoupe]

I have captured the '[[this]]' in convert_text and that's likely to be the only convert_xx method it would appear in. Since I know the top nav document (not checked in yet), I can drag in the in the other menu .md from there. The next trick I need to discover/create is proper data structure for the render phase.

[ccoupe]

Turns out that the Shoes manual markdown uses [[ internal_doc_link ]] syntax and it obviously works in Shoes.The manual is mysterious place.

[IanTrudel]

A little Christmas magic! This is a good news.

[ccoupe]

This is a good news.

Not really good news. As noted above. kramdown doesn't call any of it's convert_xxx methods for [[ x ]] - it's just text to kramdown. There may be some option or subclass thing available in kramdown but it's not going to be simple.

[IanTrudel]

Subclass is absolutely possible. I had provided in this issue links to existing custom parsers. Otherwise a simple gsub in the main convert method would do.

[ccoupe]

On the other hand, @passenger94 code to convert the manual with redcloth into .md replaced the numerous (lots and lots) [[x]] to [x] without the expected (url) following. Those we might be able to detect in convert_a since attr href doesn't exist. I don't know. It's a show stopper for converting the Shoes manual to .kd and none very pleasant if it shows up in GFM .md

[passenger94]

Not sure to follow correctly but, if you are sure that in the end the manual is going to be a mardown at its basic form why not convert every annoyance you've met into a proper/well defined, no brainer markdown, and rebuilt from that ? No backward compatibility issues, no ? (could be a little tedious at times, but advantage is starting from fresh and clean), what do i miss ? I mean building the other way around : a basic md and from there specialize for github, html or Shoes

[IanTrudel]

This would be a good approach but we need to maintain compatibility with GFM because it means we can publish on both Shoes and github Wiki at any given time. Your initial approach was a gsub in the code you submitted, hence I suggested the same in the method convert. The libraries also do similarly for such cases and well before doing more complicated things, i.e. own parsers. So far we are talking about one tag only. Not sure it is as big of a deal as @ccoupe makes it seems like. :)

[ccoupe]

[[ is ]] is not detected in the convert_a. It's treated as simple text. A gsub is fine if we have enough context at the time. Perhaps it is as simple as [[this]] will be converted to shoes

 'link "this", {open_link "this"}

and open_link (from help-picky.rb) can figure it out from there. Or something else.

[IanTrudel]

Have a try in convert rather than convert_a !

[ccoupe]

@passenger94 , The more I look at the Shoes manual code (help.rb and derivatives) I do believe converting the Shoes manual to an ebook.kd would require a manual rewrite. Help.rb and the weird markup are tied in a tight knot. Never to be untied. kramdown is just too different. Also, supporting that weirdness should not be a design goal of this ebook project. Maybe somebody can add it later.

[IanTrudel]

@ccoupe Would it be possible for you to list the current hurdles? Maybe additional eyes would help. If I know which markup are giving you troubles, I could write a proof-of-concept.

[ccoupe]

Proof of concept has run it's course. That's been done. Don't need anymore POC. Git clone the ebook, clone the wiki. Understand the difference between creating Shoes/ruby to mimic kd vs interpreting manual markdown -- very different from POC. You could certainly work on the kd_render.rb for syntax highlighting and there are some some weird conversion bugs/situations in your kd_render POC.

At the moment I have partial menus on the left side but somehow the title bar is stopped displaying. Go figure. I can create images that use local files instead of downloading. and they display just fine when the render engine is properly connected. Currently it's not but once those are working, You'll can certainly help. Just forget about what Shoes manual does and think like what an gituhb/md author wants

[IanTrudel]

How about you make a simple md file containing the special cases that I could look over and give a try? It seems from what you wrote there are more parsing problems than just double brackets.

Just forget about what Shoes manual does and think like what an gituhb/md author wants

Sure, Shoes Ebook is already beyond this point.

[ccoupe]

Sadly you you have to run any test case (md) from a .yaml created by ebook_builder and that requires the full ebook and wiki clone - or the yaml for the test (chapter-8 of the wiki). The only test framework for an md is ebook_builder.rb

[IanTrudel]

Thanks for the pointer. Would it be chapter-8 only or also including its subsections?

[ccoupe]

Actually, chapter-8/Plot-Widgett.md of the wiki has many of the difficult cases (images, codeblocks ) Chapter 8 only has one md in it and there is no subsections/methods (that's a Shoes manual thing to forget). Think of the whole wiki - chapter-1, chapter-2 ... Those are menu titles on the left sidebar and when you click on one, the you get the expanded curated list articles (each of them happens to to be an .md that has gone through the kramdown conversion into Shoes/Ruby code.

That navigation/display part is not working at the moment so before you dig into the bugs in kd-render you'll have to wait for me to fix the nav/display - next commit.

[IanTrudel]

How about using Kramdown's own GFM parser?

[IanTrudel]

@ccoupe I would highly recommend you to investigate Kramdown's GFM parser. Alternatively, you will find the following custom parser being able to successfully parse and convert GFM links. Running the code on chapter-8.md will yield: <kd:gfmlink {"gfmlink"=>"Plot Widget"}>.

require("kramdown")
require("pp")

module Kramdown
   module Parser
      class GfmLink < Kramdown
         def initialize(*doc)
            super
            @block_parsers.unshift(:gfmlink)
         end

         GFMLINK_START = /\[\[(.*?)\]\]/u

         def parse_gfmlink
            @src.pos += @src.matched_size
            el = Element.new(:gfmlink, nil, {'gfmlink' => @src[1]})
            @tree.children << el
         end
         define_parser(:gfmlink, GFMLINK_START, '\[\[')
      end
    end

    module Converter
      class Shoes < Base
         def initialize(root, options)
            super
         end

         DISPATCHER = Hash.new {|h,k| h[k] = "convert_#{k}"}

         def convert(el)
            send(DISPATCHER[el.type], el)
         end

         def convert_root(el)
            results = []
            el.children.each do |inner_el|
               results << send(DISPATCHER[inner_el.type], inner_el)
            end
            results
         end

         def convert_blank(el)
            %{para("\n")}
         end

         def convert_text(el)
            %{para("#{el.value}", :margin_left => 0, :margin_right => 0)}
         end

         def convert_header(el)
            %{para(strong("#{el.options[:raw_text]}\n"), :margin_left => 6, :margin_right => gutter)}
         end

         def convert_p(el)
            results = []
            el.children.each do |inner_el|
               results << send(DISPATCHER[inner_el.type], inner_el)
            end
            %[flow(:margin_left => 6, :margin_right => gutter) { #{results.join(";")} }]
         end

         def convert_ul(el)
            results = []
            el.children.each do |inner_el|
               results << send(DISPATCHER[inner_el.type], inner_el)
            end
            results
         end

         def convert_li(el)
            results = []
            el.children.each do |inner_el|
               results << %[flow(:margin_left => 30) { fill black; oval -10, 10, 6; #{send(DISPATCHER[inner_el.type], inner_el)} }]
            end
            results
         end
         #alias :convert_ol :convert_ul
         #alias :convert_dl :convert_ul

         def convert_smart_quote(el)
            %{para("'", :margin_left => 0, :margin_right => 0)}
         end

         def convert_a(el)
            results = []
            #pp el.attr['gfmlink']
            el.children.each do |inner_el|
               results << inner_el.value if inner_el.type.eql?(:text)
               #send(DISPATCHER[inner_el.type], inner_el)
            end
            %[para(link("#{results.join}") { open_url("#{el.attr['href']}") }, :margin_left => 0, :margin_right => 0)]
         end

         def convert_gfmlink(el)
            pp el
         end
      end
   end
end

doc = Kramdown::Document.new(File.read("chapter_8.md"), {input: 'GfmLink'}).to_shoes

[ccoupe]

I use the kramdown option input: 'GFM' which is supposed to tell kramdown to use that flavor of markdown. It's a yaml setting (defalut) w/o a GUI way to change it.

[IanTrudel]

Alright. Hope the custom parser solves the problem.

[passenger94]

The more I look at the Shoes manual code (help.rb and derivatives) I do believe converting the Shoes manual to an ebook.kd would require a manual rewrite.

yep, basically what i'm suggesting, take this ebook opportunity to start from something clean/standard markdown, make it the editable reference, then with scripts (regex, gsub, kramdown converts/custom, travis and friends maybe ??) adapt for the target (github, shoes, ...) (as i didn't look at code i may be totally off, apologizes if so)

[IanTrudel]

A manual rewrite is absolutely unnecessary. Both Shoes Manual and Markdown languages are of similar nature. It may however need some more tuning. Once again, @ccoupe share with us the special cases then @passenger94 and I look over them.

Converting Shoes Manual to Markdown should be a one-time thing in any case. Shoes Manual language will be subsequently deprecated.

[ccoupe]

load_docs and dewikify... define the the Shoes manual parsing. Figure out how to do that in the kramdown converter and ask yourself is it worth it the effort to use md for the manual.

[IanTrudel]

Shouldn't Kramdown conversion to Shoes objects entirely supersedes dewikify? load_docs also doesn't make much sense anymore when everything is sifted through Kramdown.

Perhaps we see things differently. The Picky proof-of-concept always been a full search engine (ftsearch) replacement that would lean on existing Shoes manual (load_docs, dewikify). However, the Kramdown proof-of-concept always meant to be a full replacement of Shoes manual. The whole point is to get rid of Shoes Manual format and let Kramdown becomes the defacto backend.

[IanTrudel]

I glanced at the code in this repository and it seems you are already doing as it should. However, now I don't know what you are talking about with dewikify.

[ccoupe]

I had to resort to a little paper drawing of the output screen we want. on the top is title bar (and icon) on the left side is the sidebar and the remaining space is where the chosen content is displayed. What would the sidebar look like if it was Shoes wiki? We'd have entries for Intro, Chapter-1, Chapter-2, ...,Chapter-8 Mentally click the chapter-2 link and what drops down? 11 titles of Articles (each with some kramdown from 1 of 11 md files, converted to Shoe/Ruby waiting to be drawn. in the content area.

That's very different from the Manual/help.rb view of one big document where the sidebar content is controlled by the =, == ,=== lines in the file in that single document. As all Shoes applications do, help.rb mixes up the model/view/controller. I've tried not to do that in the kramdown parser because we wanted to pre-generate those picky databases and possibly the ruby from kramdown. That requires knowing the TOC and contents (mostly) independent of help.rb mixing things up.

One could write a different program that uses one huge file and have the kramdown converter play dynamic TOC Shoes gui games when it gets a #, ##, ### and friends but you have populate the picky DB as you discover things in the kramdown converter and it's likely you'll need to track and update internal state of kramdown parse in Shoes terms. That certainly can be done and might be useful for some authors but it would be a different program.

It's very similar to the difference between compiling and interpreting.

[IanTrudel]

The compilation approach should do fine for now (and a long while). The challenges versus rewards as far as interpretation goes may be too little at the moment. You also mentioned before that rendering to Shoes is too slow for runtime interpretation. So the important is to get a decent working first version.

[ccoupe]

Performance is not too bad interpreting one md, although I'm not indexing picky yet.

There are many, many, many things to do. From minor to head scratchers.

1. Whats the proper line break setting for GFM - you can see the it gets confused Is that a yaml setting and should the ebook-builder gui capture that.
2. codeblocks are ignored (kd_render.rb)
3. There sidebar is too narrow for long winded titles.
4. default font choices and size are not optimal for many elements. Shoes this be user settable? If so write a gui screen to edit them for the yaml settings
5. Missing the 'Home' or Intro chapter link on the sidebar - I'll this one next.
6. Haven't done the [[go to here"]] detection and processing, and testing
7. Doesn't handle mult-level documents - another task I'll do.

[IanTrudel]

The screenshot looks good. Progress is welcomed.

Whats the proper line break setting for GFM - you can see the it gets confused Is that a yaml setting and should the ebook-builder gui capture that.

You may need gfm_quirks and possibly hard_wrap.

codeblocks are ignored (kd_render.rb)

How about codespan ?

default font choices and size are not optimal for many elements. Shoes this be user settable? If so write a gui screen to edit them for the yaml settings

We should set the default. May indeed also be user settings.

Haven't done the [[go to here"]] detection and processing, and testing

The sample parser I provided here will help you with this. You only have to focus on module Parser and convert_gfmlink in module Convert. You might want to use it as a mixins, subclass and/or any other way you see fit that would allow you to use both custom parser and GFM (or user settings).

[ccoupe]

After some experimenting, if we can gsub [[7.0 Contributed Tips]] to [[7.0 Contributed Tips]]() then convert_a gets something we can use

convert a called <kd:a { "href" => "" } { :location => 5, :ial => nil } [<kd:text "[7.0 Contributed Tips]" nil { :location => 5 }>]>

just a small matter of writing a good regexp to append ()

[ccoupe]

I decided to follow @BackOrder hints above and subclass Kramdown::Parser::GFM and I think if got something that will possibly work. Just a matter digging through the ebook structures to find the target in order to build the para(link), :click