Closed joelkuiper closed 10 years ago
Ah yeah. We still have some unsupported Pandoc nodetypes. You can usually get around these by tweaking the markdown a bit.
Can you post your input markdown file? We'll be fixing the converter when we get some time.
It's the one from your example https://github.com/substance/tutorial
Ow... that is not good. Will have a look.
I just did a checkout this morning. These are the relevant versions (I think)
~ » npm --version 1.3.11 ~ » node --version v0.10.21 ~ » pandoc --version pandoc 1.12.1 Compiled with texmath 0.6.5.2, highlighting-kate 0.5.5. Syntax highlighting is supported for the following languages: actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog, clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d, diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang, fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc, javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell, lua, makefile, mandoc, matlab, maxima, metafont, mips, modula2, modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml, octave, pascal, perl, php, pike, postscript, prolog, python, r, relaxngcompact, rhtml, roff, ruby, rust, scala, scheme, sci, sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, texinfo, verilog, vhdl, xml, xorg, xslt, xul, yacc, yaml Default user data directory: /home/joel/.pandoc Copyright (C) 2006-2013 John MacFarlane Web: http://johnmacfarlane.net/pandoc This is free software; see the source for copying conditions. There is no warranty, not even for merchantability or fitness for a particular purpose.
Just tried another fresh install on my Macbook, same issue!
Thanks, I will take a look...
let me see if I can dig something up as well. The tool looks great but I still need to find a way to move from Emacs org-mode (.org) files to the json without too much work
@joelkuiper Could you post the output of
pandoc --version
BTW, this is how it looks for me (Ubuntu 12.04, pandoc 1.12.0.2)
/tmp $ git clone https://github.com/substance/tutorial
Cloning into 'tutorial'...
WARNING: gnome-keyring:: couldn't connect to: /tmp/keyring-ztcCZb/pkcs11: No such file or directory
remote: Counting objects: 76, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 76 (delta 22), reused 62 (delta 11)
Unpacking objects: 100% (76/76), done.
/tmp $ cd tutorial/
/tmp/tutorial (master) $ cd 001-convert-md-to-substance
/tmp/tutorial/001-convert-md-to-substance (master) $ npm install
npm WARN package.json substance-tutorial-001@0.1.0 No description
npm WARN package.json substance-tutorial-001@0.1.0 No README data
npm http GET https://registry.npmjs.org/substance-converter/0.1.0
npm http 304 https://registry.npmjs.org/substance-converter/0.1.0
npm http GET https://registry.npmjs.org/substance-article/0.1.0
npm http GET https://registry.npmjs.org/substance-util/0.2.0
npm http GET https://registry.npmjs.org/underscore
npm http GET https://registry.npmjs.org/substance-document/0.5.0
npm http GET https://registry.npmjs.org/xmlhttprequest/1.5.0
npm http GET https://registry.npmjs.org/request
npm http 304 https://registry.npmjs.org/substance-article/0.1.0
npm http 304 https://registry.npmjs.org/substance-util/0.2.0
npm http 304 https://registry.npmjs.org/underscore
npm http 304 https://registry.npmjs.org/substance-document/0.5.0
npm http 304 https://registry.npmjs.org/xmlhttprequest/1.5.0
npm http 304 https://registry.npmjs.org/request
npm http GET https://registry.npmjs.org/substance-application/0.1.0
npm http GET https://registry.npmjs.org/substance-operator/0.2.0
npm http GET https://registry.npmjs.org/substance-nodes/0.1.0
npm http GET https://registry.npmjs.org/substance-chronicle/0.2.0
npm http GET https://registry.npmjs.org/substance-data/0.8.0
npm http GET https://registry.npmjs.org/substance-regexp/0.1.0
npm http GET https://registry.npmjs.org/forever-agent
npm http GET https://registry.npmjs.org/tunnel-agent
npm http GET https://registry.npmjs.org/http-signature
npm http GET https://registry.npmjs.org/aws-sign
npm http GET https://registry.npmjs.org/cookie-jar
npm http GET https://registry.npmjs.org/node-uuid
npm http GET https://registry.npmjs.org/oauth-sign
npm http GET https://registry.npmjs.org/mime
npm http GET https://registry.npmjs.org/form-data
npm http GET https://registry.npmjs.org/hawk
npm http GET https://registry.npmjs.org/qs
npm http GET https://registry.npmjs.org/json-stringify-safe
npm http 304 https://registry.npmjs.org/substance-nodes/0.1.0
npm http 304 https://registry.npmjs.org/substance-chronicle/0.2.0
npm http 304 https://registry.npmjs.org/substance-application/0.1.0
npm http 304 https://registry.npmjs.org/substance-operator/0.2.0
npm http 304 https://registry.npmjs.org/substance-data/0.8.0
npm http GET https://registry.npmjs.org/estraverse
npm http GET https://registry.npmjs.org/esprima
npm http GET https://registry.npmjs.org/escodegen
npm http GET https://registry.npmjs.org/MD5
npm http GET https://registry.npmjs.org/resolve
npm http 304 https://registry.npmjs.org/substance-regexp/0.1.0
npm http 304 https://registry.npmjs.org/forever-agent
npm http 304 https://registry.npmjs.org/tunnel-agent
npm http 304 https://registry.npmjs.org/http-signature
npm http 304 https://registry.npmjs.org/aws-sign
npm http 304 https://registry.npmjs.org/cookie-jar
npm http 304 https://registry.npmjs.org/node-uuid
npm http 304 https://registry.npmjs.org/oauth-sign
npm http 304 https://registry.npmjs.org/mime
npm http 304 https://registry.npmjs.org/form-data
npm http 304 https://registry.npmjs.org/hawk
npm http 304 https://registry.npmjs.org/qs
npm http 304 https://registry.npmjs.org/json-stringify-safe
npm http 304 https://registry.npmjs.org/estraverse
npm http 304 https://registry.npmjs.org/esprima
npm http 304 https://registry.npmjs.org/escodegen
npm http 304 https://registry.npmjs.org/MD5
npm http GET https://registry.npmjs.org/combined-stream
npm http GET https://registry.npmjs.org/async
npm http 304 https://registry.npmjs.org/resolve
npm http GET https://registry.npmjs.org/ctype/0.5.2
npm http GET https://registry.npmjs.org/assert-plus/0.1.2
npm http GET https://registry.npmjs.org/asn1/0.1.11
npm http GET https://registry.npmjs.org/charenc
npm http GET https://registry.npmjs.org/crypt
npm http GET https://registry.npmjs.org/source-map
npm http GET https://registry.npmjs.org/boom
npm http GET https://registry.npmjs.org/sntp
npm http GET https://registry.npmjs.org/hoek
npm http GET https://registry.npmjs.org/cryptiles
npm http 304 https://registry.npmjs.org/combined-stream
npm http 304 https://registry.npmjs.org/async
npm http 304 https://registry.npmjs.org/asn1/0.1.11
npm http 304 https://registry.npmjs.org/ctype/0.5.2
npm http 304 https://registry.npmjs.org/assert-plus/0.1.2
npm http GET https://registry.npmjs.org/delayed-stream/0.0.5
npm http 304 https://registry.npmjs.org/charenc
npm http 304 https://registry.npmjs.org/crypt
npm http 304 https://registry.npmjs.org/source-map
npm http 304 https://registry.npmjs.org/boom
npm http 304 https://registry.npmjs.org/sntp
npm http GET https://registry.npmjs.org/amdefine
npm http 304 https://registry.npmjs.org/hoek
npm http 304 https://registry.npmjs.org/cryptiles
npm http 304 https://registry.npmjs.org/delayed-stream/0.0.5
npm http 304 https://registry.npmjs.org/amdefine
substance-converter@0.1.0 node_modules/substance-converter
├── substance-util@0.2.0
├── xmlhttprequest@1.5.0
├── underscore@1.5.2
├── substance-document@0.5.0 (substance-regexp@0.1.0, substance-operator@0.2.0, substance-data@0.8.0, substance-chronicle@0.2.0)
├── request@2.24.0 (json-stringify-safe@5.0.0, forever-agent@0.5.0, aws-sign@0.3.0, qs@0.6.5, tunnel-agent@0.3.0, oauth-sign@0.3.0, cookie-jar@0.3.0, node-uuid@1.4.1, mime@1.2.11, http-signature@0.10.0, form-data@0.1.2, hawk@1.0.0)
└── substance-article@0.1.0 (substance-operator@0.2.0, substance-nodes@0.1.0, substance-application@0.1.0)
/tmp/tutorial/001-convert-md-to-substance (master) $ node convert.js lorem_ipsum.md lorem_ipsum.json
/tmp/tutorial/001-convert-md-to-substance (master) $ pandoc --version
pandoc 1.12.0.2
Compiled with texmath 0.6.4, highlighting-kate 0.5.5.
Syntax highlighting is supported for the following languages:
actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog,
clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d,
diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang,
fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc,
javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell,
lua, makefile, mandoc, matlab, maxima, metafont, mips, modula2, modula3,
monobasic, nasm, noweb, objectivec, objectivecpp, ocaml, octave, pascal,
perl, php, pike, postscript, prolog, python, r, relaxngcompact, rhtml, roff,
ruby, rust, scala, scheme, sci, sed, sgml, sql, sqlmysql, sqlpostgresql,
tcl, texinfo, verilog, vhdl, xml, xorg, xslt, xul, yacc, yaml
Default user data directory: /home/oliver/.pandoc
Copyright (C) 2006-2013 John MacFarlane
Web: http://johnmacfarlane.net/pandoc
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.
tutorial/001-convert-md-to-substance [master●] » pwd /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance tutorial/001-convert-md-to-substance [master●] » npm install npm WARN package.json substance-tutorial-001@0.1.0 No description npm WARN package.json substance-tutorial-001@0.1.0 No README data npm WARN package.json substance-converter@0.1.0 No repository field. tutorial/001-convert-md-to-substance [master] » node convert.js lorem_ipsum.md lorem_ipsum.json /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js:196:15 (topLevelNode ) /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js:146:23 (document ) /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js:116:17 (import ) /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/converter.js:81:34 (convert ) /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/converter.js:60:9 (ChildProcess.) events.js:98:17 (ChildProcess.EventEmitter.emit ) child_process.js:789:12 (Process.ChildProcess._handle.onexit ) { message: 'Node not supported: t', name: 'ImporterError', code: -1, __stack: [ { func: 'topLevelNode ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js', line: '196', col: '15' }, { func: 'document ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js', line: '146', col: '23' }, { func: 'import ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js', line: '116', col: '17' }, { func: 'convert ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/converter.js', line: '81', col: '34' }, { func: 'ChildProcess. ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/converter.js', line: '60', col: '9' }, { func: 'ChildProcess.EventEmitter.emit ', file: 'events.js', line: '98', col: '17' }, { func: 'Process.ChildProcess._handle.onexit ', file: 'child_process.js', line: '789', col: '12' } ] } tutorial/001-convert-md-to-substance [master] » pandoc --version pandoc 1.12.1 Compiled with texmath 0.6.5.2, highlighting-kate 0.5.5. Syntax highlighting is supported for the following languages: actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog, clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d, diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang, fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc, javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell, lua, makefile, mandoc, matlab, maxima, metafont, mips, modula2, modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml, octave, pascal, perl, php, pike, postscript, prolog, python, r, relaxngcompact, rhtml, roff, ruby, rust, scala, scheme, sci, sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, texinfo, verilog, vhdl, xml, xorg, xslt, xul, yacc, yaml Default user data directory: /Users/joelkuiper/.pandoc Copyright (C) 2006-2013 John MacFarlane Web: http://johnmacfarlane.net/pandoc This is free software; see the source for copying conditions. There is no warranty, not even for merchantability or fitness for a particular purpose.
Ok... seems to be a problem with the new pandoc version. I will dig into it and return to you when this is fixed.
Thanks!
Concerning Emacs org-mode (.org) files... I don't know if they can be processed with pandoc? Could you create a gist with one example file?
Well I can do org-mode > markdown > json
, if we can get this to work :wink:
Doesn't seem to be the pandoc version, I did a install of your version through cabal
but no success
tutorial/001-convert-md-to-substance [master●] » node convert.js lorem_ipsum.md lorem_ipsum.json /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js:196:15 (topLevelNode ) /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js:146:23 (document ) /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js:116:17 (import ) /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/converter.js:81:34 (convert ) /Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/converter.js:60:9 (ChildProcess.) events.js:98:17 (ChildProcess.EventEmitter.emit ) child_process.js:789:12 (Process.ChildProcess._handle.onexit ) { message: 'Node not supported: t', name: 'ImporterError', code: -1, __stack: [ { func: 'topLevelNode ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js', line: '196', col: '15' }, { func: 'document ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js', line: '146', col: '23' }, { func: 'import ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/pandoc_importer.js', line: '116', col: '17' }, { func: 'convert ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/converter.js', line: '81', col: '34' }, { func: 'ChildProcess. ', file: '/Users/joelkuiper/Desktop/tutorial/001-convert-md-to-substance/node_modules/substance-converter/src/converter.js', line: '60', col: '9' }, { func: 'ChildProcess.EventEmitter.emit ', file: 'events.js', line: '98', col: '17' }, { func: 'Process.ChildProcess._handle.onexit ', file: 'child_process.js', line: '789', col: '12' } ] } tutorial/001-convert-md-to-substance [master●] » pandoc --version pandoc 1.12.0.2 Compiled with texmath 0.6.5.2, highlighting-kate 0.5.5. Syntax highlighting is supported for the following languages: actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog, clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d, diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang, fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc, javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell, lua, makefile, mandoc, matlab, maxima, metafont, mips, modula2, modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml, octave, pascal, perl, php, pike, postscript, prolog, python, r, relaxngcompact, rhtml, roff, ruby, rust, scala, scheme, sci, sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, texinfo, verilog, vhdl, xml, xorg, xslt, xul, yacc, yaml Default user data directory: /Users/joelkuiper/.pandoc Copyright (C) 2006-2013 John MacFarlane Web: http://johnmacfarlane.net/pandoc This is free software; see the source for copying conditions. There is no warranty, not even for merchantability or fitness for a particular purpose.
Hmpf. And ufff...
Then, could you post the output of
pandoc -f markdown -t json lorem_ipsum.md
that uuh seemed to have worked I guess
After formatting https://gist.github.com/joelkuiper/7359963
Ok... it looks crappy.... I will post the output from mine pandoc that you see the problem...
I think the problem is related to AESON.. don't know why it behaves that way for you....
Hmm that's a completely different json format it seems. Any other library version what might cause that? I did a fresh install on both OS X Mavericks and Ubuntu 13.10 of the entire Node.js and Haskell stack, so I am slightly confused.
I know that AESON lately introduced something to control the JSON serialization... Maybe we should ask John McFarlane?
Sorry, one of the things I did not manage to become yet, is a haskell dude ;)
@jgm we're having some trouble with the different json outputs from pandoc any thoughts?
Ow it says so pretty clearly in the changelog
ext.Pandoc.Definition: Changed default JSON serialization format. Instead of {"Str": "foo"}, for example, we now have {"t": "Str", "c": "foo"}. This new format is easier to work with outside of Haskell. Incidentally, “t” stands for “tag”, “c” for “contents”
I guess I can see if I can either change the output definition of pandoc or the parser in this lib
Ok... then it's my part to update the converter... But why did it do the same with 1.12.02 for you? .... nevermind...
This will take a day...
I have no idea why it didn't work with the old version. Perhaps something was still cached, but on the upside your initial guess was correct!
I'm currently going over the packages to see if I can somehow get the old version pandoc to work
Thank you by the way :+1:
Hmmmm... don't invest too much time...(or leave it right away)
Actually, I considered this change after 12.01 ... should be actually five minutes of work... However, you would need to stick with the edge version... as I feel there are some side-effects to make an instant release... I return to you with instructions...
And thanks to figure that out :)
ok... it's not a 5 min task... however I will fix it today...
Yeah I thought it'd be a quick change from Str to something else, but it's a change from Str to {t: "Str", c: "foo"} which you're in a better position to fix!
Exactly ;)
On 07.11.2013 21:05, Joël Kuiper wrote:
Yeah I thought it'd be a quick change from Str to something else, but it's a change from Str to -> {t: Str, c: "foo"} which you're in a better position to fix!
— Reply to this email directly or view it on GitHub https://github.com/substance/converter/issues/11#issuecomment-28001199.
Almost there... https://travis-ci.org/substance/converter/builds/13659364
I prepare a substance-tutorial and substance-io branch that uses the adapted converter. We will release that the next days.
I fixed the converter to work with Pandoc 1.12.1.
The tutorial is not yet fully working (example 2+3). However, The conversion example (1) should work now.
@joelkuiper could you check it?
git clone https://github.com/substance/tutorial
git checkout pandoc_1_12_1
Then do the example 001
As you said the conversion seems to be without error now, but the display (second tutorial) gives an empty page (fine on master though)
@michael could you take over here?
I added a bundled version of substance-io@pandoc_1_12_1
to the tutorial@pandoc_1_12_1
.
However, there is something wrong with the integration for the examples 2+3.
@joelkuiper in the meantime you could have a look at the development version of substance-io.
Follow the instructions as given in https://github.com/substance/io/blob/0.2.x/README.md#development-environment, but using the pandoc_1_12_1 branch:
git clone -b pandoc_1_12_1 https://github.com/substance/io.git
Then clone substance/docs and adapt it to show one of your markdown files.
Cool that seems to work! Does the substance-io thing also have an option to export a single document (I saw there was a bundle command for io, but couldn't get it to work) ?
I'll have a look at the examples...
@joelkuiper substance-io
takes a doc library as an input, for a single document you can just use the converter script plus the bundled Substance Reader. Looking at the examples now.
We did it :)
The tutorial comes now with an updated Substance Reader bundle and a Pandoc 1.12.1 compatible converter. 63b770c3d2937b33899359ab0c418717311dee9d
Many thanks both!
On Fri, Nov 8, 2013 at 3:44 PM, Oliver Buchtala notifications@github.comwrote:
Closed #11 https://github.com/substance/converter/issues/11.
— Reply to this email directly or view it on GitHubhttps://github.com/substance/converter/issues/11 .
www.joelkuiper.eu