search

msprev / panzer

pandoc + styles
BSD 3-Clause "New" or "Revised" License
159 stars 15 forks source link

Problem working with stdin #25

Closed mickley closed 7 years ago

mickley commented 7 years ago

panzer doesn't seem to work well with stdin. This is unfortunate because a number of tools (eg the Atom editor) can use a custom markdown processor (ie pandoc) but won't work with panzer.

For example if I run something like: 

cat file.md | panzer

it won't work. It detects the style, but not everything runs. Furthermore, pandoc doesn't seem to be getting the full file

The panzer output (which doesn't include postflight, filters, or html-specific options that I've defined): 

 ----- pandoc read -----
         running
         ----- style definitions -----
         global:
           AddLatexRefs  Article  Base       BookReview  Braindead   
           CV            Draft    Letter     Manuscript  Marking     
           Mumford       Notes    NotesSans  Plain       Thesis-Draft
           Thesis-Final                                              
         ----- document style -----
         style:
           Thesis-Draft
         full hierarchy:
           Base, Manuscript, Thesis-Draft
         writer:
           html
         ----- pandoc read with metadata options -----
         pandoc reading with options:
           --smart
         ----- run list -----
           empty
         ----- pandoc write -----
         pandoc writing with options:
           --self-contained --standalone --dpi=300

The output to stdout is just a blank html doc as if pandoc got nothing:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <title></title>
  <style type="text/css">code{white-space: pre;}</style>
</head>
<body>

</body>
</html>
mickley commented 7 years ago

pandoc itself parses it correctly:

cat file.md | pandoc --self-contained --standalone

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <title></title>
  <style type="text/css">code{white-space: pre;}</style>
</head>
<body>
<p>Hi there</p>
</body>
</html>
msprev commented 7 years ago

The example you give runs fine for me. Are you sure you are running the latest version of panzer? panzer has recently been updated as pandoc 1.18 has introduced some significant breaking changes that required changes on panzer's side too.

Can you reproduce this error?

BTW, there is 1 peculiarity worth noting with panzer in its handling of stdin. Panzer needs to slurp all its stdin input before starting to process -- i.e. read until an EOF. This is because metadata in a md pandoc doc can occur anywhere. Panzer needs to know all the metadata before starting work (otherwise style inheritance won't work, for example). This would not explain the error you are experiencing, but it's worth noting that by design panzer could never be a stream based processor continually eating stdin input and spewing output. pandoc is in a similar position I believe for different reasons.

mickley commented 7 years ago

Hmm, alright I isolated this to the --smart flag. Things work correctly as expected without that flag. Pandoc itself has no trouble though when using --smart.

Example:

file.md

---
style: Test
...

# Hello World!

styles.yaml

Test:
    all:
        commandline:
            smart: true
            standalone: true
    html:
        commandline:
            self-contained: true

cat file.md | panzer produces this output:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <title></title>
  <style type="text/css">code{white-space: pre;}</style>
</head>
<body>

</body>
</html>

cat file.md | pandoc --smart --standalone --self-contained produces this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <title></title>
  <style type="text/css">code{white-space: pre;}</style>
</head>
<body>
<h1 id="hello-world">Hello World!</h1>
</body>
</html>
msprev commented 7 years ago

I believe I've fixed it now. It was a small error in the logic for reading command line arguments. cat x.md | panzer - would work, but cat x.md | panzer did not. Both work now.

Thank you for catching this. Please let me know if any other issues remain.

mickley commented 7 years ago

Hmm, that didn't fix it for me.
cat file.md | panzer using the example above still produces an empty HTML document.

mickley commented 7 years ago

cat file.md | panzer - works though. Double-checked that I'm building off the latest commit.

msprev commented 7 years ago

I can't reproduce this. Both work fine for me after this fix. I think that you might still not be using the latest commit.

Can you try this on the command line:

printf "---\nstyle: Test\nstyledef:\n  Test:\n    all:\n      commandline:\n        smart: true\n...\n hello" | panzer
printf "---\nstyle: Test\nstyledef:\n  Test:\n    all:\n      commandline:\n        smart: true\n...\n hello" | panzer -

If you are using the latest commit, both should produce the same output <p>hello</p>. If not, then only the second should produce output.

mickley commented 7 years ago

Alright I have both working now, but it took an uninstall to make it work.

python3 setup.py install --force wasn't sufficient.

msprev commented 7 years ago

Great. I'm not sure why the update didn't work, but glad to know that the issue is fixed.