::: An unexpected error has occurred.

atanasj commented 6 years ago

Hi,

I am getting the following error with pandocomatic when trying to export from Scrivener using @iandol Scrivomatic workflow.

YAML front matter is:

---
pandocomatic_:
  use-template:
    - docx-refs
---

The relevant sections of my pandocomatic.yaml file are:

# These are pandocomatic templates.
settings:
  recursive: true
  follow-symlinks: false
  skip: ['.*', 'pandocomatic.yaml']
templates:
##############################################################################
##############################################################################-GENERAL
##############################################################################
  refs:
    setup: []
    preprocessors: []
    pandoc:
      standalone: true
      filter:
        # 'pandoc-crossref'
        # 'pandoc-newpage.pl' #honours newpage
        #'@filters/lua-filters-master/tasklist/tasklist.lua' #creates tasklists'
        # 'pandoc-tablenos'
        # 'pandoc-fignos'
        # 'pandoc-eqnos'
        # 'pandoc-xnos'
        - 'pandoc-citeproc' #['pandoc-citeproc'] #process citations
      bibliography: 'MyLib.bib' #JSON is faster than BibTeX, symlinked to ~/.pandoc / chnagned from Core.json to MyLib.bib
      csl: csl/apa-old-doi-prefix.csl
      #citation-abbreviations: cite-abbr.json. This needs to be symlinked from Zotero styles forlder.
      #Zotero styles folders is found: ~/Zotero/styles/
    metadata:
      reference-section-title: "References"
      notes-after-punctuation: false
      link-citations: true
      #linkReferences: true
      #nameInLink: true
      # pandoc-tablenos metadata
      #tablenos-capitalise: On
      #tablenos-caption-name: Table
      #tablenos-cleveref: On
      # pandoc-fignos metadata
      #fignos-capitalise: On
      #fignos-cleveref: On
      #fignos-plus-name: Figure
      #tblPrefix:
      # Table
      # Tables
      #figPrefix:
      # Figure
      # Figures
      #listing: True
      #lofTitle: "# List of Figures"
      #lotTitle: "# List of Tables"
    postprocessors: []
    cleanup: []
#########################################################################
#########################################################################—DOCX
##########################################################################
  docx:
    pandoc:
      from: markdown+smart+implicit_figures+table_captions+multiline_tables+latex_macros+compact_definition_lists+subscript+superscript+yaml_metadata_block+raw_tex+line_blocks+backtick_code_blocks+fenced_code_attributes+link_attributes+tex_math_dollars+native_spans+pandoc_title_block #markdown+compact_definition_listsfenced_code_blocks+
      to: docx
      standalone: true
      filter:
        # '_filters/removeHR' #remove horizontal rules
        # '_filters/authorSimplifyMetadata' #collapse down metadata "author: [name: affiliation:]" to author:
        # '_filters/prependInstitute' #add affiliations paragraph if institute: metadata present
        - 'pandoc-newpage.pl' #honours newpage. This is only relevant to .docx output, since attached to this template.
        # 'filters/lua-filters-master/tasklist/tasklist.lua' #creates tasklists
        # 'pandoc-crossref' # This needs to go before pancoc-cietreproc, so omit now. #crossref filter. Consider using fignos and tablenos instead as mroe stable.
        - 'pandoc-docx-pagebreak'
      reference-doc: '@templates/reference.docx'  #templates/custom.docx
      dpi: 300
##########################################################################
  docx-refs:
    extends: ['docx','refs']
##########################################################################

Please note, that I have hashed out some of the filters as I thought these could be causing the issue, but it still failed. The error log is below:

=== ##################################################### ===
=== Scrivomatic V1.0.11 Report @ 2018-02-22 09:24:24 +1100 ===
 Running under Ruby 2.3.3
 Working directory: /Users/atanas/Dropbox/Uni/PhD/_pandoc–md2docx
====== Input Options: ======
#<struct Scrivomatic::OPT input="'2.md'", output=nil, to=nil, command="pandocomatic", envpath="/Library/TeX/texbin:/Users/atanas/bin:/usr/local/bin", build=false, verbose=true, dry_run=false>
====== Final ENV PATH: ======
/Library/TeX/texbin:/Users/atanas/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
====== TOOL PATHS: ======
---pandoc: /usr/local/bin/pandoc | V: 2.1
---pandocomatic: /usr/local/bin/pandocomatic | V: Pandocomatic version 0.2.2.1
---ruby: /usr/bin/ruby | V: ruby 2.3.3p222 (2016-11-21 revision 56859) [universal.x86_64-darwin17]
---rbenv: /usr/local/bin/rbenv
---python: /usr/bin/python
---xelatex: /Library/TeX/texbin/xelatex

 … running pandocomatic, please wait …
====== COMMAND OUTPUT: ======
:: Running: /usr/local/bin/pandocomatic --debug '2.md' 
::: An unexpected error has occurred. You can report this bug via https://github.com/htdebeer/pandocomatic/issues/new.
::: /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandoc_metadata.rb:81:in `merge!': no implicit conversion of Array into Hash (TypeError)
:::     from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandoc_metadata.rb:81:in `initialize'
:::     from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandoc_metadata.rb:69:in `new'
:::     from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandoc_metadata.rb:69:in `load_file'
:::     from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/command/convert_file_multiple_command.rb:50:in `initialize'
:::     from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandocomatic.rb:100:in `new'
:::     from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandocomatic.rb:100:in `run'
:::     from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/bin/pandocomatic:3:in `<top (required)>'
:::     from /usr/local/bin/pandocomatic:22:in `load'
:::     from /usr/local/bin/pandocomatic:22:in `<main>'
:: exit status: pid 11340 exit 1
!!!---scrivomatic::runCommand() RETURN non-zero value: /usr/local/bin/pandocomatic --debug '2.md' !!!

I have tested the underlying pandoc, pandoc-citeproc, pandoc-fignos and pandoc-tablenos on the md file, and it works as expected. The terminal command I used to test this is:

pandoc -F pandoc-placetable --filter pandoc-tablenos --filter pandoc-fignos --listing -r markdown+smart+implicit_figures+table_captions+pipe_tables+latex_macros+compact_definition_lists+subscript+superscript+yaml_metadata_block+raw_tex+fenced_code_attributes+line_blocks+backtick_code_blocks+fenced_code_blocks+link_attributes+tex_math_dollars+native_spans+pandoc_title_block -t docx -s --bibliography /Users/atanas/Dropbox/Uni/PhD/#Zotero/MyLib.bib -F pandoc-newpage.pl -F pandoc-docx-pagebreak --lua-filter=/Users/atanas/.pandoc/filters/lua-filters-master/task-list/task-list.lua -F pandoc-citeproc --csl /Users/atanas/Zotero/styles/apa-old-doi-prefix.csl -o 2.docx 2.md

The relevant package version numbers are:

pandoc 2.1
Compiled with pandoc-types 1.17.3, texmath 0.10.1, skylighting 0.5.1
Pandocomatic version 0.2.2.1
pandoc-fignos (1.2.0)
pandoc-tablenos (1.2.0)
Scrivomatic V1.0.11

Please let me know if you need more info. Thanks in advance---both for your help, and for a very useful package.

htdebeer commented 6 years ago

If I remove also the filters "pandoc-newpage.pl" and "pandoc-docx-pagebreak", and remove all "@" characters from paths, converting the following "minimal.md" works without a hitch:

---
pandocomatic_:
    use-template:
      - docx-refs
---

this is a test

Can you do the following using pandocomatic:

(a) Try running the same scenario as I did?
If (a) does not work, the issue might be in the difference between our two systems.
If (a) works, (b) add in the two filters again and try again?
If (b) does not work, can you find out which filter is the culprit, if any?
If (b) works, (c)try your original document again?
If (c) does not work, can you create a minimal test markdown input file where the issue does occurs?
If (c) works, (d) try adding the "@" characters to the paths again and

atanasj commented 6 years ago

Thanks for your response.

Right, so. I gave it a crack. I had to really strip back on a lot of things that were both in the pandocomatic.yaml file, as well as the YAML header. Using your header:

---
pandocomatic_:
    use-template:
      - docx-xnos
---

And re-working the pandocomatic.yaml file to be:

# These are pandocomatic templates.
settings:
  recursive: true
  follow-symlinks: false
  skip: ['.*', 'pandocomatic.yaml']
templates:
##############################################################################
##############################################################################-citation-links
##############################################################################
  link:
    pandoc:
    metadata:
      link-citations: true
##############################################################################
##############################################################################-crossreferencing
##############################################################################
  xnos:
    pandoc:
      filter:
        - 'pandoc-tablenos'
        - 'pandoc-fignos'
        - 'pandoc-eqnos'
      tablenos-capitalise: On  # pandoc-tablenos
      tablenos-caption-name: Table  # pandoc-tablenos
      tablenos-cleveref: On  # pandoc-tablenos
      fignos-plus-name: Figure
      fignos-capitalise: On  # pandoc-fignos
      fignos-cleveref: On  # pandoc-fignos
##############################################################################
  refs:
    setup: []
    preprocessors: []
    pandoc:
      standalone: true
      filter: 'pandoc-citeproc'  # process citations
      bibliography: "MyLib.bib"
      csl: "csl/apa-old-doi-prefix.csl"
      metadata:
        notes-after-punctuation: false
    postprocessors: []
    cleanup: []
##############################################################################
##############################################################################—DOCX
##############################################################################
  docx:
    extends: ['refs']
    pandoc:
      from: markdown+smart+implicit_figures+table_captions+multiline_tables+compact_definition_lists+subscript+superscript+yaml_metadata_block+line_blocks+backtick_code_blocks+fenced_code_attributes+tex_math_dollars+native_spans
      to: docx
      standalone: true
      filter:
        - 'pandoc-newpage.pl'  # command \newpage
        - 'pandoc-docx-pagebreak'  # commands PAGEBREAK & PAGEBREAKLANDSCAPE
      reference-doc: "_templates/reference.docx"  #templates/custom.docx
      dpi: 300
      toc: false
##############################################################################
  docx-xnos:
    extends: ['link','xnos','docx']
##############################################################################
  docx-changes:
    extends: ['docx']
    preprocessors: ['preprocessors/criticmarkup'] #process criticmarkup to HTML that pandoc can use
##############################################################################
##############################################################################
##############################################################################

This seems to work now. I tried to insert another section in the .yaml file, such as:

##############################################################################
  crossref:
    padnoc:
      filter:
        - 'pandoc-crossref'
        linkReferences: true  # pandoc-crossref
        nameInLink: true  # pandoc-crossref
        tblPrefix:  # pandoc-crossref
        - Table  # pandoc-crossref
        - Tables  # pandoc-crossref
        figPrefix:  # pandoc-crossref
        - Figure  # pandoc-crossref
        - Figures  # pandoc-crossref
        listing: True  # pandoc-crossref
        lofTitle: "# List of Figures"  # pandoc-crossref
        lotTitle: "# List of Tables"  # pandoc-crossref
##############################################################################

To have as an option, but all tests failed - even when I did not call the template. Do you know why? I have removed for now.

iandol commented 6 years ago

@atanasj - there is a spelling mistake of padnoc and you cannot mix metadata into your filter list I think...

iandol commented 6 years ago

Also, off topic but you do not need to explicitly enable the pandoc extensions, i.e. from: markdown+... — most of these are already enabled, you only need to enable the non-standard extensions.

iandol commented 6 years ago

I wonder if the problem was with the \@ sign in the path in the end? What happens if with your no-working pandocomatic.yaml you now go back to reference-doc: '@templates/reference.docx' ?

htdebeer commented 6 years ago

@iandol is correct in observing that mixing filters and metadata in the filter property is not allowed. So, in your latest question:

This seems to work now. I tried to insert another section in the .yaml file, such as:

##############################################################################
 crossref:
   padnoc:
     filter:
       - 'pandoc-crossref'
       linkReferences: true  # pandoc-crossref
       nameInLink: true  # pandoc-crossref
       tblPrefix:  # pandoc-crossref
       - Table  # pandoc-crossref
       - Tables  # pandoc-crossref
       figPrefix:  # pandoc-crossref
       - Figure  # pandoc-crossref
       - Figures  # pandoc-crossref
       listing: True  # pandoc-crossref
       lofTitle: "# List of Figures"  # pandoc-crossref
       lotTitle: "# List of Tables"  # pandoc-crossref
##############################################################################

To have as an option, but all tests failed - even when I did not call the template. Do you know why? I have removed for now

You could rewrite that example to

crossref:
    pandoc:
        filter:
        - 'pandoc-crossref'
    metadata:
        linkReferences: true  # pandoc-crossref
        nameInLink: true  # pandoc-crossref
        tblPrefix:  # pandoc-crossref
        - Table  # pandoc-crossref
        - Tables  # pandoc-crossref
        figPrefix:  # pandoc-crossref
        - Figure  # pandoc-crossref
        - Figures  # pandoc-crossref
        listing: True  # pandoc-crossref
        lofTitle: "# List of Figures"  # pandoc-crossref
        lotTitle: "# List of Tables"  # pandoc-crossref

Note the difference between the pandoc.filter property and the metadata property. Pandocomatic is quite strict where you can put what: each property needs to be in the correct section.

atanasj commented 6 years ago

Thanks so much @htdebeer and @iandol for your help. The suggested changes / tips have got it all back purring again.

htdebeer / pandocomatic

::: An unexpected error has occurred. #57