HTMLPagesCreator doesn't work properly on Linux

[SicroAtGit]

In Issue #13 we found out that the tool doesn't work correctly under Linux.

I would like to dedicate a new issue to this problem here, because it has nothing to do with the actual problem in issue #13.

[SicroAtGit]

Executing: _assets/HTMLPagesCreator.pb

[12:09:52] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HTMLPagesCreator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[12:09:52] Debug Level: 4
[12:09:52] Project's Root Path: '/home/alexander/Programmieren/Git/PBCodeArcProto/'
[12:09:52] ================================================================================
STEP 1/4 | Build Categories List
================================================================================
[12:09:52] Scanning project to build list of categories and resources:
[12:09:52]  |-- /_TEST/  (ignore folder)
[12:09:52]  |-+ /Gadget/
[12:09:52]  | |-+ /ScintillaGadget/
[12:09:52]  | | |-- index.html  (ignore file)
[12:09:52]  | | |-- /../  (ignore folder)
[12:09:52]  | | |-- /./  (ignore folder)
[12:09:52]  | | |-- ScintillaText/CodeInfo.txt
[12:09:52]  | | |-- README.md  (ignore file)
[12:09:52]  | |
[12:09:52]  | |-+ /HyperLinkGadget/
[12:09:52]  | | |-- index.html  (ignore file)
[12:09:52]  | | |-- /../  (ignore folder)
[12:09:52]  | | |-- /./  (ignore folder)
[12:09:52]  | | |-- CursorHovering[Mac].pbi  
[12:09:52]  | | |-- README.md  (ignore file)
[12:09:52]  | |
[12:09:52]  | |-- index.html  (ignore file)
[12:09:52]  | |-- /../  (ignore folder)
[12:09:52]  | |-- /./  (ignore folder)
[12:09:52]  | |-- PrintGadget[Mac].pb  
[12:09:52]  | |-- Modul_MouseOver[Win,Lin].pbi  
[12:09:52]  | |-- Modul_AddEvents[Win].pbi  
[12:09:52]  | |-- GetGadgetClass[Mac].pbi  
[12:09:52]  | |-- Hide_or_change_cursor_over_gadgets_without_OS-API.pb  
[12:09:52]  | |-- TransformationOfGadgetsAtRuntime.pbi  
[12:09:52]  | |-- Tooltip[Win].pbi  
[12:09:52]  | |-- NamedGadgetData.pbi  
[12:09:52]  | |-- MultiGraph[Win]/CodeInfo.txt
[12:09:52]  | |-- WindowManager_ParentGadget_ParentWindow_ScaleWindow.pbi  
[12:09:52]  | |-- README.md  (ignore file)
[12:09:52]  |
[12:09:52]  |-- /_assets/  (ignore folder)
[12:09:52]  |-+ /Date/
[12:09:52]  | |-- index.html  (ignore file)
[12:09:52]  | |-- /../  (ignore folder)
[12:09:52]  | |-- /./  (ignore folder)
[12:09:52]  | |-- FormatDateEx.pbi  
[12:09:52]  | |-- Date64[Win,Lin,Mac].pbi  
[12:09:52]  | |-- SimpleTimer.pbi  
[12:09:52]  | |-- ISOWeek.pbi  
[12:09:52]  | |-- README.md  (ignore file)
[12:09:52]  | |-- DayLightSavingTime.pbi  
[12:09:52]  | |-- DaysInMonth.pbi  
[12:09:52]  |
[12:09:52]  |-- index.html  (ignore file)
[12:09:52]  |-- /../  (ignore folder)
[12:09:52]  |-- /.git/  (ignore folder)
[12:09:52]  |-- /./  (ignore folder)
[12:09:52]  |-- /_morgue/  (ignore folder)
[12:09:52]  |-- .gitattributes  (ignore file)
[12:09:52]  |-- .gitignore  (ignore file)
[12:09:52]  |-- PBCodeArcREADME.md  (ignore file)
[12:09:52]  |-- /_ERRORS/  (ignore folder)
[12:09:52]  |-- /_tempwork/  (ignore folder)
[12:09:52]  |-- Concept.md  (ignore file)
[12:09:52]  |-- .nojekyll  (ignore file)
[12:09:52]  |-- README.md  (ignore file)
[12:09:52] 
[12:09:52] - Categories found: 4 (excluding root folder)
[12:09:52] - Resources found: 18 (2 are subfolders)
[12:09:52] Root Categories:
[12:09:52]   1. 'Gadget'
[12:09:52]   2. 'Date'
[12:09:52] ================================================================================
STEP 2/4 | Checking Project Integrity
================================================================================
[12:09:52] meta.yaml file seems Ok.
[12:09:52] All tests passed.
[12:09:52] ================================================================================
STEP 3/4 | Process Categories
================================================================================
[12:09:52] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CATEGORY 1/5 | ./
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[12:09:52] Category name: ''
[12:09:52] Category path: ''
[12:09:52] Current working dir: /home/alexander/Programmieren/Git/PBCodeArcProto/
[12:09:52] path2root$: ''
[12:09:52] BREADCRUMBS:
--------------------------------------------------------------------------------
<li><a href='index.html'>Home</a></li>

--------------------------------------------------------------------------------
[12:09:52] pathLevels: 0
[12:09:52] subPaths: 0
[12:09:52] pathSeg1: 
[12:09:52] pathSeg2: 
[12:09:52] pathSeg3: 
[12:09:52] SIDEBAR:
--------------------------------------------------------------------------------
<li><a href='Gadget/index.html'>Gadget</a></li>
<li><a href='Date/index.html'>Date</a></li>

--------------------------------------------------------------------------------
[12:09:52] SubCatLinks:
--------------------------------------------------------------------------------

---

# Subcategories

- [Gadget](./Gadget/index.html)
- [Date](./Date/index.html)

--------------------------------------------------------------------------------
[12:09:52] !!! Current Category has no Resources !!!
[12:09:52] >>>>>>>>>> PandocConvert() >>>>>>>>>>
[12:09:52] > PANDOC CURR DIR: /home/alexander/Programmieren/Git/PBCodeArcProto/
[12:09:52] > PANDOC OPTS: -f markdown_github+yaml_metadata_block+raw_attribute --template=/home/alexander/Programmieren/Git/PBCodeArcProto/_assets/template.html5 --eol=native -V ROOT= -V BREADCRUMBS="<li><a href='index.html'>Home</a></li>
" -V SIDEBAR="<li><a href='Gadget/index.html'>Gadget</a></li>
<li><a href='Date/index.html'>Date</a></li>
" -o index.html
[12:09:52] <<<<<<<<<< PandocConvert() <<<<<<<<<<
[12:09:52] \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
********************************************************************************
[12:09:52] PANDOC ERROR — An error was encountered while interacting with pandoc.

Pandoc exited with error (1):
 | pandoc: href='index.html'>Home</a></li>: openBinaryFile: does not exist (No such file or directory)

[12:09:52] Aborting program execution...
[12:09:52] ********************************************************************************
////////////////////////////////////////////////////////////////////////////////

[SicroAtGit]

At first I thought the problem existed because the parameters passed to Pandoc contain line breaks. But removing the line breaks didn't change the error.

options = RemoveString(options, #EOL)
Pandoc = RunProgram("pandoc", options, currDir, #PANDOC_PROC_FLAGS)

[SicroAtGit]

If these two lines are commented out, the tool runs completely without errors. The error is very probably there. https://github.com/tajmone/PBCodeArcProto/blob/a3e0615fbba75222bb6dcdbbc795dde80e4f01ab/_assets/HTMLPagesCreator.pb#L766-L767

[SicroAtGit]

If I rewrite the code like this, the tool runs completely:

; Although the variable values are within quotation marks, Pandoc has problems
; with whitespaces here
BREADCRUMBS$ = RemoveString(BREADCRUMBS$, " ")
SIDEBAR$     = RemoveString(SIDEBAR$, " ")

pandocOpts.s = "-f " + #PANDOC_FORMAT_IN +
               " --template=" + ASSETS$ + #PANDOC_TEMPLATE +
               " --eol=native" +
               " -V ROOT=" + path2root$ +
               ~" -V BREADCRUMBS=\"" + BREADCRUMBS$ + ~"\"" +
               ~" -V SIDEBAR=\"" + SIDEBAR$ + ~"\"" +
               " -o index.html"

; Line breaks in parameters aren't allowed
pandocOpts = RemoveString(pandocOpts, #EOL)

[tajmone]

Yes, command line variables were intened for simple usage, not such long strings with EOLs and special chars.

I'll change the whole pandoc invocation part and instead of passing variables via parameters I'll define them via a YAML string, and then join together the markdown source file, the YAML settings file and the autogenerated YAML block as a single string which I then pass to pandoc via STDIN.

This way we'll have no problems on any OS, no compatibility break with pandoc updates, and improve speed because some YAML settings file are used at each pandoc invocation, and we can keep them in memory and avoid redundant disk access. Also the final HTML source will look cleaner.

[tajmone]

No More Pandoc Vars Passed Via CLI Options: YAML Only!

Now I've rewritten the code parts that used to build raw HTML tags for the Breadcrumbs bar and the Sidebar NavMenu. Instead of raw HTML, now the app builds a YAML block with structured variables, as a string which is injected after the markdown page contents (which is fed to pandoc via STDIN).

So, now no more variables are being passed via command line options, and we don't have to deal with any long strings, special chars, and EOLs. My guess is that this will have fixed any comptability problems with Linux and macOS.

The template is now much cleaner as it uses pandoc control keywords to iterate through the various entries of Breadcrumbs and Sidebar (check the commit diff):

</section>
$------------------------------------------------------------>>> BREADCRUMBS BAR
<div id="breadcrumbsbar">
  <div class="container">
    <nav class="breadcrumb" aria-label="breadcrumbs">
    <ul>
      <li><a href="$ROOT$index.html">Home</a></li>
$for(breadcrumbs)$
      <li><a href="$breadcrumbs.link$">$breadcrumbs.text$</a></li>
$endfor$
    </ul>
    </nav>
  </div>
</div>

When you have time (no hurry), let me know if you still get errors when building now. I'm 100% optimist it should work fine now.

[SicroAtGit]

Wonderful! Now the tool runs almost error-free under Linux.

Now I have found another problem: It looks like NextDirectoryEntry() doesn't navigate the directory structure identically under all operating systems, so you should sort the file lists alphabetically:

$ git diff Gadget/ScintillaGadget/index.html
diff --git a/Gadget/ScintillaGadget/index.html b/Gadget/ScintillaGadget/index.html
index 496c401..2560d87 100644
--- a/Gadget/ScintillaGadget/index.html
+++ b/Gadget/ScintillaGadget/index.html
@@ -53,12 +53,12 @@
           Categories
         </p>
         <ul class="menu-list">
-          <li><a HREF="../../Date/index.html">Date</a></li>
           <li><a class="is-active" HREF="../../Gadget/index.html">Gadget</a>
           <ul>
-            <li><a HREF="../../Gadget/HyperLinkGadget/index.html">HyperLinkGadget</a>
             <li><a class="is-active" HREF="../../Gadget/ScintillaGadget/index.html">ScintillaGadget</a>
+            <li><a HREF="../../Gadget/HyperLinkGadget/index.html">HyperLinkGadget</a>
           </ul></li>
+          <li><a HREF="../../Date/index.html">Date</a></li>
         </ul>
       </aside>

[tajmone]

I didn't think of that — but, of course, PB leverages the OS API for similar tasks, so it was to be expected.

Good that you caught this!

Since all categories are stored in lists, SortStructuredList() should handle the job fine.

I'll fix it ASAP!

[tajmone]

I've now ensured that all lists that could affect the order of elements in the final HTML page are sorted equally on all OS — ie: categories, subcategories and resource files lists, which were built from listing directories via the OS native filesys API, are now being sorted before processing them.

This should have fixed the issue for all OSs.

[SicroAtGit]

Very nice!

At the moment I don't see any more problems. Therefore I close this issue.