MD to PDF - Githubissues

drwetter commented 5 years ago

In the end a PDF should be available.

While github is great in dealing with markdown documents I suspect it's not as easy to generate a handy PDF from the markdown files.

Help or hints would be appreciated.

ghost commented 5 years ago

I believe Read The Docs should be able to do this automatically (Github Markdown to PDF).

https://readthedocs.org/

On Mar 19, 2019, at 9:31 AM, Dirk Wetter notifications@github.com wrote:

In the end a PDF should be available.

While github is great in dealing with markdown documents I suspect it's not as easy to generate a handy PDF from he markdown files.

Help or hints would be appreciated.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

drwetter commented 5 years ago

Oh, yes thanks. Sphinx rings a bell too.

PauloASilva commented 5 years ago

Hi @drwetter, You can check how we did it for Go Secure Coding Practices: we're using gitbook-cli v2.3.0. In the dist folder you'll find the resulting output formats: ePub, MOBI and PDF.

GitBook might also be an option: it is free for open-source & non-profit teams.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Thanks, @PauloASilva

we're using gitbook-cli v2.3.0.

Was that done automatically or did it require manual work?

PauloASilva commented 5 years ago

Hi @drwetter, No manual work required: you just have to run

$ npm install && npm run build

We can do it locally or automate it, for instance, on Travis CI using GitHub Hooks.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

That sounds good!

Mind to help?

PauloASilva commented 5 years ago

That sounds good!

Mind to help?

Not at all, I will be glad to do it :smile:

drwetter commented 5 years ago

Sounds great.

How about starting with it early? I created a separate branch (gitbook) for it so that we can play with it before it looks like it's "the" document?

PauloASilva commented 5 years ago

I'm on it already: I forked your repo and soon I will do a pull request.

I did an initial test but the build failed due to some parsing errors. As soon as I get some output, I'll share it with you so that we can decide how to proceed.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Cool! -- Sent from my mobile. Excuse my brevity&typos+the phone's autocorrection

PauloASilva commented 5 years ago

Hi @drwetter, We have a first draft: please check it on my fork. The resulting formats (PDF, ePub, MOBI) are in the dist root folder.

If we decide to move forward with gitbook, we should:

Create a Makefile to build the book inside an ephemeral docker container. Now I did it by hand

[pauloasilva@XXX Docker-Security]$ docker run -it --rm -v $(pwd):/app node:current-slim bash -c "cd /app && apt-get update && apt-get install -y calibre && npm i && npm run build"

Rename the 000 - Introduction chapter (gitbook expects README.md to be the Introduction chapter)
Validate D04 - Secure Defaults and Hardening.md Markdown as there are some parsing problem regarding dots on a command line sample marked with FIXME.

Please share your thoughts on this.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Am 21. April 2019 13:23:01 MESZ schrieb PauloASilva notifications@github.com:

Hi @drwetter, We have a first draft: please check it on my fork. The resulting formats ([PDF][2], [ePub][3], [MOBI][4]) are in the dist root folder.

If we decide to move forward with gitbook, we should:
Create a Makefile to build the book inside an ephemeral docker container. Now I did it by hand
[pauloasilva@XXX Docker-Security]$ docker run -it --rm -v $(pwd):/app
node:current-slim bash -c "cd /app && apt-get update && apt-get install
-y calibre && npm i && npm run build"
Rename the 000 - Introduction chapter (gitbook expects README.md to be the Introduction chapter)

Validate D04 - Secure Defaults and Hardening.md Markdown as there are some parsing problem regarding dots on a command line sample marked with FIXME.
Please share your thoughts on this.

Cheers, Paulo A. Silva

[2]: https://github.com/PauloASilva/Docker-Security/blob/feature-build/dist/owasp-docker-security.pdf?raw=true [3]: https://github.com/PauloASilva/Docker-Security/blob/feature-build/dist/owasp-docker-security.epub?raw=true [4]: https://github.com/PauloASilva/Docker-Security/blob/feature-build/dist/owasp-docker-security.mobi?raw=true

Hi Paulo,

thanks & Happy Easter!

Will look into it next week. Cheers. Dirk

-- Sent from my mobile. Excuse my brevity&typos+the phone's autocorrection

PauloASilva commented 5 years ago

Hi @drwetter, Happy Easter to you too!

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Hi Paulo,

thanks!

Rename the 000 - Introduction chapter (gitbook expects README.md to be the Introduction chapter)

can't this be configured?

Speaking of it: the font (calibre) looks in my eyes rather painful. A sans-serif font would do much better to the eye. :-)

Validate D04 - Secure Defaults and Hardening.md Markdown as there are some parsing problem regarding dots on a command line sample marked with FIXME

I was working on that last week and correct that today or later. If it holds you up and you don't want to hotfix it in your fork first, please let me know.

Upon opening the PDF I saw the graphics and said "wow" :-)

Cheers, Dirk

drwetter commented 5 years ago

D04 should be fine for you now.

PauloASilva commented 5 years ago

Hi Dirk,

gitbook-cli supports changing the default introduction README.md, but the file name should match the following RegEx ^[a-zA-Z-._\d,\s]+$ what 000 - Introduction.md doesn't. To be honest I don't like the current naming convention because:
- it includes spaces (and symbols), requiring escaping on CLIs
- it mixes lower and upper case letters, what may cause troubles on case sensitive file systems
Would you agree renaming files as follows?
- 000 - Introduction.md -> 000-introduction.md
- D03 - Network Separation and Firewalling.md -> d03-network-separation-firewalling.md
- E11 - What'\''s Next?.md -> e11-whats-next.md
Now I am forcing the fontFamily to sans-serif (gitbook-cli PDF options)
The cover was an Easter Egg :smile:

I'll pull your changes and see if they solve the markdown parsing issue.

Best, Paulo A. Silva

drwetter commented 5 years ago

Hi Paulo,

the Easter Egg arrived on time ;-)

I can work on Readme and the Introduction to rename them.

As far as the naming of the files in general are concerned: I guess it's a matter of taste. At least ^[a-zA-Z-._\d,\s]+$ matches 000 - Introduction.md, see e.g. https://regex101.com/. I am assuming this pattern applies for all filenames?

I am a friend of readable filenames. Why should a human sacrifice readability to a machine? Just because some sloppy shell programmers forgot to quote properly?

Don't know what OS you're using? Even on Linux (which I use) I only have little problems with such filenames. On MacOS and Windows they are common since a longer while back.

In E11 the question mark and quote was too much maybe. I pushed a change.

Cheers! Dirk

PauloASilva commented 5 years ago

Hi Dirk, Please don't change anything on your side because I am not sure what's wrong: although gitbook complains about the file name I am not being able to use a value other than README.md.

I'll let you know when I have figured out what's wrong and how to proceed with the PoC.

I am a Linux user too. After a few years of software development, working with other developers with different OS preferences, I had to adopt a standard for file naming to avoid undesired problems.

The most recent incident happened during the OWASP Top 10 translation. I got a volunteer who is a MacOS user. In the repository there are two files sharing the same name: one lower case, the other upper case. The volunteer gave up when he started receiving errors from git.

I am aware of MacOS/git settings regarding file system case sensitivity but I don't think it is fair to ask someone to change his system to contribute. Nowadays I prefer to sacrifice readability a little to avoid facing such boring and counter-productive problems.

Cheers, Paulo A. Silva

PauloASilva commented 5 years ago

Hi Dirk, In fact, there's a GitBook bug performing file name validation :disappointed: I will fix it and do the PR.

Meanwhile, please check the resulting PDF (and other formats): no need for file name renaming and the font family issue was also addressed.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

PDF checked and looks great so far! Thanks!

More later...

drwetter commented 5 years ago

@PauloASilva : How about form feeds?

Is there anything what I can/need to do? On some pages the headline looks more like a footnote, see . e.g. D01, p7:

How can I find out?

Configuration

Then comes the form feed.

PauloASilva commented 5 years ago

You're asking for control over the page breaks, right?

I did a quick search and according gitbook PR#183 it should be possible, but I don't see these options in the current master branch.

I need to further investigate.

drwetter commented 5 years ago

yes, form feed is maybe more the technical term / nerd speak for a character which does a page break.

PauloASilva commented 5 years ago

Hi @drwetter, GitBook does not offer a way to manually control form feeds. The PDF options mentioned before are just to control form feeds style: whether it should be a page break, a rule or both.

Of course we can work around it using empty paragraphs but then we're defeating the premise of focusing solely on the content, leaving the styling thing to the tool.

Do you think this is a major issue? Footnotes have their own style: please check D03 p.13.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Hi @PauloASilva ,

unfortunately it would be an issue.

You're sure it hasn't been integrated? The PR you were referring to was 2015. If I look at the code, I cannot find the files of that PR anymore but if I search the master (https://github.com/GitbookIO/gitbook/search?q=pageBreaksBefore&unscoped_q=pageBreaksBefore) I see those changes.

Maybe somebody just forgot to document that change?

Cheers, Dirk

PauloASilva commented 5 years ago

Hi @drwetter, Maybe my explanation was not clear. In fact those options exist on master branch but they do a different thing: instead of creating a page break when an h1 is found, we can change it to --- (rule).

When I first checked this I was not able to see how this could be used but now, while writing, maybe it could be an option: we can start each chapter with --- causing the page break as well for "special cases" (h2, h3, ...) like the one you spotted.

Let me give it another try.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

If I read that correctly it seems "pageBreaksBefore": "<someusualchar>" would do it.

PauloASilva commented 5 years ago

Yap. I hope so. Let me first build a test version and then we'll discuss the result.

A segunda, 6/05/2019, 14:13, Dirk Wetter notifications@github.com escreveu:

If I read that correctly it seems `"pageBreaksBefore": "" would do it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OWASP/Docker-Security/issues/5#issuecomment-489615364, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTJFI6K26U4R6KY7UHESTPUAVHRANCNFSM4G7SLAWA .

PauloASilva commented 5 years ago

Hi @drwetter After reading Calibre documentation and play with PDF options pageBreaksBefore does what we're looking for, with some limitations:

=== pagebreak === (//*[contains(text(), '=== pagebreak ===')]) A page break is added before the marker, but the marker itself is also printed. Not an option.
<pagebreak /> (//*[name()='pagebreak']) Does exactly what we want: we can now force the page break. Setting chapterMark to none won't prevent page breaks to be added before detected chapter (doesn't look like an issue)

Pleas check the latest PDF.

If we agree on moving forward with GitBook

I should fork and fix the regular expression issue
Build the book on a Docker container (I am already doing it locally but image needs to be shrunken so that it can be disposable)

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Looks good to me, thanks!

Let's proceed.

Cheers, Dirk

drwetter commented 5 years ago

Hi @PauloASilva ,

could you do a rebase pls? We didn't discuss pictures before, didn't we? Just merged the first picture.

Second request: Is manual work still required? I just looked @ your fork and it seems the PDF is not automatically build yet.

Cheers, Dirk

PauloASilva commented 5 years ago

Hi Dirk, Sorry, I have been a little busy.

Tomorrow I'll be able to commit and push everything to my fork and open the Pull Request.

Cheers,

A quinta, 20/06/2019, 17:56, Dirk Wetter notifications@github.com escreveu:

Hi @PauloASilva https://github.com/PauloASilva ,

could you do a rebase pls? We didn't discuss pictures before, didn't we? Just merged the first picture.

Second request: Is still manual work required? I just looked @ your fork and it seems the PDF is not automatically build yet.

Cheers, Dirk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OWASP/Docker-Security/issues/5?email_source=notifications&email_token=AAWTJFLFT4YZ6BNX5JFUVN3P3OZCJA5CNFSM4G7SLAWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYGAACQ#issuecomment-504102922, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTJFKJRHVOVJM35NH3Q2LP3OZCJANCNFSM4G7SLAWA .

drwetter commented 5 years ago

Hi Paolo,

no reason to be sorry -- almost everybody has to do payed work. ;) -- including myself.

First step for me would be just to have a look in your fork how it works with the picture and whether I need to do anything in the main repo. An automatic build of the document files would only makes sense if they look good, So maybe we can use your fork to get this ready first?

Cheers, Dirk

PauloASilva commented 5 years ago

Sure. You're not expected to have to do any changes to your repo.

We will test everything on my fork. If we agree moving forward, then I'll open the PR.

Cheers,

A quinta, 20/06/2019, 19:30, Dirk Wetter notifications@github.com escreveu:

Hi Paolo,

no reason to be sorry -- almost everybody has to do payed work. ;) -- including myself.

First step for me would be just to have a look in your fork how it works with the picture and whether I need to do anything in the main repo. An automatic build of the document files would only makes sense if they look good, So maybe we can use your fork to get this ready first?

Cheers, Dirk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OWASP/Docker-Security/issues/5?email_source=notifications&email_token=AAWTJFKGOQEEBUJZG645POTP3PEDDA5CNFSM4G7SLAWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYGHSAI#issuecomment-504133889, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTJFM2SIQ4JMQSDXLBUZDP3PEDDANCNFSM4G7SLAWA .

PauloASilva commented 5 years ago

Hi @drwetter, Please check my feature-build branch. To test the build system and the resulting PDF, do:

$ git clone -b feature-build https://github.com/PauloASilva/Docker-Security.git
$ docker-compose run --rm builder

Please note that:

book source files were moved into src sub-directory
if you want to add additional page breaks, you can insert <pagebreak /> in the markdown files (e.g. D01 - Secure User Mapping)
new "How to Build PDF version" section was added in the root README.md file
when building for the first time a docker image will be create (build container is ephemeral)

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Hi Paulo,

thanks! Doesn't work somehow because of a version conflict.

On 6/21/19 3:38 PM, PauloASilva wrote:

Hi @drwetter https://github.com/drwetter, Please check my |feature-build| branch https://github.com/PauloASilva/Docker-Security/tree/feature-build. To test the build system and the resulting PDF, do:

|$ git clone -b feature-build https://github.com/PauloASilva/Docker-Security.git $ docker-compose run --rm builder

"ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version." (Debian buster and others)

Haven't bothered yet to find out what's wrong. Still busy and it'll take a few days.

Cheers, Dirk

PauloASilva commented 5 years ago

The compose file is using version 3.7. I think this is what's causing the issue.

What docker version are you using?

I can downgrade de docker-compose version since I am not using any special feature of it.

Cheers, Paulo A. Silva

A domingo, 23/06/2019, 21:37, Dirk Wetter notifications@github.com escreveu:

Hi Paulo,

thanks! Doesn't work somehow because of a version conflict.

On 6/21/19 3:38 PM, PauloASilva wrote:

Hi @drwetter https://github.com/drwetter, Please check my |feature-build| branch https://github.com/PauloASilva/Docker-Security/tree/feature-build. To test the build system and the resulting PDF, do:

|$ git clone -b feature-build https://github.com/PauloASilva/Docker-Security.git $ docker-compose run --rm builder

"ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version." (Debian buster and others)

Haven't bothered yet to find out what's wrong. Still busy and it'll take a few days.

Cheers, Dirk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OWASP/Docker-Security/issues/5?email_source=notifications&email_token=AAWTJFMMTQD7DOC6X6T34CTP37NHDA5CNFSM4G7SLAWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYLGQDQ#issuecomment-504784910, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTJFJDR2WGBTSP4FCGKNTP37NHDANCNFSM4G7SLAWA .

drwetter commented 5 years ago

I downgraded the version in the yaml file yesterday.docker-compose run --rm build did it then. That's quite a container :-)

However it didn't go though for me, see below.

Screenshot_20190624_093629

PauloASilva commented 5 years ago

Hi @drwetter, The container is huge because of package calibre and its dependencies.

Please give it another try: the /build directory is now created during the image build with proper ownership and filesystem permissions.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Hi @PauloASilva ,

thanks!

Still can't detect the picture.

book source files were moved into src sub-directory

Not sure I like that. Can't that be done just by the container, to copy?

Cheers, Dirk

PauloASilva commented 5 years ago

Hi @drwetter

Hi @PauloASilva ,

thanks!

Still can't detect the picture.

Maybe the picture path is wrong. I'll check it.

book source files were moved into src sub-directory

Not sure I like that. Can't that be done just by the container, to copy?

If you don't mind to mix building stuff (e.g. Dockerfile, docker-compose.yml) with the book source, we can put everything together, but then remind that sorting alphabetically (like GitHub WebUI does) will mix things together.

Please check the OWASP/API-Security repository structure. The repository root is used for general stuff like the README.md, CONTRIBUTING.md, repo configurations (e.g. .gitignore, .gitattributes), "coding standards" (e.g. .editorconfig), etc. Then we have:

year: we plan to update de Top 10 every 2 or 3 years
lang: we plan to track translations in this same repo
- src: the source files (e.g. MD, PNG, ...)
- dist: the distributable formats (e.g. PDF, ODT, ...)

Let me know whether you want to move the source files back to the repository root.

Cheers, Paulo A. Silva

Cheers, Dirk

drwetter commented 5 years ago

Hi Paolo,

Still can't detect the picture.
Maybe the picture path is wrong. I'll check it.

Did you get anywhere? Your fork needs a rebase to include the picture. But still if I copy locally my files over to your pulled repo, the PDF doesn't show the image in "001 - Threats.md".

Cheers, Dirk

PauloASilva commented 5 years ago

Hi @drwetter, yes, I did (a minute ago :D). Please check my fork.

The file in the assets directory was moved into the src/images directory. Other images and book related assets should be place there too.

Check the resulting PDF.

Please let me known whether you want to flatten the repository directory structure.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Hi Paolo,

I see the picture now, thanks! And so fast ;-)

Please let me known whether you want to flatten the repository directory structure.

For the second edition the first structure you suggested (OWASP/API-Security) may sound ok (unless one doesn't want to use branches for older releases)

For the first edition: couldn't we just keep my structure, create a subdir for the PDF build and work with COPY commands from the main directory so that a user doesn't have to look where the right directory is where he/she has to go to?

Cheers, Dirk

PauloASilva commented 5 years ago

Hi @drwetter,

For the first edition: couldn't we just keep my structure, create a subdir for the PDF build and work with COPY commands from the main directory so that a user doesn't have to look where the right directory is where he/she has to go to?

Cheers, Dirk

Of course we can. I'll re-arrange everything today.

Cheers, Paulo A. Silva

PauloASilva commented 5 years ago

Hi @drwetter, Please, check whether my fork now meets your expectations.

Cheers, Paulo A. Silva

drwetter commented 5 years ago

Hi Paolo,

looks good, thank you! Let me check details later.

Cheers, Dirk

drwetter commented 4 years ago

Closing with a big thank you, Paolo!

OT: Are you around in Amsterdam?

PauloASilva commented 4 years ago

Closing with a big thank you, Paolo!

My please, @drwetter.

OT: Are you around in Amsterdam?

Nop, not this time but if you meet the OWASP API Security Top 10 guys, please send them my greetings: we have been working hard to make it happen.

Keep strong & safe, Paulo A. Silva

OWASP / Docker-Security

MD to PDF #5

How can I find out?

Configuration