Closed drwetter closed 4 years ago
I believe Read The Docs should be able to do this automatically (Github Markdown to PDF).
On Mar 19, 2019, at 9:31 AM, Dirk Wetter notifications@github.com wrote:
In the end a PDF should be available.
While github is great in dealing with markdown documents I suspect it's not as easy to generate a handy PDF from he markdown files.
Help or hints would be appreciated.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
Oh, yes thanks. Sphinx rings a bell too.
Hi @drwetter,
You can check how we did it for Go Secure Coding Practices: we're using gitbook-cli v2.3.0.
In the dist
folder you'll find the resulting output formats: ePub, MOBI and PDF.
GitBook might also be an option: it is free for open-source & non-profit teams.
Cheers, Paulo A. Silva
Thanks, @PauloASilva
we're using gitbook-cli v2.3.0.
Was that done automatically or did it require manual work?
Hi @drwetter, No manual work required: you just have to run
$ npm install && npm run build
We can do it locally or automate it, for instance, on Travis CI using GitHub Hooks.
Cheers, Paulo A. Silva
That sounds good!
Mind to help?
That sounds good!
Mind to help?
Not at all, I will be glad to do it :smile:
Sounds great.
How about starting with it early? I created a separate branch (gitbook) for it so that we can play with it before it looks like it's "the" document?
I'm on it already: I forked your repo and soon I will do a pull request.
I did an initial test but the build failed due to some parsing errors. As soon as I get some output, I'll share it with you so that we can decide how to proceed.
Cheers, Paulo A. Silva
Cool! -- Sent from my mobile. Excuse my brevity&typos+the phone's autocorrection
Hi @drwetter,
We have a first draft: please check it on my fork.
The resulting formats (PDF, ePub, MOBI) are in the dist
root folder.
If we decide to move forward with gitbook, we should:
Makefile
to build the book inside an ephemeral docker container. Now I did it by hand
[pauloasilva@XXX Docker-Security]$ docker run -it --rm -v $(pwd):/app node:current-slim bash -c "cd /app && apt-get update && apt-get install -y calibre && npm i && npm run build"
000 - Introduction
chapter (gitbook expects README.md
to be the Introduction chapter)D04 - Secure Defaults and Hardening.md
Markdown as there are some parsing problem regarding dots on a command line sample marked with FIXME
.Please share your thoughts on this.
Cheers, Paulo A. Silva
Am 21. April 2019 13:23:01 MESZ schrieb PauloASilva notifications@github.com:
Hi @drwetter, We have a first draft: please check it on my fork. The resulting formats ([PDF][2], [ePub][3], [MOBI][4]) are in the
dist
root folder.If we decide to move forward with gitbook, we should:
- Create a
Makefile
to build the book inside an ephemeral docker container. Now I did it by hand[pauloasilva@XXX Docker-Security]$ docker run -it --rm -v $(pwd):/app node:current-slim bash -c "cd /app && apt-get update && apt-get install -y calibre && npm i && npm run build"
- Rename the
000 - Introduction
chapter (gitbook expectsREADME.md
to be the Introduction chapter)- Validate
D04 - Secure Defaults and Hardening.md
Markdown as there are some parsing problem regarding dots on a command line sample marked withFIXME
.Please share your thoughts on this.
Cheers, Paulo A. Silva
[2]: https://github.com/PauloASilva/Docker-Security/blob/feature-build/dist/owasp-docker-security.pdf?raw=true [3]: https://github.com/PauloASilva/Docker-Security/blob/feature-build/dist/owasp-docker-security.epub?raw=true [4]: https://github.com/PauloASilva/Docker-Security/blob/feature-build/dist/owasp-docker-security.mobi?raw=true
Hi Paulo,
thanks & Happy Easter!
Will look into it next week. Cheers. Dirk
-- Sent from my mobile. Excuse my brevity&typos+the phone's autocorrection
Hi @drwetter, Happy Easter to you too!
Cheers, Paulo A. Silva
Hi Paulo,
thanks!
- Rename the
000 - Introduction
chapter (gitbook expectsREADME.md
to be the Introduction chapter)
can't this be configured?
Speaking of it: the font (calibre) looks in my eyes rather painful. A sans-serif font would do much better to the eye. :-)
Validate
D04 - Secure Defaults and Hardening.md
Markdown as there are some parsing problem regarding dots on a command line sample marked withFIXME
I was working on that last week and correct that today or later. If it holds you up and you don't want to hotfix it in your fork first, please let me know.
Upon opening the PDF I saw the graphics and said "wow" :-)
Cheers, Dirk
D04 should be fine for you now.
Hi Dirk,
gitbook-cli supports changing the default introduction README.md
, but the file name should match the following RegEx ^[a-zA-Z-._\d,\s]+$
what 000 - Introduction.md
doesn't.
To be honest I don't like the current naming convention because:
Would you agree renaming files as follows?
000 - Introduction.md
-> 000-introduction.md
D03 - Network Separation and Firewalling.md
-> d03-network-separation-firewalling.md
E11 - What'\''s Next?.md
-> e11-whats-next.md
fontFamily
to sans-serif
(gitbook-cli PDF options)I'll pull your changes and see if they solve the markdown parsing issue.
Best, Paulo A. Silva
Hi Paulo,
the Easter Egg arrived on time ;-)
I can work on Readme and the Introduction to rename them.
As far as the naming of the files in general are concerned: I guess it's a matter of taste. At least ^[a-zA-Z-._\d,\s]+$
matches 000 - Introduction.md
, see e.g. https://regex101.com/. I am assuming this pattern applies for all filenames?
I am a friend of readable filenames. Why should a human sacrifice readability to a machine? Just because some sloppy shell programmers forgot to quote properly?
Don't know what OS you're using? Even on Linux (which I use) I only have little problems with such filenames. On MacOS and Windows they are common since a longer while back.
In E11 the question mark and quote was too much maybe. I pushed a change.
Cheers! Dirk
Hi Dirk,
Please don't change anything on your side because I am not sure what's wrong: although gitbook complains about the file name I am not being able to use a value other than README.md
.
I'll let you know when I have figured out what's wrong and how to proceed with the PoC.
I am a Linux user too. After a few years of software development, working with other developers with different OS preferences, I had to adopt a standard for file naming to avoid undesired problems.
The most recent incident happened during the OWASP Top 10 translation. I got a volunteer who is a MacOS user. In the repository there are two files sharing the same name: one lower case, the other upper case. The volunteer gave up when he started receiving errors from git.
I am aware of MacOS/git settings regarding file system case sensitivity but I don't think it is fair to ask someone to change his system to contribute. Nowadays I prefer to sacrifice readability a little to avoid facing such boring and counter-productive problems.
Cheers, Paulo A. Silva
Hi Dirk, In fact, there's a GitBook bug performing file name validation :disappointed: I will fix it and do the PR.
Meanwhile, please check the resulting PDF (and other formats): no need for file name renaming and the font family issue was also addressed.
Cheers, Paulo A. Silva
PDF checked and looks great so far! Thanks!
More later...
@PauloASilva : How about form feeds?
Is there anything what I can/need to do? On some pages the headline looks more like a footnote, see . e.g. D01, p7:
Then comes the form feed.
You're asking for control over the page breaks, right?
I did a quick search and according gitbook PR#183 it should be possible, but I don't see these options in the current master
branch.
I need to further investigate.
yes, form feed is maybe more the technical term / nerd speak for a character which does a page break.
Hi @drwetter, GitBook does not offer a way to manually control form feeds. The PDF options mentioned before are just to control form feeds style: whether it should be a page break, a rule or both.
Of course we can work around it using empty paragraphs but then we're defeating the premise of focusing solely on the content, leaving the styling thing to the tool.
Do you think this is a major issue? Footnotes have their own style: please check D03 p.13.
Cheers, Paulo A. Silva
Hi @PauloASilva ,
unfortunately it would be an issue.
You're sure it hasn't been integrated? The PR you were referring to was 2015. If I look at the code, I cannot find the files of that PR anymore but if I search the master (https://github.com/GitbookIO/gitbook/search?q=pageBreaksBefore&unscoped_q=pageBreaksBefore) I see those changes.
Maybe somebody just forgot to document that change?
Cheers, Dirk
Hi @drwetter,
Maybe my explanation was not clear. In fact those options exist on master branch but they do a different thing: instead of creating a page break when an h1
is found, we can change it to ---
(rule).
When I first checked this I was not able to see how this could be used but now, while writing, maybe it could be an option: we can start each chapter with ---
causing the page break as well for "special cases" (h2
, h3
, ...) like the one you spotted.
Let me give it another try.
Cheers, Paulo A. Silva
If I read that correctly it seems "pageBreaksBefore": "<someusualchar>"
would do it.
Yap. I hope so. Let me first build a test version and then we'll discuss the result.
A segunda, 6/05/2019, 14:13, Dirk Wetter notifications@github.com escreveu:
If I read that correctly it seems
`"pageBreaksBefore": ""
would do it.— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OWASP/Docker-Security/issues/5#issuecomment-489615364, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTJFI6K26U4R6KY7UHESTPUAVHRANCNFSM4G7SLAWA .
Hi @drwetter
After reading Calibre documentation and play with PDF options
pageBreaksBefore
does what we're looking for, with some limitations:
=== pagebreak ===
(//*[contains(text(), '=== pagebreak ===')]
)
A page break is added before the marker, but the marker itself is also
printed. Not an option.<pagebreak />
(//*[name()='pagebreak']
)
Does exactly what we want: we can now force the page break. Setting
chapterMark
to none
won't prevent page breaks to be added before detected
chapter (doesn't look like an issue)Pleas check the latest PDF.
If we agree on moving forward with GitBook
Cheers, Paulo A. Silva
Looks good to me, thanks!
Let's proceed.
Cheers, Dirk
Hi @PauloASilva ,
could you do a rebase pls? We didn't discuss pictures before, didn't we? Just merged the first picture.
Second request: Is manual work still required? I just looked @ your fork and it seems the PDF is not automatically build yet.
Cheers, Dirk
Hi Dirk, Sorry, I have been a little busy.
Tomorrow I'll be able to commit and push everything to my fork and open the Pull Request.
Cheers,
A quinta, 20/06/2019, 17:56, Dirk Wetter notifications@github.com escreveu:
Hi @PauloASilva https://github.com/PauloASilva ,
could you do a rebase pls? We didn't discuss pictures before, didn't we? Just merged the first picture.
Second request: Is still manual work required? I just looked @ your fork and it seems the PDF is not automatically build yet.
Cheers, Dirk
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OWASP/Docker-Security/issues/5?email_source=notifications&email_token=AAWTJFLFT4YZ6BNX5JFUVN3P3OZCJA5CNFSM4G7SLAWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYGAACQ#issuecomment-504102922, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTJFKJRHVOVJM35NH3Q2LP3OZCJANCNFSM4G7SLAWA .
Hi Paolo,
no reason to be sorry -- almost everybody has to do payed work. ;) -- including myself.
First step for me would be just to have a look in your fork how it works with the picture and whether I need to do anything in the main repo. An automatic build of the document files would only makes sense if they look good, So maybe we can use your fork to get this ready first?
Cheers, Dirk
Sure. You're not expected to have to do any changes to your repo.
We will test everything on my fork. If we agree moving forward, then I'll open the PR.
Cheers,
A quinta, 20/06/2019, 19:30, Dirk Wetter notifications@github.com escreveu:
Hi Paolo,
no reason to be sorry -- almost everybody has to do payed work. ;) -- including myself.
First step for me would be just to have a look in your fork how it works with the picture and whether I need to do anything in the main repo. An automatic build of the document files would only makes sense if they look good, So maybe we can use your fork to get this ready first?
Cheers, Dirk
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OWASP/Docker-Security/issues/5?email_source=notifications&email_token=AAWTJFKGOQEEBUJZG645POTP3PEDDA5CNFSM4G7SLAWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYGHSAI#issuecomment-504133889, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTJFM2SIQ4JMQSDXLBUZDP3PEDDANCNFSM4G7SLAWA .
Hi @drwetter,
Please check my feature-build
branch.
To test the build system and the resulting PDF, do:
$ git clone -b feature-build https://github.com/PauloASilva/Docker-Security.git
$ docker-compose run --rm builder
Please note that:
src
sub-directory<pagebreak />
in the markdown files (e.g. D01 - Secure User Mapping)README.md
fileCheers, Paulo A. Silva
Hi Paulo,
thanks! Doesn't work somehow because of a version conflict.
On 6/21/19 3:38 PM, PauloASilva wrote:
Hi @drwetter https://github.com/drwetter, Please check my |feature-build| branch https://github.com/PauloASilva/Docker-Security/tree/feature-build. To test the build system and the resulting PDF, do:
|$ git clone -b feature-build https://github.com/PauloASilva/Docker-Security.git $ docker-compose run --rm builder
"ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version." (Debian buster and others)
Haven't bothered yet to find out what's wrong. Still busy and it'll take a few days.
Cheers, Dirk
The compose file is using version 3.7. I think this is what's causing the issue.
What docker version are you using?
I can downgrade de docker-compose version since I am not using any special feature of it.
Cheers, Paulo A. Silva
A domingo, 23/06/2019, 21:37, Dirk Wetter notifications@github.com escreveu:
Hi Paulo,
thanks! Doesn't work somehow because of a version conflict.
On 6/21/19 3:38 PM, PauloASilva wrote:
Hi @drwetter https://github.com/drwetter, Please check my |feature-build| branch https://github.com/PauloASilva/Docker-Security/tree/feature-build. To test the build system and the resulting PDF, do:
|$ git clone -b feature-build https://github.com/PauloASilva/Docker-Security.git $ docker-compose run --rm builder
"ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version." (Debian buster and others)
Haven't bothered yet to find out what's wrong. Still busy and it'll take a few days.
Cheers, Dirk
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OWASP/Docker-Security/issues/5?email_source=notifications&email_token=AAWTJFMMTQD7DOC6X6T34CTP37NHDA5CNFSM4G7SLAWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYLGQDQ#issuecomment-504784910, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWTJFJDR2WGBTSP4FCGKNTP37NHDANCNFSM4G7SLAWA .
I downgraded the version in the yaml file yesterday.docker-compose run --rm build
did it then. That's quite a container :-)
However it didn't go though for me, see below.
Hi @drwetter,
The container is huge because of package calibre
and its dependencies.
Please give it another try: the /build
directory is now created during the image build with proper ownership and filesystem permissions.
Cheers, Paulo A. Silva
Hi @PauloASilva ,
thanks!
Still can't detect the picture.
book source files were moved into src sub-directory
Not sure I like that. Can't that be done just by the container, to copy?
Cheers, Dirk
Hi @drwetter
Hi @PauloASilva ,
thanks!
Still can't detect the picture.
Maybe the picture path is wrong. I'll check it.
book source files were moved into src sub-directory
Not sure I like that. Can't that be done just by the container, to copy?
If you don't mind to mix building stuff (e.g. Dockerfile
, docker-compose.yml
) with the book source, we can put everything together, but then remind that sorting alphabetically (like GitHub WebUI does) will mix things together.
Please check the OWASP/API-Security repository structure.
The repository root is used for general stuff like the README.md
, CONTRIBUTING.md
, repo configurations (e.g. .gitignore
, .gitattributes
), "coding standards" (e.g. .editorconfig
), etc.
Then we have:
MD
, PNG
, ...)PDF
, ODT
, ...)Let me know whether you want to move the source files back to the repository root.
Cheers, Paulo A. Silva
Cheers, Dirk
Hi Paolo,
Still can't detect the picture.
Maybe the picture path is wrong. I'll check it.
Did you get anywhere? Your fork needs a rebase to include the picture. But still if I copy locally my files over to your pulled repo, the PDF doesn't show the image in "001 - Threats.md".
Cheers, Dirk
Hi @drwetter, yes, I did (a minute ago :D). Please check my fork.
The file in the assets
directory was moved into the src/images
directory.
Other images and book related assets should be place there too.
Check the resulting PDF.
Please let me known whether you want to flatten the repository directory structure.
Cheers, Paulo A. Silva
Hi Paolo,
I see the picture now, thanks! And so fast ;-)
Please let me known whether you want to flatten the repository directory structure.
For the second edition the first structure you suggested (OWASP/API-Security) may sound ok (unless one doesn't want to use branches for older releases)
For the first edition: couldn't we just keep my structure, create a subdir for the PDF build and work with COPY commands from the main directory so that a user doesn't have to look where the right directory is where he/she has to go to?
Cheers, Dirk
Hi @drwetter,
For the first edition: couldn't we just keep my structure, create a subdir for the PDF build and work with COPY commands from the main directory so that a user doesn't have to look where the right directory is where he/she has to go to?
Cheers, Dirk
Of course we can. I'll re-arrange everything today.
Cheers, Paulo A. Silva
Hi @drwetter, Please, check whether my fork now meets your expectations.
Cheers, Paulo A. Silva
Hi Paolo,
looks good, thank you! Let me check details later.
Cheers, Dirk
Closing with a big thank you, Paolo!
OT: Are you around in Amsterdam?
Closing with a big thank you, Paolo!
My please, @drwetter.
OT: Are you around in Amsterdam?
Nop, not this time but if you meet the OWASP API Security Top 10 guys, please send them my greetings: we have been working hard to make it happen.
Keep strong & safe, Paulo A. Silva
In the end a PDF should be available.
While github is great in dealing with markdown documents I suspect it's not as easy to generate a handy PDF from the markdown files.
Help or hints would be appreciated.