Restore wikilinks and images lost during conversion from mediawiki

kingthorin commented 5 years ago

I've started to work on this. In some cases it's non-trivial.

Along with that we should probably:

~Establish some style guidelines.~ > Edit: Started to do this in README, https://github.com/OWASP/OWASP-Testing-Guide-v5#stylecontribution-guidelines
~Figure out a repo structure. (I'm going to try a few things and see if I can come up with a suggestion/direction.) [ex: Where do we want images to live? Or, where do they need to live for github to show them properly?]~ > Edit: I've decided to place an images directory at the top of each chapter, images can then be referenced relatively.

kingthorin commented 5 years ago

Items to fix:

"wikilinks" during migration the OWASP Mediawiki links were tagged "wikilink" these needs to be converted to proper absolute URLs.
- Unless the wikilink is to content within OTG, in which case it should be the proper new github relative URL (URL Encoded).
Any / terminating lines need to be removed and appropriate fixed up for markdown (it's likely a spot that requires a double line-feed prior to an image or other element).
In some cases (due to the source of the text) there are broken apostrophes ' they should just be restored.
Occurrences of \$ should become &dollar;
At symbols @ should become &commat;
Images need to be saved to <chapter (or sub-section)>/images/<image.ext> and linked relatively like images/image.ext. (Images in a folder along side the relevant content. [Even if that means duplicates in the overall project.])
Links should be in-line instead of footnote/references.
- If Foot notes/references MUST be used for some weird reason: use numeric reference type tags, ex: Content ref: [[1]] Footer ref: [1]: http://example.org (Also note with this style the footnote doesn't actually display in the web rendered markdown, so there's no need for a "References" heading that would endup blank.)
Headers should be properly formatted with leading #, per: https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#headers

_{Note: If you need examples of the above items, the latest versions of chapters 1, 2, and soon 3 should have it.}

kingthorin commented 5 years ago

Ref: https://github.com/OWASP/CheatSheetSeries/#conversion-rules

kingthorin commented 5 years ago

I've been trying to come up with a non-HTML way to handle foot notes. Anyone got any ideas?

Initially I had hoped to use something like this for the superscript part https://gist.github.com/molomby/9bc092e4a125f529ae362de7e46e8176 but that didn't pan out.

I then learned that some markdown supports ^ for superscript, but that doesn't seem applicable to GFM.

(Note for both of the above there's a small chance I did something wrong, I'll admit I was rushing.)

My current suggestion would be to use ¹ type entity codes for the super script and then level 6 headings at the end of the document to provide the in page anchors. So for example, in content: [¹](#foot1) at the end of the doc:

###### foot1

Some footnote goes here

(Obviously "foot1" could be something more descriptive.) Here's a working example: https://gist.github.com/kingthorin/b723c4549aa36bad8dbdda185fed1ae0

The downside to that is when we get into two digit footer identifiers, you have to use two entities. Not the end of the world but it starts to get messy.

I guess we could try to move away from footnotes to some sort of in-line note.

I'm open to thoughts/suggestions.

To see the type of content this would apply to check the current section 2: https://github.com/OWASP/OWASP-Testing-Guide-v5/blob/master/document/2%20Introduction/2%20Introduction.md

kingthorin commented 5 years ago

Re: footnotes, the other option is to just have a end/bottom of page section for references and link the section heading not the individual notes. (Could use ordered list(s), in order to maintain numeric associations.)

ThunderSon commented 5 years ago

Why didn't this work: https://gist.github.com/molomby/9bc092e4a125f529ae362de7e46e8176 There is no easy solution. I would only use those in mathematical functions if need be. References should be done as such: Some text [2] where is 2 is the number reference. No need to use superscript for this. In whitepapers it's done as such as well. Footnotes can be switched with anything really, it's just a section title in here.

kingthorin commented 5 years ago

Why didn't this work: https://gist.github.com/molomby/9bc092e4a125f529ae362de7e46e8176

They wouldn't display for me (could be a Firefox issue, but if it's an issue in one major browser then it essentially doesn't work IMHO).


You don't think this looks a little [weird][2]?
Or like this [[3]]?

[2]:http://example.com
[3]:http://example.org

Without the footnote or ref number actually being visible (2):

You don't think this looks a little weird? Or like this [3]?

I guess 3 isn't so bad???? Though they still aren't displayed (on github anyway), I guess if we republished elsewhere that would be fine (or manageable).

ThunderSon commented 5 years ago

I am talking about the 3 example, guess you already got it 😄 Yeah that's another discussion about re-publishing, even though I believe that convention is widely used in whitepapers and not in books.

kingthorin commented 5 years ago

Ok, I'm good with us settling on that. I'll tweak my post above and do a PR for sections 1, 2, 3, and the start of 4 later this weekend.

Thanks @ThunderSon

patrickceg commented 5 years ago

Hello all, I'd like to try some of the cleanup of the links / images in section 4 in general. Is there any work currently being done for link cleanup in section 4? (just so I don't conflict with an equivalent effort) Thanks!

kingthorin commented 5 years ago

I don’t have anything on the go aside from existing PRs, and no further short terms plans.

patrickceg commented 5 years ago

Thanks for the note. I've started clicking through with the sole purpose of looking for links and images broken due to conversion specifically. Section 4.1 looks OK, and I'm starting to list out the issues in section 4.2.

I'll get something together for the minor fixes over the next 3 days. (Again, specifically with the scope of section 4 links broken from conversion. We fling the words "out of scope" so often at my work that it's baked in to how I work now...)

kingthorin commented 5 years ago

@patrickceg Some 4.10 tweaks are queued up here: https://github.com/OWASP/OWASP-Testing-Guide-v5/pull/57 some 4.2 footnote changes are in-flight here: https://github.com/OWASP/OWASP-Testing-Guide-v5/pull/59/ Older 4.1 and 4.2 tweaks: https://github.com/OWASP/OWASP-Testing-Guide-v5/pull/41 Otherwise there's plenty of subsections of chapter 4 that need attention.

I had originally suggested images be at the top of the chapter, but now I think it's best to have an images directory at the same level as the content. (Thoughts? Ex: document/4 Web Application Security Testing/4.10 Testing for weak Cryptography/images/SSL_Certificate_Validity_Testing_Firefox_Warning.gif vs document/4 Web Application Security Testing/images/SSL_Certificate_Validity_Testing_Firefox_Warning.gif)

patrickceg commented 5 years ago

@kingthorin On lumping in a folder structure change: I'm not sure if it happens much in group document projects, but with a codebase, if you lump a whole lot of different modifications together, you get giant pull requests no one wants to look at. Even if they do look, they're looking for so many things at once that the quality of the review goes down. This issue #10 only talks about making the links and images work again, and I'm not comfortable going down the rabbit hole of flipping the conventions at the same time, without at least redefining this issue.

By redefining the issue, I mean close this issue #10 "won't fix" (because the goal is no longer duct taping the links to make them work again), and make a new issue "Implement new file and folder structure".

I haven't skimmed enough of the guide as it is to decide. Here's some considerations:

We want to remain consistent, so those kinds of instructions (where to put images, etc) should go in the contribution guide.
Will we ever share images between sections? If so, putting them in a subsection would be bad, because you're stuck with the tough decision between moving them or duplicating them. I guess there's also symlinks, but if I recall most web server stuff rejects symbolic links because they're a path traversal vulnerability waiting to happen.
We'll want to keep it easy to look for dangling or broken images. I was thinking of building a continuous integration script like a build to crawl and look for broken links and unused resources of any kind. (i.e. it flags any link that returns 404 and any file that isn't touched by the crawler as warnings). That said, I'll have to do quite a bit of reading still to figure that out. I could probably throw something together with a Python scraping framework, but Markdown static web pages are so popular these days, I'll look to see if someone else solved the problem already.

EDIT: Until that decision is made, I'll therefore make the repairs to section 4 except for sections 4.10, 4.1, 4.2, and whatever has a different open pull request with comments on the time I start. (If it's a pull request that looks like it's about to merge and doesn't have major ccomments, I'll resolve the merge conflict on my own branch.)

kingthorin commented 5 years ago

@patrickceg all good points.

We want to remain consistent, so those kinds of instructions (where to put images, etc) should go in the contribution guide.

Exactly, hence I was trying to have us come up with the choice. As things currently sit there are no images aside from the few I'd already fixed. Luckily those done so far were in early chapters where there aren't subsections so it hasn't been relevant. The PR for 4.10 establishes a subsection images directory so I guess in the absence of a decision I'll make it for the project until there's some debate or descenting opinion. (Images in a folder along side the relevant content. [Even if that means duplicates in the overall project.])

ThunderSon commented 5 years ago

What do you think about breaking down this issue. Let's make this a milestone/project, and start assigning issues to it. This is just way too unorganized, and as such, the above discussion erupted. I believe you agreed that no file structure changes should be done, and I agree to that.

@kingthorin I believe we need to prioritize our points. Let's move this to slack 😄

kingthorin commented 5 years ago

It’s not a change it’s a decision about how it’s going to be, because right now the images don’t exist or have a home at all.

ThunderSon commented 5 years ago

Currently there is an images folder in the top layer folder of each chapter. There is a place for the images to exist ... I agree though it's mainly about taking the decision. I already sent my point of view on the channel.

kingthorin commented 5 years ago

Grin. They only exist where I’ve created them 😀 /me wanders off to slack

patrickceg commented 5 years ago

I finished my analysis looking at all the stuff looking for conversion issues (brutally copied from the word processor I was working with)...

Issues • Table of contents page completely broken • 5 of the of the subsection summary pages have broken links • Links to references have no formatting standard: ○ 2 - has a academic style reference URLs [1], [2] ○ Section 4 bounces between links on their own, text + bare URL, link + text, referneces with no URL, etc • 0 Foreward - "Testing Guide" link - Probably doesn't have to point to itself • 1.2 "by-laws" - no need for it to link to Wikipedia (since the OWASP foundation by-laws are a link right underneath) • 1.2 "code of ethics" - looks a bit intimidating to have member revocation as the only link? • 1.2 "Submit an Inquiry" link points to a site that has no HTTPS • 1.2 "searching" link to OWASP Google custom search is broken • 2 - Link [9] does not have HTTPS, Link [13] http://cm.bell-labs.com/who/ken/trust.html doesn't resolve • 1, 2 - Lots of floating bold: should decide on headers • 3 - missing sections? • 4.3.1 - Has a different reference setup with no URLs

Recommendations

• Lots of sections link to each other, so we should do the file naming first
• For any fix-ups, keep an eye out for:
    ○ Stray backslashes \  - they're everywhere on converted pages
    ○ Broken code blocks and lists
    ○ Weird formatting for headers (completely broken headers, bold or other items that are not headers)
• Create convention for file and folders:
    ○ [Section]-[Subject]-([OWASP reference code]).md?
    ○ By reference code, I mean like (OTG-AUTHN-003)
    ○ If there's multiple reference codes, it's probably better to use the most fitting one and place additional codes in the article text
    ○ Spaces or underscores? (Markdown seems functional enough which spaces for links, so it's not useful to pretend that it's MS DOS and using dashes or underscores instead of spaces unless there is another constraint)
• Define what to do with links, and where to put them (in-line? references section? both)
• What to do for print references with no web sites
• We shouldn't have "wishy-washy" wording lik this in 4.3.1.
    ○ "There are some GUI-based administration tools for Apache (like NetLoony) but they are not in widespread use yet."
    ○ Instead, we should can have an "Emerging techniques" subsection for items that are not very well tested
• Keep the "Tools" and "References" section with well-defined boundaries
    ○ "Tools" for something you download / buy and use (whether it's a test tool or a vulnerable service like Juice Shop to try your test against)
    ○ "References" for further READING (standards, other Wikis, other guides / books / blogs)
• 4.5.10 - missing references?
• Section https://github.com/OWASP/OWASP-Testing-Guide-v5/blob/master/document/4%20Web%20Application%20Security%20Testing/4.7%20Session%20Management%20Testing/4.7.1%20Testing%20for%20Session%20Management%20Schema%20(OTG-SESS-001).md#related-security-activities has weird "related security activities" section that looks like it can be rolled into defences, remediation, or references

Template suggestion: • File name • Sections: a. Summary b. List of OWASP codes (e.g. OTG-AUTHN-003) c. Text body d. Defences e. Remediation f. Tools (URL links ot tools) g. References

ThunderSon commented 5 years ago

@patrickceg Thank you for this!

File names should be _ separated. Having spaces will force the author to use URL encoding.
I like the template suggestion, yet it's going to be a bit hard to implement. There is already a template from v3. I can't nor @kingthorin amend it AFAIK.
Links and references: All linked URIs should be inline [text](URI ref). Additional readings in the reference, as you mentioned.
Links should be modified to call local files when possible (another testing guide that exists in the repository)
I am against having guides with not ready to use or safe to use tools. A note/issue could be opened to keep track of said tools by the author in order to later on implement them when they are ready. If the whole testing guide being written is an edge case, it'll be handled then based on how beneficial it is, and what is their supporting it.

What's left is how to break what you mentioned into issues:

Setup a way for contribution. We can check out CS Series and build from it what we need.
Create milestones, create issues and link them.
I'd recommend adding a linter that'd help us identify all the markdown issues. The following tool can assist us.
Handle markdown issues by section.
Ensure that all links resolve. The following tool can be used.

@kingthorin and @patrickceg I'll be awaiting your feedback. We can go deeper into this on Slack as it'd be better to cook this there.

rejahrehim commented 4 years ago

Still anything pending anything in this issue?

kingthorin commented 4 years ago

It's getting close (96%). (Two tickets [including this one], one PR [which is underway and addresses the second ticket].)

ThunderSon commented 4 years ago

This issue was a milestones of its own. This issue is the biggest massacre in GitHub's history.

rejahrehim commented 4 years ago

Can we close this?

OWASP / wstg

Restore wikilinks and images lost during conversion from mediawiki #10