Programmatically found broken internal links

ff6347 commented 2 months ago

Most appropriate sections of the p5.js website?

Other (specify if possible)

What is your operating system?

Mac OS

Web browser and version

not related to browsers

Actual Behavior

Broken links generate a 404

Expected Behavior

Links should serve the linked content

Steps to reproduce

Since I saw some reports of broken links in the website I tried a programmatic approach using broken-link-checker.

How to:

Edit: To make it easier to generate an overview I created a script that outputs GitHub flavored markdown. https://github.com/ff6347/find-broken-links

npm install broken-link-checker
npx blc -roe https://p5js.org > report-blc-prod-live.txt

# The flags do the following
# --exclude-external, -e  Will not check external links.
# --ordered, -o           Maintain the order of links as they appear in their HTML document.
# --recursive, -r         Recursively scan ("crawl") the HTML document(s).

The output needs some cleaning using regular expressions afterwards.

Remove all reports that are okay:

^├───OK───.*?$\n

Then remove all reports of urls that have no broken links:

^Getting links from:.*?$\nFinished! \d{1,1000} links found. \d{1,1000} excluded. 0 broken.

Fix some reports where there is no new line between the report and the next.

look for:

broken\.\n(\w)

Replace with:

broken.\n\n$1

These three should do most of the grunt work. There are some false positive reports about images that I removed manually at the end.

Here is what I've found:

Low Hanging Fruits

There are some links in footer and reference that create 301 redirects which can be avoided through small tweaks.

These are the lines that need a trailing slash:

Generally these are neglect-able, but a HEAD request against them returns a 301 HTTP response. This could be fixed since it is a low hanging fruit.

301 redirects

There are many links in the reference generated form the JSDoc comments in the source of p5.js that are lacking the trailing slash and generate a 301 HTTP redirect. These need some manual labor in the source code. This is a bit tedious work. It should not be a problem keeping this as is since the site-host + Astro take care of the redirects.

Broken Links

There are some broken links also generated from the JSDoc that need fixing and I found broken links in the tutorials. They are hard-coded into the .mdx files and need to be fixed one by one.

Next Steps

IMO the next steps should be

creating separate issues for these since fixing it just based on this list is error prone.
check external links as well when the internal ones are fixed.

Below you can find the report

report created: Fri, 13 Sep 2024 11:06:07 GMT

[ ] https://p5js.org/contribute/friendly_error_system/
- [ ] _friendlyFileLoadError() | https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#_friendlyerror
- [ ] _validateParameters() | https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#_validateparameters
- [ ] Dev Notes | https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#-development-notes
[ ] https://p5js.org/contribute/release_process/
- [ ] “New p5.js release” | https://p5js.org/contribute/.github/workflows/release.yml
[ ] https://p5js.org/contribute/steward_guidelines/
- [ ] p5.js’ design principles | https://p5js.org/contribute/contributor_guidelines.md#software-design-principles
- [ ] design principles | https://p5js.org/contribute/contributor_guidelines.md#software-design-principles
- [ ] contributor’s guidelines | https://p5js.org/contribute/contributor_guidelines.md#working-on-p5js-codebase
- [ ] ../tasks/build/eslint-samples.js | https://p5js.org/contribute/tasks/build/eslint-samples.js
- [ ] ../tasks/build/browserify.js | https://p5js.org/contribute/tasks/build/browserify.js
- [ ] ../tasks/test/mocha-chrome.js | https://p5js.org/contribute/tasks/test/mocha-chrome.js
[ ] https://p5js.org/contribute/webgl_mode_architecture/
- [ ] null | https://p5js.org/contribute/webgl_mode_architecture/images/line-diagram.svg
- [ ] null | https://p5js.org/contribute/webgl_mode_architecture/images/joins.svg
- [ ] null | https://p5js.org/contribute/webgl_mode_architecture/images/caps.svg
- [ ] null | https://p5js.org/contribute/webgl_mode_architecture/images/flags.svg
[ ] https://p5js.org/examples/imported-media-video-canvas/
- [ ] Video | https://p5js.org/examples/dom-video.html
[ ] https://p5js.org/examples/imported-media-video-capture/
- [ ] Video | https://p5js.org/examples/dom-video.html
- [ ] Video Canvas | https://p5js.org/examples/dom-video-canvas.html
[ ] https://p5js.org/examples/imported-media-video/
- [ ] Video Canvas | https://p5js.org/examples/dom-video-canvas.html
[ ] https://p5js.org/people/
- [ ] Ryan Slade | https://p5js.org/people/rdslade.github.io
[ ] https://p5js.org/reference/console/log/
- [ ] console.log | https://p5js.org/reference/p5/console/log
- [ ] string | https://p5js.org/reference/p5/string
[ ] https://p5js.org/reference/JSON/stringify/
- [ ] string | https://p5js.org/reference/p5/string
[ ] https://p5js.org/reference/p5.sound/p5.Oscillator/
- [ ] p5.SinOsc | https://p5js.org/reference/p5.SinOsc
- [ ] p5.TriOsc | https://p5js.org/reference/p5.TriOsc
- [ ] p5.SqrOsc | https://p5js.org/reference/p5.SqrOsc
- [ ] p5.SawOsc | https://p5js.org/reference/p5.SawOsc
[ ] https://p5js.org/reference/p5.Table/addColumn/
- [ ] addColumn() | https://p5js.org/reference/p5/addColumn
[ ] https://p5js.org/reference/p5.Table/addRow/
- [ ] addRow() | https://p5js.org/reference/p5/addRow
[ ] https://p5js.org/reference/p5.Table/removeColumn/
- [ ] removeColumn() | https://p5js.org/reference/p5/removeColumn
[ ] https://p5js.org/reference/p5.Table/rows/
- [ ] getRows() | https://p5js.org/reference/p5/getRows
[ ] https://p5js.org/reference/p5/blue/
- [ ] colorMode() | https://p5js.org/reference/colorMode
[ ] https://p5js.org/reference/p5/brightness/
- [ ] colorMode() | https://p5js.org/reference/colorMode
[ ] https://p5js.org/reference/p5/createFilterShader/
- [ ] Introduction to Shaders | https://p5js.org/learn/getting-started-in-webgl-shaders.html
[ ] https://p5js.org/reference/p5/describe/
- [ ] Writing accessible canvas descriptions | https://p5js.org/learn/accessible-labels.html
[ ] https://p5js.org/reference/p5/describeElement/
- [ ] Writing accessible canvas descriptions | https://p5js.org/learn/accessible-labels.html
[ ] https://p5js.org/reference/p5/endShape/
- [ ] writing a custom shader | https://p5js.org/learn/getting-started-in-webgl-shaders.html
[ ] https://p5js.org/reference/p5/green/
- [ ] colorMode() | https://p5js.org/reference/colorMode
[ ] https://p5js.org/reference/p5/gridOutput/
- [ ] Writing accessible canvas descriptions | https://p5js.org/learn/accessible-labels.html
[ ] https://p5js.org/reference/p5/gt/
- [ ] > | https://p5js.org/reference/p5/%3E
[ ] https://p5js.org/reference/p5/gte/
- [ ] >= | https://p5js.org/reference/p5/%3E=
[ ] https://p5js.org/reference/p5/hue/
- [ ] colorMode() | https://p5js.org/reference/colorMode
[ ] https://p5js.org/reference/p5/image/
- [ ] p5.Texture | https://p5js.org/reference/p5/p5.Texture
- [ ] p5.FramebufferTexture | https://p5js.org/reference/p5/p5.FramebufferTexture
[ ] https://p5js.org/reference/p5/lerpColor/
- [ ] lerp | https://p5js.org/reference/lerp
- [ ] colorMode() | https://p5js.org/reference/colorMode
[ ] https://p5js.org/reference/p5/lightness/
- [ ] colorMode() | https://p5js.org/reference/colorMode
[ ] https://p5js.org/reference/p5/lt/
- [ ] << a=""> evaluates to true if the left value is less than the right value. | https://p5js.org/reference/p5/%3C
[ ] https://p5js.org/reference/p5/lte/
- [ ] <=< a=""> evaluates to true if the left value is less than or equal to the right value. | https://p5js.org/reference/p5/%3C=
[ ] https://p5js.org/reference/p5/model/
- [ ] loadGeometry() | https://p5js.org/reference/p5/loadGeometry
[ ] https://p5js.org/reference/p5/p5.Table/
- [ ] getRows() | https://p5js.org/reference/p5/getRows
- [ ] addRow() | https://p5js.org/reference/p5/addRow
- [ ] addColumn() | https://p5js.org/reference/p5/addColumn
- [ ] removeColumn() | https://p5js.org/reference/p5/removeColumn
[ ] https://p5js.org/reference/p5/red/
- [ ] colorMode() | https://p5js.org/reference/colorMode
[ ] https://p5js.org/reference/p5/saturation/
- [ ] colorMode() | https://p5js.org/reference/colorMode
[ ] https://p5js.org/reference/p5/textOutput/
- [ ] Writing accessible canvas descriptions | https://p5js.org/learn/accessible-labels.html
[ ] https://p5js.org/reference/p5/WEBGL/
- [ ] the learn page about coordinates and transformations | https://p5js.org/learn/getting-started-in-webgl-coords-and-transform.html
- [ ] the learn page about custom geometry | https://p5js.org/learn/getting-started-in-webgl-custom-geometry.html
- [ ] the learn page for styling and appearance | https://p5js.org/learn/getting-started-in-webgl-appearance.html
- [ ] the learn page section about cameras | https://p5js.org/learn/getting-started-in-webgl-appearance.html#camera
- [ ] introduction to shaders | https://p5js.org/learn/getting-started-in-webgl-shaders.html
- [ ] all the interactive WEBGL tutorials | https://p5js.org/learn/#:~:text=Getting%20Started%20in%20WebGL
[x] https://p5js.org/tutorials/color-gradients/
- [x] createButton() | https://p5js.org/reference/p5/createbutton/
- [x] Color Radial Gradient Example | https://p5js.org/examples/color-radial-gradient.html
[x] https://p5js.org/tutorials/conditionals-and-interactivity/
- [x] Variables and Motion Tutorial | https://p5js.org/tutorials/variables-and-motion/
- [x] logical operators | https://p5js.org/examples/control-logical-operators.html
[x] https://p5js.org/tutorials/custom-geometry/
- [x] Styling and Appearance | https://p5js.org/tutorials/styling-and-appearance/
[x] https://p5js.org/tutorials/custom-shapes-and-smooth-curves/
- [x] Melody App | https://p5js.org/tutorials/melody-app-with-nodejs/
[x] https://p5js.org/tutorials/data-structure-garden/
- [x] Organizing Code with Functions | https://p5js.org/tutorials-organizing-code-with-functions/
- [x] Melody App | https://p5js.org/tutorials/melody-app/
[x] https://p5js.org/tutorials/get-started/
- [x] string | https://p5js.org/reference/p5/string/
- [x] Strings | https://p5js.org/reference/p5/string/
- [x] string | https://p5js.org/reference/p5/string/
- [x] strings | https://p5js.org/reference/p5/string/
- [x] string | https://p5js.org/reference/p5/string/
[x] https://p5js.org/tutorials/getting-started-with-nodejs/
- [x] Melody App | https://p5js.org/tutorials/melody-app-with-nodejs/
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
[x] https://p5js.org/tutorials/how-to-optimize-your-sketches/
- [x] p5.Vector | https://p5js.org/reference/p5.Vector
- [x] p5.Vector | https://p5js.org/reference/p5.Vector
[x] https://p5js.org/tutorials/layered-rendering-with-framebuffers/
- [x] the createFramebuffer() documentation | https://p5js.org/reference/en/p5/createFramebuffer/
- [x] an example of Framebuffer depth blur | https://p5js.org/examples/3d-blur-using-framebuffer-depth.html
- [x] createFramebuffer() | https://p5js.org/reference/en/p5/createFramebuffer/
[x] https://p5js.org/tutorials/lights-camera-materials/
- [x] orbitControl() | https://p5js.org/reference/en/p5/orbitControl/
- [x] ambientLight() | https://p5js.org/reference/en/p5/ambientLight/
- [x] directionalLight() | https://p5js.org/reference/en/p5/directionalLight/
- [x] pointLight() | https://p5js.org/reference/en/p5/pointLight/
- [x] spotLight() | https://p5js.org/reference/en/p5/spotLight/
- [x] imageLight() | https://p5js.org/reference/en/p5/imageLight/
- [x] noLight() | https://p5js.org/reference/en/p5/noLight/
- [x] fill() | https://p5js.org/reference/en/p5/fill/
- [x] ambientMaterial() | https://p5js.org/reference/en/p5/ambientMaterial/
- [x] specularMaterial() | https://p5js.org/reference/en/p5/specularMaterial/
- [x] shininess() | https://p5js.org/reference/en/p5/shininess/
- [x] emissiveMaterial() | https://p5js.org/reference/en/p5/emissiveMaterial/
[x] https://p5js.org/tutorials/repeating-with-loops/
- [x] Variables and Motion | https://p5js.org/tutorials/variables-and-motion/
[x] https://p5js.org/tutorials/responding-to-inputs/
- [x] Conditions and Interactivity | https://p5js.org/tutorials/conditions-and-interactivity/
[x] https://p5js.org/tutorials/setting-up-your-environment/
- [x] 2D Primitive shapes | https://p5js.org/referencegroup-Shape/
- [x] using the p5.js Web Editor with a screen reader. | https://p5js.org/tutorials/using-web-editor-with-screen-reader/
- [x] How to Use the p5.js Web Editor with a Screen Reader | https://p5js.org/tutorials/using-web-editor-with-screen-reader/
- [x] color | https://p5js.org/referencegroup-Color/
[x] https://p5js.org/tutorials/simple-melody-app/
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
- [x] p5.Element | https://p5js.org/reference/p5.Element
- [x] DOM objects | https://p5js.org/referencegroup-DOM/
[x] https://p5js.org/tutorials/speak-with-your-hands/
- [x] Recursive Tree | https://p5js.org/examples/07_Repetition/05_Recursive_Tree/
- [x] Smoke Particles | https://p5js.org/examples/15_Math_And_Physics/03_Smoke_Particle_System/
- [x] Interactivity in p5.js | https://p5js.org/tutorial/conditionals-and-interactivity/
[x] https://p5js.org/tutorials/variables-and-change/
- [x] frameCount | https://p5js.org/reference/p5/framecount/
- [x] frameCount | https://p5js.org/reference/p5/framecount/

Would you like to work on the issue?

yes

Qianqianye commented 2 months ago

Thanks @ff6347, this looks great! We have multiple open issues about broken links that we are gradually fixing by batches, since the reasons for the broken links vary.

To provide some context:

p5.js website Contribute pages are generated directly from p5.js repo contributor_docs folder, so broken links will be fixed there.
p5.js website Reference pages are generated directly from p5.js repo src folder, so we’ll address broken links in that folder.
p5.js website Tutorials broken links can be fixed directly on p5.js website repo tutorial folder.
Any broken links related to p5.sound are currently on hold, as we’re planning to release a new version of the p5.sound library within the next month.

Here’s my suggestion: Let's use this issue as the main issue to consolidate all the broken link issues. We can update the issue with the latest report when some links are fixed, for examples, some broken links in the report generated on 2024-09-08 are already fixed. For anyone interested in contributing, feel free to start by fixing broken links in the Contribute or Tutorial pages in batches, then we can move to Reference page and others. Please go ahead and open PRs as you work through them. Thank you!

ff6347 commented 2 months ago

hi @Qianqianye thanks for the clarification where to look and start. 👌🏾

I updated the issue description to have check boxes rather then plaintext to track the progress. I also created a script to regenerate the report. https://github.com/ff6347/find-broken-links

I'll ping you as soon as I have the first PR.

thegodworm commented 1 month ago

Hi! I'm new to contributing and I wanted to let you know that I'm interested in fixing the broken links. I will open PR gradually.

thegodworm commented 1 month ago

So for the contributor docs, I don't get it. reading the doc ; p5js-websites pulls from p5.js then uses astro to build the static website right? " https://p5js.org/contribute/contributor_guidelines.md#software-design-principles" doesn't work but

https://p5js.org/contribute/contributor_guidelines/#software-design-principles works, it seems it's all due to .md not being present, thus refering to the correct page. I don't know how to correct this ( p5.js or p5jswebsite repo?) or the conversion script is missconfigured?

ff6347 commented 1 month ago

@thegodworm Astro transforms all Markdown .md files to a folder with the name of that file (minus the extension) and an index.html in it.

thegodworm commented 1 month ago

after inspections, the links are broken because they are not built correctly (via script?)

<a href="./pathToContent.md#anchor"> point To X</a> -> error 404.

works when corrected like this:

<a href="../pathToContent#anchor">** point To X</a>

adding ../ relative path correctly and removing the md.

I would ask what would be the best course of actions to take?

writing a script automatically addressing the issues? manually?

or modifying how the script for astro handle site building?

davepagurek commented 1 month ago

@thegodworm the content is intentionally specifying a .md file so that they work when viewed on GitHub, so I think doing it as part of a script makes sense here. There's already some initial parsing and converting done here that can be modified: https://github.com/processing/p5.js-website/blob/a872651a35b08201d84392e3e65e5e8ed85b8d4c/src/scripts/builders/contribute.ts#L90-L95

thegodworm commented 1 month ago

@davepagurek yes , I saw that, I will see then what I can do :) . I prefer to ask first because I want to be sure as it's my first collaboration 😃

davepagurek commented 1 month ago

No problem! Happy to have you helping out 🙂

ffd8 commented 1 week ago

Just a quick chime in while checking my teaching docs to see if/what broke and found a pretty big one: https://p5js.org/learn/ » https://p5js.org/tutorials/ ?

One idea would be to implement legacy dirs that forward, or better yet a custom 404 page that parses the URL window.location.href and compares to a list of new homes for those links? It could include a form input (if a true 404 without redirect) that creates an issue with expected URL and new URL if they research the solution.

ff6347 commented 6 days ago

I was thinking the same thing. The site needs a custom 404 page.

I'm not 100 % sure where this is deployed. If would be on cloudflare or netlify there is the possibility to configure redirect.

But I think it's GitHub pages with Cloudflare in front of it.

As far as I know, there is not the possibility to do this on GitHub pages. So a 404 page with some common links and a search might be necessary.

processing / p5.js-website