Consider adding a script to automatically clean old files

[bparks13]

When using Sphinx, there was an option in the make command to clean a directory to remove stale files that are not correctly being served anymore.

There is no readily available option for this in docfx that I have found, but we should be able to create a script that can automatically remove files from the _site directory, remove any existing SVG files that might have an old filename attached to them, and delete the files from the api directory in the case that the submodule has been updated and file names have changed.

There might be other things we can automate with this to help keep the working directory clean, but I think this covers the majority of cases.

[cjsha]

super down for this. This annoys me pretty often

[bparks13]

Based on proposed changes in #28, we will have to be careful not to remove any *.md files in the api folder.

[cjsha]

ok yeah. I think I can put them in a different directory without changing the URL. We should do that with the automatically exported SVGs as well, right? I just assume it's simpler to delete directories without sorting file extensions. I'll make it a .bat or .ps1 script unless there are any objections.

[bparks13]

The SVGs are fairly simple, since we can assume that the only filetype to be deleted in the workflows directories are .svg files. Plus, we really don't want to delete all of the .bonsai files in those directories!

I more so wanted to make sure we were excluding the correct files from other directories (i.e., don't delete .gitignore and *.md files). Moving these markdown files out makes it easier. I have no preference on the script type, as long as we highlight it in the README so everyone understands how to use it.

[cjsha]

I want to extend this issue to making a script for linkchecking and wanted to know what you think and if you have any input @bparks13

[bparks13]

@cjsha are you thinking about making two scripts, one for linkchecking and the other for cleaning? I think that's a really good idea actually, but it might require that we package lychee in some way, or provide specific instructions on how to set it up for use with a script. In general I am a fan of scripting things and automating them to a degree.

[cjsha]

To do a proper linkcheck, the script should start from a clean build anyway, so I was thinking I would just put both in one script and depending on what the user wants to do they could run ./docfx-util.ps1 -clean or ./docfx-util.ps1 -linkcheck <lychee/path>. Running the command with both ./docfx-util.ps1 -clean -linkcheck <lychee/path> arguments is equivalent to running it with just ./docfx-util.ps1 -linkcheck <lychee/path>. And we can add more functionality in the future as we see fit. I agree it would be nice to package it someway so we don't have to specify the path, but for now, what do you think?

[bparks13]

Hmm, I can see the benefit of packing a utility script all together.

If we go this route, I would recommend that we keep the utilities totally separate and only run when the explicit flag is given. So then here is my proposal:

• -clean: deletes old files and SVGs as discussed previously
• -linkcheck: With no arguments this simply runs .\build.ps1 --logLevel Warning --warningsAsErrors and dotnet DocLinkChecker -v -f .github/workflows/DocLinkChecker.config
  • To me, this is the first level of link checking, since it is mostly internal to the docfx system
• -linkcheck /path/to/lychee.exe: Runs the first level of link checking, and also runs the lychee executable using the same commands as found in the actions file
• -full or -full /path/to/lychee: This would accomplish what you suggest as the -linkcheck flag, which is to say this would run the clean command, then do a full rebuild and check all links

As I write this out I wonder if there is some way to read the actions file and directly pull out the commands from there? We don't want to get into a situation where the two scripts become desynchronized, because then local builds could succeed when the actions fail, or vice versa.

[bparks13]

Also, to stick with convention, any words we use as flags should be preceded by --, and if we wanted to add short-form flags those would be -. So we could have --clean and -c.

[cjsha]

As I write this out I wonder if there is some way to read the actions file and directly pull out the commands from there? We don't want to get into a situation where the two scripts become desynchronized, because then local builds could succeed when the actions fail, or vice versa.

Ah great idea. I did a brief search, and I came across the suggestion to put all the linkcheck commands in the powershell script and to run the powershell script during the GitHub Actions build process (instead of trying to pull the commands into the powershell script from the build.yml). This seems like the most simple way to keep these things synchronized.

Also, to stick with convention, any words we use as flags should be preceded by --, and if we wanted to add short-form flags those would be -. So we could have --clean and -c.

I had the same thought but it seems Powershell scripts have -parameterKey <parameter-value>/-parameterKey:<parameter-value> syntax already built-in, and I didn't want to look much into getting around that while we have other past due milestones/issues.

• -clean: deletes old files and SVGs as discussed previously
• -linkcheck: With no arguments this simply runs .\build.ps1 --logLevel Warning --warningsAsErrors and dotnet DocLinkChecker -v -f .github/workflows/DocLinkChecker.config
  • To me, this is the first level of link checking, since it is mostly internal to the docfx system
• -linkcheck /path/to/lychee.exe: Runs the first level of link checking, and also runs the lychee executable using the same commands as found in the actions file
• -full or -full /path/to/lychee: This would accomplish what you suggest as the -linkcheck flag, which is to say this would run the clean command, then do a full rebuild and check all links

This is fine. It is annoying to export all .svgs every time if I just wanna do a lychee linkcheck.

Do we prefer -linkcheck or -checklinks? I realize I have the -checklinks implemented, but I wrote -linkcheck all over my previous comment.

[cjsha]

I'm also not immediately sure there's be a way to run -linkcheck without additional arguments so I might end up separating the 2nd and 3rd bullet points into -linkcheck and -lychee <path/to/lyche>

[bparks13]

put all the linkcheck commands in the powershell script and to run the powershell script during the GitHub Actions build process

That sounds great to me, let me know whenever this is finalized and merged, then I'll work on integrating the script into the actions. I think that makes the most sense

I had the same thought but it seems Powershell scripts have -parameterKey /-parameterKey: syntax already built-in

Got it, I'm not as familiar with Powershell but if that's already there then we can utilize it

This is fine. It is annoying to export all .svgs every time if I just wanna do a lychee linkcheck.

What would be an intermediate then? If we are already splitting functionality, maybe we can word it in a way so that the lychee check is separate from the other checks? Also I'm fine with -checklinks.

Maybe we split it up so that we have:

• -clean: see above
• -checklinks: builds and runs DocLinkChecker
• -lychee /path/to/lychee.exe: only runs lychee
• -full /path/to/lychee.exe or -all or something like that, I don't love either of these but in that vein

Obviously -full (or whatever we decide on) will run everything, but if we only want to run the -lychee command we can.