Netlify deployment not deploying dot files/directories

[daveaglick]

As reported by @patriksvensson if you create a folder like .well-known it'll get generated and output as expected, but won't deploy to Netlify. Thanks to @nils-a for finding this Netlify thread that seems to explain and confirm the behavior: https://answers.netlify.com/t/netlify-deploy-api-removes-files-and-directories-beginning-with-a-period/37728

So the questions is how to fix it? We could automatically look for this pattern, move the files, and create redirect entries as suggested in https://github.com/netlify/cli/issues/11#issuecomment-401854074, but that seems really hacky.

We could also switch to a whole other deployment mechanism with Netlify, but I don't love changing something that otherwise has been stable and working well (not to mention the time it'll take).

Maybe the best hope is that Netlify fixes the behavior soon.

[nils-a]

I feel netlify is at fault here and Statiq can not be expected to provide a solution. I feel we could, however provide an easy workaround.

What if we created a module that would automatically redirect a whole folder based on a setting?

Say the user creates a folder named dotWellKnown/ and supplies the setting to redirect from .well-known/ to dotWellKnown/.

Statiq would then go its normal way of outputting the documents to the dotWellKnown/ folder, create redirect documents in the .well-known/ folder and, since we are on netlify also write the redirects into the _redirects file.

Now, when the deployment to netlify occurs, the documents in .well-known/ will not be deployed. This won't be a problem though, because the _redirects file will be deployed.

[daveaglick]

I feel netlify is at fault here and Statiq can not be expected to provide a solution.

I generally agree, but the pragmatic side of me also feels like if there's something we can do, might as well do what we can.

The more I thought about this, the more I think automating essentially what you described might work. I.e. if a particular flag is set (something like NetlifyDotFolderRedirects, which we can set automatically when Netlify deployment is turned on, or user can set manually if deploying to Netlify externally) then we'll find dot folders, rename them, and add the correct _redirects entries.

Now the complication is that it looks like we're also skipping _redirects files, so I need to look into that first. Have a tiny bit of time so might end up hacking this whole feature together today if it's not too tough.

[nils-a]

Now the complication is that it looks like we're also skipping _redirects files [...]

I think we're only skipping them when they are not automatically created. If "we" create them they do have a nice contentType associated and will be added to the output.

https://github.com/statiqdev/Statiq.Framework/blob/57dd8075157c7bc4b42e51fb22ac9d6f3cd73d98/src/core/Statiq.Core/Modules/Content/GenerateRedirects.cs#L111-L125

However, I feel that logic currently misses some way to define static redirects to be added. (Or rather add or modify the list of redirects)

[daveaglick]

I think we're only skipping them when they are not automatically created. If "we" create them they do have a nice contentType associated and will be added to the output.

Yep - went down a rabbit hole on this a bit this afternoon in preparation for making it work with dot files/folders. I wanted an existing _redirects file to carry over to output when NetlifyRedirects is true (which seemed like a reasonable signal for making sure a _redirects file makes it to output since it's a Netlify file convention). It now also makes sure that any redirects we generate are combined with an existing redirects file rather than overwriting it.

Now that that's done, I'll likely tackle the dot file/folders thing tomorrow.

[daveaglick]

Ended up not quite making this automatic. It was too hard to perform renaming of documents from other pipelines, so instead it works by generating content in the _redirects file for any file or folder that starts with a double underscore __. This behavior is automatically turned on when NetlifyRedirects is true (but can be turned off by setting NetlifyUnderscoreRedirects to false). A side benefit is that it'll work for any file or folder with a double underscore prefix just in case there are other situations like this where you can't upload or serve the asset in Netlify directly.

So as an example, for a .well-known folder, you'd name that __.well-known in your input folder, set NetlifyRedirects to true, and you're done.

[daveaglick]

Small change of plans - the prefix for designating these is now a double tilde ~~ instead of underscores since underscore files are already ignored by Statiq by default and revising that would be a whole thing.

So as a revised example, for a .well-known folder, you'd name that ~~.well-known in your input folder, set NetlifyRedirects to true, and you're done.

[daveaglick]

One final update: since ~~ was reserved for a totally different reason (it gets stripped if filename optimization is turned on), the final default prefix is ^, but it can also be changed by using a NetlifyRedirectPrefix setting.

With that, this is ready to ship in the next release once I clean up some Statiq Framework stuff I added to support it, add some more tests, and write documentation.