Error when building to a folder not owned by the Hugo process

[klakegg]

What version of Hugo are you using (hugo version)?

$ hugo version
Hugo Static Site Generator v0.71.0-06150C87/extended linux/amd64 BuildDate: 2020-05-18T16:15:29Z

Does this issue reproduce with the latest release?

Yes it does.

The error occures when the Hugo process have write access to the destination folder but does not own it.

To reproduce, create a public folder owned by root in your Hugo site project (requires sudo access):

sudo sh -c "mkdir public && chmod a+w public"

Run Hugo:

hugo

Log:

Building sites … WARN 2020/05/21 14:09:25 found no layout file for "HTML" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
Total in 111 ms
Error: Error copying static files: chtimes [..]/public/: operation not permitted

[davidsneighbour]

Can you please check the file rights on the folder? ls -al foldername should do the trick on a linux system. Nobody should be able to use a folder with only root rights, so that is not a bug. The issue is that you want to give write access to group and all. Worst case something around 0777. chmod 0777 public will solve the issue. The question is if that is safe enough for your environment and setup. If you need to run the creation of the folder with root then something is wrong with your users and setup, not Hugo. Hugo is doing the right thing.

[klakegg]

@davidsneighbour Thank you for having a look at my issue report.

I'm aware of the safety aspects of this, I simply constructed an example that should make it easy to reproduce what I reported.

Let me make it step by step, basically as described in my initial post.

Listing before manually creating public folder:

hugo:/workspaces/site-klakegg/src$ ls -la
total 32
drwxr-xr-x 7 hugo hugo 4096 May 21 15:13 .
drwxr-xr-x 7 hugo hugo 4096 May 21 06:34 ..
drwxr-xr-x 3 hugo hugo 4096 May  1 20:20 assets
-rw-r--r-- 1 hugo hugo  282 May  1 20:09 config.yml
drwxr-xr-x 3 hugo hugo 4096 May  1 20:18 content
drwxr-xr-x 4 hugo hugo 4096 May  1 20:18 layouts
drwxr-xr-x 3 hugo hugo 4096 May  1 19:19 resources
drwxr-xr-x 2 hugo hugo 4096 May  1 20:17 static

Creating the public folder, and setting rights accoring to your suggestion:

hugo:$ sudo mkdir public
hugo:$ sudo chmod 0777 public

Here is the listing with the public folder created:

hugo:$ ls -la        
total 36
drwxr-xr-x 8 hugo hugo 4096 May 21 14:27 .
drwxr-xr-x 7 hugo hugo 4096 May 21 06:34 ..
drwxr-xr-x 3 hugo hugo 4096 May  1 20:20 assets
-rw-r--r-- 1 hugo hugo  282 May  1 20:09 config.yml
drwxr-xr-x 3 hugo hugo 4096 May  1 20:18 content
drwxr-xr-x 4 hugo hugo 4096 May  1 20:18 layouts
drwxrwxrwx 4 root root 4096 May 21 14:27 public
drwxr-xr-x 3 hugo hugo 4096 May  1 19:19 resources
drwxr-xr-x 2 hugo hugo 4096 May  1 20:17 static

Triggering build:

hugo:$ hugo
Building sites … WARN 2020/05/21 15:10:36 found no layout file for "HTML" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
Total in 103 ms
Error: Error copying static files: chtimes [..]/public/: operation not permitted

Clearly chmod 0777 public did not solve the issue.

It is not clear to me why it is important to trigger chtimes on the folder containing the produced content.

[davidsneighbour]

That is weird. With those file rights I would expect everyone being able to change things in the folder. I googled around and found, that chtimes is a GO function, not a shell thing - assumed the first. It might still be file rights related, see the following issue unrelated:

https://github.com/rclone/rclone/issues/738#issuecomment-248603775

1. Why do you need to create the folder public as user root, not user hugo? Can you try creating it as user hugo (should somehow work via sudo -u hugo or the like)?
2. Is the file location somehow mounted and are the mount rights either set restrictive or without rights as the issue above discussed?

What's the operating system you are using here?

[klakegg]

1. I've create an example using the root user as I guess that user is accessible to most non-Windows users following this repo.
2. The folder is not mounted, however everything runs inside a Docker container without any mounting magic.

I've verified it using both Debian and Ubuntu.

[davidsneighbour]

I can confirm that it's not docker related. I tried on my own repo locally with a root user created public folder (process got frozen) that was then set to 0777 (process ended up with a chtimes error).

So it's probably an issue in how chtimes is called in Go or Hugo.

sudo hugo (the script kiddie solution) by the way ends up in Hugo not finding the configuration.

This all does not solve the mystery why you would create a folder owned by root instead of creating a folder owned by a more public user (the one running hugo, or a completely public one).

Some ideas that probably won't work:

• add root to the usergroup hugo
• add hugo to the usergroup root/sudo (unsafe)
• and still, why is root creating a public directory?

This is where my knowledge about the system ends and @bep or another Go-God might have a look at it.

[klakegg]

It is not related to the root user. Try making the folder this way (1337 or other uid not used):

$ mkdir public
$ chmod 0777 public
$ sudo chown 1337 public

Please notice the group ownership is not changed in the above commands.

This will result in the same error when triggering Hugo. The root ownership is simply not relevant, the fact that the folder is owned by any other entity than id -u is the important part.

[bep]

Error: Error copying static files: chtimes [..]/public/: operation not permitted

Does it work if you do

hugo --noChmod

Or maybe

hugo --noChmod --noTimes

[klakegg]

Using --noTimes makes it work, both with and without --noChmod.

[bep]

Using --noTimes

OK, so this is what's giving a permission error https://golang.org/pkg/os/#Chtimes

[jmooring]

TLDR: This is an OS restriction. I recommend closing this issue.

According to POSIX.1-2017:

Only a process with the effective user ID equal to the user ID of the file, or with write access to the file, or with appropriate privileges may use futimens() or utimensat() with a null pointer as the times argument or with both tv_nsec fields set to the special value UTIME_NOW. Only a process with the effective user ID equal to the user ID of the file or with appropriate privileges may use futimens() or utimensat() with a non-null times argument that does not have both tv_nsec fields set to UTIME_NOW and does not have both tv_nsec fields set to UTIME_OMIT.

What does this mean?

1. If you have write permissions to the file, you can update both access and modify times with touch foo.txt
2. You must be the file owner, not just a member of the group, to:
   • Change only the access time with touch -a foo.txt
   • Change only the modify time with touch -m foo.txt

touch foo.txt
chmod 666 foo.txt
sudo chown root foo.txt #  The behavior is not specific to root; could be any user
stat foo.txt

  File: foo.txt
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: 802h/2050d  Inode: 6685210     Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/    root)   Gid: ( 1000/jmooring)
Access: 2021-11-27 06:34:11.052026188 -0800
Modify: 2021-11-27 06:34:11.052026188 -0800
Change: 2021-11-27 06:38:54.189880908 -0800
 Birth: -

touch foo.txt  # No problems, all times are changed
stat foo.txt

  File: foo.txt
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: 802h/2050d  Inode: 6685210     Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/    root)   Gid: ( 1000/jmooring)
Access: 2021-11-27 06:41:43.049344443 -0800
Modify: 2021-11-27 06:41:43.049344443 -0800
Change: 2021-11-27 06:41:43.049344443 -0800
 Birth: -

touch -a foo.txt  # Operation not permitted
touch -m foo.txt  # Operation not permitted

I recommend closing this issue:

• It is an OS restriction
• The setup (a publishDir owned by somebody else) is not common
• There is a CLI option to overcome the restriction when needed (--noTimes)

[bglw]

If this cannot be fixed, could the error message instead be improved to catch this case and recommend the --noTimes solution?

[somu1795]

A simple solution would be to change the ownership of the directory to the ID with which you execute hugo. no further cli parameters required.

[bglw]

Not always possible in a CI environment, unfortunately.

[smaudet]

I also ran into this, I want a user to be able to run this without owning the files in question (group access).

I guess chtimes is a user-level permission?

I made a uhugo shortcut that aliases to hugo --noTimes - maybe this shouldn't be a default option (or detect if you have the permissions and warn instead of fail).

[jmooring]

@smaudet You can set noTimes = true in the site configuration instead of the command line flag. That might be a bit easier.