search

gohugoio / hugo

The world’s fastest framework for building websites.
https://gohugo.io
Apache License 2.0
75.7k stars 7.53k forks source link

ignorerFiles cannot ignore folders recursively (?) #7748

Closed chunyunma closed 4 years ago

chunyunma commented 4 years ago

What version of Hugo are you using (hugo version)?

$ hugo version
0.75.1

Does this issue reproduce with the latest release?

Yes

Some of my content needs to be knitted using R. To streamline the process, I set up an Rproject inside the content folder. As part of this workflow, I also use revn to manage packages, which means I have several files/folders inside content/blog that need to be ignored before rendering. One of the folders has multiple levels, i.e., folder/subfolder/subsubfolder/.... I could not figure out how to ignore all of them using ignoreFiles inside config.toml.

So far, I have tried:

ignoreFiles = [ ...,  "folder", ...]
ignoreFiles = [ ...,  "folder/.*", ...]
ignoreFiles = [ ...,  "folder/\\*\\*/\\*", ...]
ignoreFiles = [ ...,  "folder/\\*\\*/\\.\\*", ...]

Sadly, none of them worked. Each time, Hugo would try to render some files within this to-be-ignored folder and crash.

I have searched online and came across several mentioning of issues related to ignoreFiles, but none of them mentioned explicitly a folder with multiple layers.

Any suggestion would be greatly appreciated!!

vitkhab commented 4 years ago

Hey, @chunyunma. I was trying to reproduce the problem and have failed. Could you provide a slice of your site's source code for testing? How do you run hugo (What command do you use)?

Further I describe how I tested. I created file hugo-site-root/content/post/folder/subfolder/index.md containing:

---
title = "Some title
---

On rebuilt I've got an error:

ERROR 2020/10/04 17:25:36 "/home/vitkhab/hugo-site-root/content/post/folder/subfolder/index.md:2:1": failed to unmarshal YAML: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `title =...` into map[string]interface {}

The problem was fixed when I did put any of options below in config.toml:

ignoreFiles = [ "subfolder" ]
ignoreFiles = [ "folder" ]
ignoreFiles = [ "index.md" ]
chunyunma commented 4 years ago

Hi @vitkhab , thank you very much for the detailed reply! I may have identified the problem but have yet to find a solution (sad face).

I was able to reproduce the error and solution you described, which led me to look more closely at the folder I was trying to ignore previously. It turns out that the original folder (renv) has some symbolic links in addition to regular files. Here is a simplified structure of my Hugo folder:

|-- Hugo-site-root |----- Content |-------- blog |----------- renv |------------- symbolic link1 |------------- symbolic link2 ...

All the symbolic links point to /Users/chunyun/Library/Application Support/renv/cache/...

When I run "hugo server", I get the following message: Watching for changes in /Users/chunyun/{Documents,Library} Watching for config changes in ... /config.toml fatal error: pipe failed

Even though I already specified in my config file ignoreFiles = ["renv"], it seems that Hugo still gets "distracted" by these symbolic links and try to watch for changes in /Users/chunyun/{Documents,Library}. In contrast, when I remove the renv folder, and run hugo server, I would get the usual & normal message: "Watching for changes in /Users/chunyun/Documents/hugo-site-root/{archetypes,content,data,layouts,static}".

Does this mean I cannot have a folder like renv co-exist with hugo content?

Thank you again!

vitkhab commented 4 years ago

I tried moving folder with faulty file outside hugo-site-root and making symlink to it. ignoreFiles worked well when I specified symlink name or symlinked folder name. I still can't reproduce the issue.

I'll try later with renv.

vitkhab commented 4 years ago

I've initialized renv inside hugo-site-root/content/post folder. I've installed some random packages. I can see symlinks inside hugo-site-root/content/post/renv/library/R-4.0/x86_64-pc-linux-gnu leading to /home/vitkhab/.local/share/renv/cache.

Some packages have good-enough README.md with them and Hugo compiles them well. Some packages have broken README.md and after their installation Hugo site broke. ignoreFiles = [ "renv" ] "fixed" issue. As I understand, renv using basic symlinks.

But I tested on Ubuntu 20.04 running on WSL2 on Windows 10. And I see you are using MacOS. It might be OS-specific problem. Like hugo doesn't have permissions to Library folder in MacOS and it crashes trying to read files from it (as a suggestion, I don't know what the real problem is). I'm thinking how to test it. Luckily I've also have Mac =)

chunyunma commented 4 years ago

Thank you so much for the update!! It's very intriguing that Hugo worked fine on your machine. BTW, I am running Hugo on macOS Catalina. Should have mentioned that at the beginning.

My apology if I was not being thorough, but the error message I received also has a large chunk after fatal error: pipe failed. I did not include them in my previous message because they made no sense to me.

They read something like:

goroutine 1 [running]:
runtime.throw(0x5c92adc, 0xb)
    /usr/local/Cellar/go/1.15.2/libexec/src/runtime/panic.go:1116 +0x72 fp=0xc001389860 sp=0xc001389830 pc=0x4039332
runtime.sigNoteSetup(0x6d7c740)
    /usr/local/Cellar/go/1.15.2/libexec/src/runtime/os_darwin.go:98 +0xc5 fp=0xc001389888 sp=0xc001389860 pc=0x4036085
os/signal.signal_enable(0x3e64303d00000002)
    /usr/local/Cellar/go/1.15.2/libexec/src/runtime/sigqueue.go:198 +0xa5 fp=0xc0013898a8 sp=0xc001389888 pc=0x406dc05
os/signal.enableSignal(...)
    /usr/local/Cellar/go/1.15.2/libexec/src/os/signal/signal_unix.go:49
os/signal.Notify.func1(0x2)
    /usr/local/Cellar/go/1.15.2/libexec/src/os/signal/signal.go:144 +0x88 fp=0xc0013898c8 sp=0xc0013898a8 pc=0x55ae748
os/signal.Notify(0xc0010854a0, 0xc001389ae0, 0x2, 0x2)

On a hunch, I ran brew info Hugo and noticed a x in 

==> Dependencies
Build: go x

If I understood this correctly, my Hugo install thinks that I do not have go (?)

However, when I ran brew info go, it did show that I already had go installed on my machine.

Could this have anything to do with the error I've been getting?

Thank you again for all your help so far!

vitkhab commented 4 years ago

When I symlinked to folder without permissions I've got an error on the start:

ERROR 2020/10/05 19:49:04 walker:  walk: open "post/renv/renv-test" (""): open /Users/vitkhab/hugo-site-root/content/post/renv/renv-test: operation not permitted

It's very different from your problem. I've tried with hugo installed using brew and downloaded binary from github with and without ignoreFiles. The result was the same. So I don't know how to reproduce the problem.

But hugo version caught my eye. If you download binary from GitHub it should tell: Hugo Static Site Generator v0.75.1-A4A7BAB7 darwin/amd64 BuildDate: 2020-09-15T06:47:14Z

When I've installed it using brew, I've got: Hugo Static Site Generator v0.75.1/extended darwin/amd64 BuildDate: unknown

But you have 0.75.1. Is it possible that you are using hugo built in some strange way? Or you just trimmed output? Can you try to use hugo binary downloaded from GitHub (you might need to double click on it and give permissions to run)?

Also can you attach full log from hugo startup?

chunyunma commented 4 years ago

I previously installed Hugo using brew.
I uninstalled and replaced it with the binary as you suggested (confirmed with hugo version). I also created a dummy site from scratch using the binary install Hugo. I made the following changes and tested Hugo at each step:

hugo new site ./hugo-site-root hugo server --- normal output

Created an R project inside content folder hugo server --- normal output

renv::init() hugo server --- normal output

Install.packages("tidyverse") hugo server --- error

Below is the full output from the last hugo server

Start building sites … 

                   | EN  
-------------------+-----
  Pages            |  6  
  Paginator pages  |  0  
  Non-page files   |  1  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Built in 6 ms
Watching for changes in /Users/chunyun/{Documents,Library}
Watching for config changes in /Users/chunyun/Documents/hugo-site-root/config.toml
fatal error: pipe failed

goroutine 1 [running]:
runtime.throw(0x2ae8165, 0xb)
    /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc000d99860 sp=0xc000d99830 pc=0x1036712
runtime.sigNoteSetup(0x3b859e0)
    /usr/local/go/src/runtime/os_darwin.go:98 +0xc5 fp=0xc000d99888 sp=0xc000d99860 pc=0x1033465
os/signal.signal_enable(0xa256d17700000002)
    /usr/local/go/src/runtime/sigqueue.go:198 +0xa5 fp=0xc000d998a8 sp=0xc000d99888 pc=0x106aea5
os/signal.enableSignal(...)
    /usr/local/go/src/os/signal/signal_unix.go:49
os/signal.Notify.func1(0x2)
    /usr/local/go/src/os/signal/signal.go:144 +0x88 fp=0xc000d998c8 sp=0xc000d998a8 pc=0x25a0508
os/signal.Notify(0xc001233500, 0xc000d99ae0, 0x2, 0x2)
    /usr/local/go/src/os/signal/signal.go:164 +0x162 fp=0xc000d99940 sp=0xc000d998c8 pc=0x259ff22
github.com/gohugoio/hugo/commands.(*commandeer).serve(0xc0009248f0, 0xc0004c97c0, 0x749, 0xa00)
    /root/project/hugo/commands/server.go:480 +0x625 fp=0xc000d99b60 sp=0xc000d99940 pc=0x25d8be5
github.com/gohugoio/hugo/commands.(*serverCmd).server(0xc0004c97c0, 0xc0001e2dc0, 0x3b853f8, 0x0, 0x0, 0x0, 0x0)
    /root/project/hugo/commands/server.go:274 +0x2b6 fp=0xc000d99ca8 sp=0xc000d99b60 pc=0x25d73b6
github.com/gohugoio/hugo/commands.(*serverCmd).server-fm(0xc0001e2dc0, 0x3b853f8, 0x0, 0x0, 0x0, 0x0)
    /root/project/hugo/commands/server.go:131 +0x52 fp=0xc000d99cf0 sp=0xc000d99ca8 pc=0x25e7592
github.com/spf13/cobra.(*Command).execute(0xc0001e2dc0, 0x3b853f8, 0x0, 0x0, 0xc0001e2dc0, 0x3b853f8)
    /go/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:838 +0x47c fp=0xc000d99dc0 sp=0xc000d99cf0 pc=0x11ef95c
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002e7340, 0xc000412dc0, 0x8, 0xc00043b000)
    /go/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:943 +0x336 fp=0xc000d99e98 sp=0xc000d99dc0 pc=0x11f0496
github.com/gohugoio/hugo/commands.Execute(0xc0000ba010, 0x1, 0x1, 0x1005fe5, 0xc000094058, 0x0, 0x0)
    /root/project/hugo/commands/hugo.go:90 +0xb9 fp=0xc000d99f28 sp=0xc000d99e98 pc=0x25c4259
main.main()
    /root/project/hugo/main.go:23 +0x76 fp=0xc000d99f88 sp=0xc000d99f28 pc=0x25e8cd6
runtime.main()
    /usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc000d99fe0 sp=0xc000d99f88 pc=0x1038ee9
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000d99fe8 sp=0xc000d99fe0 pc=0x106eaa1

goroutine 21 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00001e870)
    /go/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
    /go/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:32 +0x57

goroutine 57 [syscall]:
syscall.syscall6(0x122f740, 0x8, 0x0, 0x0, 0xc000484688, 0xa, 0x3b856a0, 0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/sys_darwin.go:85 +0x2e
golang.org/x/sys/unix.kevent(0x8, 0x0, 0x0, 0xc000484688, 0xa, 0x3b856a0, 0x0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/sys@v0.0.0-20200501145240-bc7a7d42d5c3/unix/zsyscall_darwin_amd64.go:292 +0xa6
golang.org/x/sys/unix.Kevent(0x8, 0x0, 0x0, 0x0, 0xc000484688, 0xa, 0xa, 0x3b856a0, 0x0, 0x0, ...)
    /go/pkg/mod/golang.org/x/sys@v0.0.0-20200501145240-bc7a7d42d5c3/unix/syscall_bsd.go:413 +0x71
github.com/fsnotify/fsnotify.read(0x8, 0xc000484688, 0xa, 0xa, 0x3b856a0, 0xc000484688, 0x0, 0xa, 0x0, 0x0)
    /go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:511 +0x6e
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000d32d80)
    /go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:274 +0x831
created by github.com/fsnotify/fsnotify.NewWatcher
    /go/pkg/mod/github.com/fsnotify/fsnotify@v1.4.9/kqueue.go:62 +0x199

goroutine 59 [select]:
github.com/gohugoio/hugo/commands.(*commandeer).newWatcher.func1(0xc000e02500, 0xc0009248f0, 0xc00048d458, 0xc0005fcd50)
    /root/project/hugo/commands/hugo.go:873 +0xe5
created by github.com/gohugoio/hugo/commands.(*commandeer).newWatcher
    /root/project/hugo/commands/hugo.go:871 +0x2ac

goroutine 58 [select]:
github.com/gohugoio/hugo/watcher.(*Batcher).run(0xc000e02500)
    /root/project/hugo/watcher/batcher.go:53 +0x174
created by github.com/gohugoio/hugo/watcher.New
    /root/project/hugo/watcher/batcher.go:42 +0x125

goroutine 61 [select]:
github.com/gohugoio/hugo/livereload.(*hub).run(0x3b481c0)
    /root/project/hugo/livereload/hub.go:39 +0x1e9
created by github.com/gohugoio/hugo/livereload.Initialize
    /root/project/hugo/livereload/livereload.go:98 +0x45
chunyunma commented 4 years ago

@vitkhab After days' agony, I finally found a reasonable solution! It is actually quite simple. All I did was to add one line to the .Renviron file inside content/blog/:

RENV_PATHS_LIBRARY = "/Users/chunyun/Documents/renv/hugo/"

This environment variable effectively places all the symlinked packages in a folder outside of hugo-site-root. This way, there are no symlinks inside folder renv and ignoreFiles = [ "renv" ] worked as expected, i.e., recursively.

BTW, prior to my lucky success, I also tried setting another environment variable RENV_PATHS_ROOT to a folder inside $HOME, effectively placing all the cached R packages somewhere accessible to Hugo. This attempt did not solve the problem though.

I used this documentation from renv to come up with this solution.

github-actions[bot] commented 2 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.