Caddy thinks filesystem is read-only

mpldr commented 4 years ago

Caddy Version: v2.0.0-beta.14

Command caddy was started with: sudo -u http caddy run --config /etc/caddy.conf --adapter caddyfile

/etc/caddy.conf

my-domain.com {
    reverse_proxy /jenkins/* localhost:46826
    root * /var/lib/caddy2
}

Log:

2020/02/17 15:54:28 [ERROR] my-domain.com: obtaining certificate: failed storage check: open /srv/http/.local/share/caddy/rw_test_1394922878: read-only file system - storage is probably misconfigured - backing off and retrying (attempt 9/46)...

Is it read-only? No. Tested with sudo -u http touch /srv/http/.local/share/caddy/rw_test_1394922878 and worked without problems.

mholt commented 4 years ago

I don't know what to tell you; Caddy is clearly getting "read-only file system" from the OS, so Caddy is unable to write to that file.

It's not immediately clear to me what is going on, so I'll need your help to understand it better.

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either. Note that we'll have to reproduce the exact "read-only file system" error as well. Unfortunately, I don't know anything about your system, so you'll have to figure that out.

I've attached a template below that will help make this easier and faster! It will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1:

I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```

### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```

### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```

## 2. Description

### 2a. What happens (briefly explain what is wrong)

### 2b. Why it's a bug (if it's not obvious)

### 2c. Log output

```
paste terminal output or logs here
```

### 2d. Workaround(s)

### 2e. Relevant links

## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.
Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.
- 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
- 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
- 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.
Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:
- Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
- Do not run Caddy inside containers.
- Run Caddy manually in your terminal; do not use systemd or other init systems.
- If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
- Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
- Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file: ``` { ... } ``` Open terminal and run Caddy: ``` $ caddy ... ``` Make an HTTP request: ``` $ curl ... ``` Notice that the result is ___ but it should be ___.

mpldr commented 4 years ago

1. Environment

1a. Operating system and version

Linux Hostname 4.19.102-2-ARCH #1 SMP PREEMPT Sun Feb 9 22:52:12 UTC 2020 armv7l GNU/Linux

1b. Caddy version (run `caddy version` or paste commit SHA)

(devel)

1c. Go version (if building Caddy from source; run `go version`)

go version go1.13.7 linux/arm

2. Description

2a. What happens (briefly explain what is wrong)

Caddy is unable to write to a path it should be able to write to (see above)

2c. Log output

2020/02/17 15:54:28 [ERROR] my-domain.com: obtaining certificate: failed storage check: open /srv/http/.local/share/caddy/rw_test_1394922878: read-only file system - storage is probably misconfigured - backing off and retrying (attempt 9/46)...

3. Tutorial (minimal steps to reproduce the bug)

i have no idea.

Installed ArchlinuxARM on my RasPi 4 Build Caddy Wonder why it thinks its a readonly fs

mholt commented 4 years ago

I am unable to reproduce the problem, without a tutorial. :(

This is a system misconfiguration, not a Caddy bug. Let us know when you find out what is causing it!

You can ask on our forums, maybe someone there will have advice: https://caddy.community

mpldr commented 4 years ago

Found the reason. My http user has it's home directory under /srv/http which is sysfs, a circumstance i was oblivious to.

For obvious reasons it's not displayed by df as it's a pseudo file-system.

sysfs(5) states that:

Many of the files in the sysfs filesystem are read-only, but some files are writable, allowing kernel variables to be changed.

So the issue does not lie with caddy but with the home-directory of my http-user. Changed it to /var/http and it's working now.

Additionally it's a good idea to check your System Unit for a ProtectSystem entry and exceptions if necessary.

mholt commented 4 years ago

Thanks for following-up with the fix! It will help others who have the same problem.

caddyserver / caddy