search

jonaswinkler / paperless-ng

A supercharged version of paperless: scan, index and archive all your physical documents
https://paperless-ng.readthedocs.io/en/latest/
GNU General Public License v3.0
5.38k stars 353 forks source link

[BUG] Webserver not starting after upgrade to 1.4.0 #879

Open Bart1909 opened 3 years ago

Bart1909 commented 3 years ago

After updating to 1.4.0 the web server will not start. Docker-Output is:

broker_1     | 1:C 07 Apr 2021 12:41:45.274 # Redis version=6.0.12, bits=32, commit=00000000, modified=0, pid=1, just started
broker_1     | 1:C 07 Apr 2021 12:41:45.274 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
webserver_1  | Paperless-ng docker container starting...
webserver_1  | creating directory /tmp/paperless
broker_1     | 1:M 07 Apr 2021 12:41:45.280 # Warning: 32 bit instance detected but no memory limit set. Setting 3 GB maxmemory limit with 'noeviction' policy now.
broker_1     | 1:M 07 Apr 2021 12:41:45.281 * Running mode=standalone, port=6379.
broker_1     | 1:M 07 Apr 2021 12:41:45.281 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
broker_1     | 1:M 07 Apr 2021 12:41:45.281 # Server initialized
broker_1     | 1:M 07 Apr 2021 12:41:45.281 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
broker_1     | 1:M 07 Apr 2021 12:41:45.283 * Loading RDB produced by version 6.0.12
broker_1     | 1:M 07 Apr 2021 12:41:45.283 * RDB age 20 seconds
broker_1     | 1:M 07 Apr 2021 12:41:45.283 * RDB memory usage when created 0.63 Mb
broker_1     | 1:M 07 Apr 2021 12:41:45.283 * DB loaded from disk: 0.000 seconds
broker_1     | 1:M 07 Apr 2021 12:41:45.283 * Ready to accept connections
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: unable to get time of day: Operation not permitted
webserver_1  | sudo: error initializing audit plugin sudoers_audit
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: unable to get time of day: Operation not permitted
webserver_1  | sudo: error initializing audit plugin sudoers_audit
paperless_webserver_1 exited with code 1
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: unable to get time of day: Operation not permitted
webserver_1  | sudo: error initializing audit plugin sudoers_audit
paperless_webserver_1 exited with code 1
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: unable to get time of day: Operation not permitted
webserver_1  | sudo: error initializing audit plugin sudoers_audit
paperless_webserver_1 exited with code 1
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: unable to get time of day: Operation not permitted
webserver_1  | sudo: error initializing audit plugin sudoers_audit
paperless_webserver_1 exited with code 1
jonaswinkler commented 3 years ago

Host OS? Docker version?

Bart1909 commented 3 years ago

Raspberry Pi 4 - Raspbian Buster Docker version 19.03.12, build 48a6621 docker-compose version 1.26.2, build unknown

jonaswinkler commented 3 years ago

Seems I managed to break the ARM image again.

Bart1909 commented 3 years ago

Can we support in any way?

jonaswinkler commented 3 years ago

Well, figure out why sudo does not work. I'll do that now. Worst case, I'll revert updating debian dependencies. This release has newer versions of tesseract (from debian testing), and apparently sudo from the same testing repository does not work on ARM (no issues on x86_64 though).

Very few google hits. Tricky! :)

Firing up the RPi now.

jonaswinkler commented 3 years ago

Unable to reproduce this on the RPi, sadly.

Bart1909 commented 3 years ago

I can try to set up a new Docker-compose environment tomorrow to check, if it depends on the update maybe.

Bart1909 commented 3 years ago

Which Pi do you use? I‘m currently using Pi 4, I can try on a Pi 3B also tomorrow

jonaswinkler commented 3 years ago

Pi 3B. I don't have a Pi4. Is that on aarch64?

Bart1909 commented 3 years ago

Yes, this is running on aarch64. Maybe, this is the problem? I will try on my Pi 3B tomorrow

jonaswinkler commented 3 years ago

@starbuck93 - are you running on aarch64?

starbuck93 commented 3 years ago

uname -m outputs armv7l so I think it's 32bit on a Pi 4.

Bart1909 commented 3 years ago

Are you both running on Raspbian Buster?

starbuck93 commented 3 years ago

@Bart1909 Yes, it looks like Buster / debian 10 with the 5.10.17 kernel.

Bart1909 commented 3 years ago

I‘m also running on Buster, but with kernel 4.9.118

Bart1909 commented 3 years ago

Tested it with a fresh paperless-environment on Pi 4 with the same error message. Then I tested it on the Pi 3B+, but also with the same error message. The 3B+ is also running Buster with kernel 4.19.75

tobi-bo commented 3 years ago

I have the same issue on RPi4 with Raspbian: Linux raspberrypi 5.4.83-v7l+ #1379 SMP Mon Dec 14 13:11:54 GMT 2020 armv7l GNU/Linux Docker version 20.10.2, build 2291f61 docker-compose version 1.21.0

Bart1909 commented 3 years ago

@tobi-bo which Raspbian version do you use? also Buster?

tobi-bo commented 3 years ago

@Bart1909 Yes: PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debian

colinfrei commented 3 years ago

I have the same issue on a fresh install, but with a slightly different error message: Apply database migrations... sudo: error in event loop: Operation not permitted sudo: unexpected child termination condition: 0 Paperless-ng docker container starting... creating directory /tmp/paperless Apply database migrations... sudo: error in event loop: Operation not permitted sudo: unexpected child termination condition: 0 Paperless-ng docker container starting... creating directory /tmp/paperless

I'm also running on a raspberry pi with armv7l and a fresh installation of Buster, on a Pi 3.

Using 1.3.2 instead works.

jtseven commented 3 years ago

I have the same issue on a fresh install, but with a slightly different error message: Apply database migrations... sudo: error in event loop: Operation not permitted sudo: unexpected child termination condition: 0 Paperless-ng docker container starting... creating directory /tmp/paperless Apply database migrations... sudo: error in event loop: Operation not permitted sudo: unexpected child termination condition: 0 Paperless-ng docker container starting... creating directory /tmp/paperless

I'm also running on a raspberry pi with armv7l and a fresh installation of Buster, on a Pi 3.

Using 1.3.2 instead works.

I am getting the exact same error message since upgrading to 1.4.0.

tobi-bo commented 3 years ago

Using 1.3.2 instead works.

Works for me, too.

jonaswinkler commented 3 years ago

Pushed a change to the dev branch, please try with the dev version once the image has completed building (https://github.com/jonaswinkler/paperless-ng/actions/runs/735916687).

Bart1909 commented 3 years ago

Tried with the dev Tag, but same error message:

Creating network "paperless_default" with the default driver
Creating paperless_broker_1 ... done
Creating paperless_db_1     ... done
Creating paperless_webserver_1 ... done
Attaching to paperless_db_1, paperless_broker_1, paperless_webserver_1
db_1         | 
db_1         | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1         | 
broker_1     | 1:C 10 Apr 2021 14:10:17.053 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
broker_1     | 1:C 10 Apr 2021 14:10:17.053 # Redis version=6.0.12, bits=32, commit=00000000, modified=0, pid=1, just started
broker_1     | 1:C 10 Apr 2021 14:10:17.053 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
db_1         | 2021-04-10 14:10:17.051 UTC [1] LOG:  starting PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on arm-unknown-linux-gnueabi, compiled by gcc (Debian 8.3.0-6) 8.3.0, 32-bit
broker_1     | 1:M 10 Apr 2021 14:10:17.057 # Warning: 32 bit instance detected but no memory limit set. Setting 3 GB maxmemory limit with 'noeviction' policy now.
broker_1     | 1:M 10 Apr 2021 14:10:17.057 * Running mode=standalone, port=6379.
broker_1     | 1:M 10 Apr 2021 14:10:17.058 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
broker_1     | 1:M 10 Apr 2021 14:10:17.058 # Server initialized
broker_1     | 1:M 10 Apr 2021 14:10:17.058 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
db_1         | 2021-04-10 14:10:17.052 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
webserver_1  | Paperless-ng docker container starting...
broker_1     | 1:M 10 Apr 2021 14:10:17.059 * Ready to accept connections
db_1         | 2021-04-10 14:10:17.052 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1         | 2021-04-10 14:10:17.064 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
webserver_1  | creating directory /tmp/paperless
db_1         | 2021-04-10 14:10:17.074 UTC [25] LOG:  database system was shut down at 2021-04-10 14:05:28 UTC
db_1         | 2021-04-10 14:10:17.086 UTC [1] LOG:  database system is ready to accept connections
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
paperless_webserver_1 exited with code 1
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
paperless_webserver_1 exited with code 1
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
paperless_webserver_1 exited with code 1
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
paperless_webserver_1 exited with code 1
^CGracefully stopping... (press Ctrl+C again to force)
Stopping paperless_webserver_1 ... done
Stopping paperless_db_1        ... done
Stopping paperless_broker_1    ... done
Mannshoch commented 3 years ago

I first time Installed paperless on a raspberry pi with the script (I assume it would be 1.4).

Pulling broker    ... done
Pulling gotenberg ... done
Pulling tika      ... done
Pulling webserver ... done
Creating paperless_webserver_run ... done
Paperless-ng docker container starting...
creating directory /tmp/paperless
Apply database migrations...
sudo: error in event loop: Operation not permitted
sudo: unexpected child termination condition: 0
ERROR: 1
Mannshoch commented 3 years ago

I'm not familiar with paperless and docker. I installed paperless with the script on my Raspi. But is there an easy way to use 1.3.2 or test an actual version from dev?

jtseven commented 3 years ago

I am still getting the same error with the newest docker release 1.4.1 on a Raspberry Pi 3 Model B:

webserver_1  | Paperless-ng docker container starting...
webserver_1  | creating directory /tmp/paperless
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
stex commented 3 years ago

Same, the latest update broke the webserver. I'm also running it on a Raspberry Pi 4 in Docker.

$> docker-compose --version
docker-compose version 1.25.5, build unknown

$> docker --version
Docker version 20.10.6, build 370c289

Interestingly, the exit codes alternate

paperless-ng_webserver_1 exited with code 0
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
paperless-ng_webserver_1 exited with code 1
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
paperless-ng_webserver_1 exited with code 1
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
paperless-ng_webserver_1 exited with code 0
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
paperless-ng_webserver_1 exited with code 0
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
untcha commented 3 years ago

Same here...

webserver_1  | Paperless-ng docker container starting...
webserver_1  | creating directory /tmp/paperless
webserver_1  | sudo: error in event loop: Operation not permitted
webserver_1  | sudo: unexpected child termination condition: 0
kopfpolster commented 3 years ago

seems like i have the same issue:

webserver_1 | Paperless-ng docker container starting... webserver_1 | creating directory ../data/index webserver_1 | creating directory /tmp/paperless webserver_1 | sudo: error in event loop: Operation not permitted webserver_1 | sudo: unexpected child termination condition: 0

Operating System: Raspbian GNU/Linux 10 (buster) Kernel: Linux 5.10.17-v7l+ Architecture: arm RPi4 8gb

If i can provide any more informations to solve this issue, let me know

sitarane commented 3 years ago

I have this issue too. On 1.4.1.

Standing at the ready for test requests.

Bart1909 commented 3 years ago

Hey @jonaswinkler any new ideas, how to get the new version working on raspberries?

jonaswinkler commented 3 years ago

I have no idea what's causing this issue and google is not providing any useful insights. The image also works fine on my RPi 3b, which is armhf / 32bit.

Probably has something to do with updated binary dependencies in 1.4.0 (tesseract 4.1.0 and others).

Some of these updated dependencies were required in order to build the newest versions of some python dependencies on RPi, such as pikepdf. These contain important security fixes, so I can't just roll back these changes.

In order to properly fix this issue, I need a debian-based docker base image with at least tesseract 4.1, and python 3.7 (piwheels does not have 3.8 wheels or newer), and support for all three architectures. So far, I was unable to find a well maintained image that fits these criteria.

Bart1909 commented 3 years ago

Is your Pi running on Raspbian Buster @jonaswinkler ?

jonaswinkler commented 3 years ago

Yes.

Mannshoch commented 3 years ago

I searched in internet about a solution. I found the propose to add --user $(id -u):$(id -g) to the docker-compose command in the script. docker-compose run --user $(id -u):$(id -g) --rm -e DJANGO_SUPERUSER_PASSWORD="$PASSWORD" webserver createsuperuser --noinput --username "$USERNAME" --email "$EMAIL"

If I do that I get.

Creating paperless_webserver_run ... done
Paperless-ng docker container starting...
creating directory /tmp/paperless
paperless is not in the sudoers file.  This incident will be reported.
ERROR: 1

I get the same Error if I do docker-compose with sudo.

Raspberry Pi uname -a ⇒ Linux node-red 5.10.17-v7+ #1403 SMP Mon Feb 22 11:29:51 GMT 2021 armv7l GNU/Linux

jonaswinkler commented 3 years ago

That's because the entrypoint script has to be run as root to set up permissions.

WhiteHatTux commented 3 years ago

I think we are affected by this issue: https://docs.linuxserver.io/faq#libseccomp I ran the suggested manual update by deb package on my rpi 3B+ and now version 1.4.0 is working just fine.

In the process of debugging it i stumbled upon gosu which is used by postgresql in their docker image and from the description of the debian package it is a lot easier to configure and use if you just want to run as a different user without further issues.

Bart1909 commented 3 years ago

This seems to work @WhiteHatTux ! I've followed option 2 and started paperless 1.4.1 afterwards and it works. Thank you very much!

@jonaswinkler maybe you can add this hint to the documentation for new users?

tobi-bo commented 3 years ago

I think we are affected by this issue: https://docs.linuxserver.io/faq#libseccomp I ran the suggested manual update by deb package on my rpi 3B+ and now version 1.4.0 is working just fine.

I tried suggested Option 3 in that FAQ, and it's working for me. Running on 1.4.1 now, without any issues. Thanks @WhiteHatTux

jonaswinkler commented 3 years ago

Just queued 1.4.2, please report back if this works for you. The build takes about an hour.

stex commented 3 years ago

I tested 1.4.2 and got the following error now (I didn't perform the package update described in https://github.com/jonaswinkler/paperless-ng/issues/879#issuecomment-827215412 yet):

paperless-ng_webserver_1 exited with code 132
webserver_1  | Apply database migrations...
webserver_1  | Fatal Python error: _Py_InitializeMainInterpreter: can't initialize time
webserver_1  | PermissionError: [Errno 1] Operation not permitted
webserver_1  |
webserver_1  | Current thread 0xb6f48010 (most recent call first):
webserver_1  | /sbin/docker-prepare.sh: line 39:    20 Aborted                 (core dumped) python3 manage.py migrate
webserver_1  | Executing /usr/local/bin/supervisord -c /etc/supervisord.conf
webserver_1  | Fatal Python error: _Py_InitializeMainInterpreter: can't initialize time
webserver_1  | PermissionError: [Errno 1] Operation not permitted
webserver_1  |
webserver_1  | Current thread 0xb6faf010 (most recent call first):

Edit: I performed the package update linked by @WhiteHatTux now and 1.4.2 works as expected.

WhiteHatTux commented 3 years ago

@jonaswinkler I think I might not have been specific enough in my previous comment/the PR. The fix is the libseccomp installation. I came across gosu during the debugging, but the usage of gosu does not fix this issue. I created the pr #956, because I think it might be more stable for future versions, but it won't solve the problem of an old libseccomp, as the os we use inside the container simply needs the updated libseccomp. I am currently trying to use python:3.7-slim-stretch as that contains an older version of debian and might not yet need the updated host library.

WhiteHatTux commented 3 years ago

I tried with python:3.7-slim-stretch, as the problem is not supposed to affect older versions, but it doesn't work either. Problem is also known in moby and the solution really only seems to be an update: https://github.com/moby/moby/issues/40734#issuecomment-762493722 Sorry about the confusion with gosu. :-(

jonaswinkler commented 3 years ago

No worries. I've updated libseccomp2 in the docker image.

Please report back if the dev docker tag/version works. You'll need to wait until that's done building though, see github actions.

WhiteHatTux commented 3 years ago

I will give it a try, but I'm almost certain that it won't work, as the different sources state that the new libseccomp needs to be installed on the host and not inside the image.

sitarane commented 3 years ago

Tested 1.4.2, unfortunately I don't manage to get to the point I can test it. docker-compose up makes the webserver service do this:

webserver_1  | Paperless-ng docker container starting...
webserver_1  | Installing languages...
webserver_1  | Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
webserver_1  | Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
webserver_1  | Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
webserver_1  | Get:4 http://deb.debian.org/debian bullseye InRelease [146 kB]
webserver_1  | Err:1 http://security.debian.org/debian-security buster/updates InRelease
webserver_1  |   At least one invalid signature was encountered.
webserver_1  | Err:2 http://deb.debian.org/debian buster InRelease
webserver_1  |   At least one invalid signature was encountered.
webserver_1  | Err:3 http://deb.debian.org/debian buster-updates InRelease
webserver_1  |   At least one invalid signature was encountered.
webserver_1  | Err:4 http://deb.debian.org/debian bullseye InRelease
webserver_1  |   At least one invalid signature was encountered.
webserver_1  | Reading package lists...
webserver_1  | W: GPG error: http://security.debian.org/debian-security buster/updates InRelease: At least one invalid signature was encountered.
webserver_1  | E: The repository 'http://security.debian.org/debian-security buster/updates InRelease' is not signed.
webserver_1  | W: GPG error: http://deb.debian.org/debian buster InRelease: At least one invalid signature was encountered.
webserver_1  | E: The repository 'http://deb.debian.org/debian buster InRelease' is not signed.
webserver_1  | W: GPG error: http://deb.debian.org/debian buster-updates InRelease: At least one invalid signature was encountered.
webserver_1  | E: The repository 'http://deb.debian.org/debian buster-updates InRelease' is not signed.
webserver_1  | W: GPG error: http://deb.debian.org/debian bullseye InRelease: At least one invalid signature was encountered.
webserver_1  | E: The repository 'http://deb.debian.org/debian bullseye InRelease' is not signed.
webserver_1  | Paperless-ng docker container starting...

Reverted to 1.3.2 Anyone ever got that? Internet says my disk is full (it's not).

WhiteHatTux commented 3 years ago

@sitarane Which language did you set in PAPERLESS_OCR_LANGUAGES environment variable? This error is thrown during the installation of an alternative language for ocr. But I was not able to reproduce it. 

webserver_1  | Installing languages...                           
webserver_1  | Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
webserver_1  | Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
webserver_1  | Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]               
webserver_1  | Get:4 http://deb.debian.org/debian bullseye InRelease [146 kB]
webserver_1  | Get:5 http://security.debian.org/debian-security buster/updates/main armhf Packages [279 kB]
webserver_1  | Get:6 http://deb.debian.org/debian buster/main armhf Packages [7698 kB]
webserver_1  | Get:7 http://deb.debian.org/debian buster-updates/main armhf Packages [10.8 kB]
webserver_1  | Get:8 http://deb.debian.org/debian bullseye/main armhf Packages [7954 kB]
webserver_1  | Fetched 16.3 MB in 13s (1271 kB/s)                                                            
webserver_1  | Reading package lists...                                
webserver_1  | Installing package tesseract-ocr-por...

How are you running paperless? https://github.com/docker-library/php/issues/898#issuecomment-726981840 suggests that it might also be due to the space that was assigned to docker. Depending on the installation method of docker. If you have ensured that the space on disk is actually usable for docker, you could simply try again to see if the repository issues have been resolved and the installation runs through now. Or disable the alternative OCR language, although that kind of defeats the purpose of the whole document processing. :-(

Edit I just had to scroll down a little bit more: https://github.com/docker-library/php/issues/898#issuecomment-728917256 You are also affected by the libseccomp error. @sitarane, the problem should be solved once you update to the newest libseccomp version as explained here: https://docs.linuxserver.io/faq#libseccomp

WhiteHatTux commented 3 years ago

No worries. I've updated libseccomp2 in the docker image.

Please report back if the dev docker tag/version works. You'll need to wait until that's done building though, see github actions.

@jonaswinkler I tried the image for 1.4.2 and it doesn't work. Really the only thing I have found to work is the installation of the new libseccomp on the host. :-( I'm testing dev tag now. -> Same problem

WhiteHatTux commented 3 years ago

This appears to be bugging quite some people: https://github.com/debuerreotype/docker-debian-artifacts/issues/106 Always the same solution: update your docker daemon and update libseccomp on the host. Quote: Docker version 19.03.9 or newer libseccomp version 2.4.2 or newer