ffrsam-backend Keeps Restarting Every Second

[jonahyongNTU]

I have just installed FFRSAM on CentOS7. I copy/paste the code in "database/rsam.sql" to create the default rsam database.

Currently, "docker ps" command shows that ffrsam-frontend and ffrsam-database are up and running. However, ffrsam-backend is always in a state of restarting.

docker logs ffrsam-backend: time="2022-12-12T03:21:01Z" level=info msg="read crontab: /app/crontab" time="2022-12-12T03:21:01Z" level=fatal msg="open /app/crontab: permission denied" time="2022-12-12T03:21:02Z" level=info msg="read crontab: /app/crontab" time="2022-12-12T03:21:02Z" level=fatal msg="open /app/crontab: permission denied" time="2022-12-12T03:21:03Z" level=info msg="read crontab: /app/crontab" time="2022-12-12T03:21:03Z" level=fatal msg="open /app/crontab: permission denied" time="2022-12-12T03:21:05Z" level=info msg="read crontab: /app/crontab" time="2022-12-12T03:21:05Z" level=fatal msg="open /app/crontab: permission denied"

There is no such file "/app/crontab" on my server. I do not understand what ffrsam-backend is trying to do and why it keeps restarting.

Are you able to help me, please?

I have attached screenshot of my bash output.

Thanks, Jonah

[dlnorgaard]

The backend process will get the data from the wave server, calculate the rsam, store in db, and then generate some images. The /app/crontab dir would be in the container. You should be able to connect with docker exec -it ffrsam-backend /bin/bash and have a look around. Also, from outside the container, do docker logs -f ffrsam-backend and let me know what you see.

[jonahyongNTU]

Thanks, dlnorgaard,

I have stopped and then restarted the ffrsam and then did the "docker logs" command. It shows only the same two lines of error over and over again.

Also, the "docker exec" command says it cannot do anything while the ffrsam-backend is restarting. And since it is always restarting, I cannot do anything.

Here is a screenshot of the output of both the above commands.

[dlnorgaard]

Oh right. Comment out the last line in backend. Dockerfile and then start.

[jonahyongNTU]

Hi dlnorgaard,

The last line in "backend/back.Dockerfile" is: CMD [ "supercronic", "/app/crontab" ]

So, I commented that out by putting a hash at the beginning of the line.

Then I stopped ffrsam and started it. Still the same errors, though.

Here attached is a screenshot.

[dlnorgaard]

I can't see your docker up command but did you rebuild the backend container? Run with -b option.

[jonahyongNTU]

Oops, I did not realize I had to rebuild after making any changes.

I have now stopped the ffrsam and built it again.

But I am getting the same results. The backend is just restarting every second and docker exec -it cannot work because it is restarting.

[dlnorgaard]

Also comment out the section under "Add Tini". What user are you installing this as? What is the id of the user you are using (id <username>.) Since it's a permissions denied problem, it could be couple of things: 1) doesn't have permission to run /app/crontab or 2) can't access the script the crontab is supposed to run (/opt/ffrsam/run.sh). The latter is a volume mount (see docker-compose.yml line 31) and may carry over permissions from your host system. I am wondering if the user you installed it as does not have user id 2020 and possibly throwing off permissions. I don't have access to a Linux machine where I can test this out on at the moment so sorry it's a bit of a guessing game for me. Please check permissions on those two things and let me know how they look. I was directly involved in installation for other people in the past so may have missed some steps in the instructions about ownership for the cloned repo.

[jonahyongNTU]

Hi, dlnorgaard,

Happy New Year. Sorry for the delayed response. I had completely forgot about this after the December holidays.

01) So, I have commented out the section under "Add Tini" in the backend/back.Dockerfile", and have done a rebuild.

02) The username I am installing as is "ffrsam". Previously, it had a user id of "790". However, I have now changed it to "2020".

03) There is no such directory named "/app". Likewise, there is no file named "/app/crontab".

04) ffrsam user certainly has permissions to read and execute "/home/ffrsam/ffrsam/backend/src/run.sh"

05) The docker ps still shows that the backend is restarting repeatedly.

06) "docker exec -it ffrsam-backend /bin/bash" still complains that container is restarting.

07) "docker logs ffrsam-backend" outputs nothing

Any ideas? Also, we are going to migrate our server from CentOS7 to Rocky Linux8 later this year. Will the installation be easier on Rocky Linux 8?

Thanks, Jonah

[dlnorgaard]

Hi & happy new year to you too! I will try to stand this up later in the week to see if I can figure out what's going on.

[dlnorgaard]

Actually got to it today. I used WSL (Ubuntu 22.04) though since I don't have easy access to CentOS 7. I was able to bring everything up in a matter of minutes. After cloning the repo on the Windows side these were the only command I ran in WSL:

95 cd ffrsam 96 cp template.env .env 97 vi .env 98 vi .env 99 ls 100 mkdir data 101 mkdir images 102 chmod a+w images 103 cd backend/src 104 cp config_template.py config.py 105 vi config.py 106 cd ../.. 107 docker compose build 108 docker ps 109 docker compose up -d 110 docker ps

I did have to run dos2unix command on the backend scripts for the crons to work since I cloned in Windows, but this wouldn't apply to you.

All containers are up and no error in logs at the moment. The web frontend is also working.

I noticed your /home/ffrsam is under /mnt. What type of file system is being mounted? Btw, Is SELinux enabled? If so, try disabling.

[jonahyongNTU]

I have the following 2 lines in my fstab:

# THE DEFAULT HOME FOR LDAP USERS IS /mnt/home.  symlinks will cause SSH-KEYS to fail, so use bind instead.
/home /mnt/home none bind

This means "/home" and "/mnt/home" are the same. Both of them are the contents of /home.

The file system for "/home" is xfs.

SELinux is already disabled.

[dlnorgaard]

I'm unsure if user ffrsam has access to /app/crontab or not. If you comment out the USER ffrsam line, you will be able to look in the container as root and see if /app/crontab is there.

Another thing to try might be deleting the containers and images and rebuild. docker compose down docker container prune docker image prune

[jonahyongNTU]

Hi, Diana.

"docker container prune" and "docker image prune" seemed to delete lots of stuff. But the it's still the same problem after that. ffrsam-backend is still perpertually restarting.

After commenting out the "USER ffrsam" line and recompiling, it's still the same problem. After switching the user to root, I am unable to locate "/app/crontab". In fact, even "/app" does not exist. Or perhaps I am trying to access it the wrong way?

[dlnorgaard]

Try cat /app/crontab

[jonahyongNTU]

Hi Diana,

Here is the output from "cat /app/crontab":

[dlnorgaard]

What about ls /app? /app/crontab is copied from crontab.txt on line 64 of backend/back.Dockerfile. If it's not there it is not copying for some reason. Permissions issue on crontab.txt?

[jonahyongNTU]

The directory, "/app" did not exist, probably because it is not exist on CentOS by default.

So, I manually created an "/app" directory using the root account and changed its ownership to ffrsam.

The, I did a recompile and started the ffrsam again.

It's still the same problem. backend is always restarting itself. There are no files in "/app" directory.

[dlnorgaard]

The app directory should be within the container, which is running Ubuntu. Did you change the image that it is pulling from at the top of back.Dockerfile? FROM ubuntu:bionic

[jonahyongNTU]

Hmmm, I don't have a "container". Does container only apply to virtual machines? Our server is not a virtual machine. Also, we are running CentOS instead of Ubuntu.

Hmmm, "change the image". Does it refer to which repo to download from? Are you saying that we should change this line? RUN echo "deb http://deb.obspy.org bionic main" >> /etc/apt/sources.list

So, we should change "bionic" to something else? Is there a repo for CentOS?

[dlnorgaard]

When you run docker compose up -d, the app will create 3 containers: one for frontend web UI, one for backend python cron jobs, and one for the database to store the data. Each runs off of a base image. In the case of the backend one that is failing, it is using Ubuntu. So although your host machine is CentOS, the backend code is running in a containerized Ubuntu. You should not have to change anything in the Dockerfiles.

More info on docker containers: https://www.docker.com/resources/what-container/

I would suggest starting from scratch in a different directory.

Make sure everything is stopped (in ffrsam dir): docker compose down

To view containers: docker container ls

To remove unused containers (do above command again to make sure it removed all the containers): docker container prune

Then start over with no changes except as specified in instructions.