search

bookwyrm-social / bookwyrm

Social reading and reviewing, decentralized with ActivityPub
http://joinbookwyrm.com/
Other
2.24k stars 263 forks source link

Updates not consistent. #3168

Closed lhirlimann closed 9 months ago

lhirlimann commented 9 months ago

Describe the bug I don't see some of my updates neither on my homepage, neither on my feed. It's like both of these are frozen in the past. My Homepage looks like image

It gets some updates , like the 2023 reading banner. But below it the content is from August 20th Max. if you look at the right column you see the books I added today.

My book feed is also frozen in the past around the same date https://bookwyrm.lovetux.net/books#feed image

I would expect both to show my recent reads, notations and critic.

Instance On which BookWyrm instance did you encounter this problem.

Desktop (please complete the following information):

mouse-reeve commented 9 months ago

This sounds like a problem with celery - an admin should go to https://bookwyrm.lovetux.net/settings/celery and see if it’s running or has any issues (it may be helpful to post a screenshot). The fix may be as simple as restarting celery and or redid

solimanhindy commented 9 months ago

Hello @mouse-reeve and @lhirlimann, @lhirlimann is one of my user on this Bookwyrm Instance. As you said I restarted celery:

root@bookwyrm:~# systemctl restart bookwyrm-scheduler.service 
root@bookwyrm:~# systemctl status bookwyrm-scheduler.service 
● bookwyrm-scheduler.service - BookWyrm scheduler
     Loaded: loaded (/etc/systemd/system/bookwyrm-scheduler.service; enabled; preset: enabled)
     Active: active (running) since Mon 2023-12-25 21:01:27 CET; 1s ago
   Main PID: 258867 (celery)
      Tasks: 1 (limit: 1706)
     Memory: 128.3M
        CPU: 1.365s
     CGroup: /system.slice/bookwyrm-scheduler.service
             └─258867 /opt/bookwyrm/venv/bin/python3 /opt/bookwyrm/venv/bin/celery -A celerywyrm beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler

déc. 25 21:01:27 bookwyrm systemd[1]: Started bookwyrm-scheduler.service - BookWyrm scheduler.
root@bookwyrm:~#

And the log from celery after restarting:

déc. 25 21:01:27 bookwyrm systemd[1]: Stopping bookwyrm-scheduler.service - BookWyrm scheduler...
déc. 25 21:01:27 bookwyrm celery[258785]: celery beat v5.2.7 (dawn-chorus) is starting.
déc. 25 21:01:27 bookwyrm celery[258785]: __    -    ... __   -        _
déc. 25 21:01:27 bookwyrm celery[258785]: LocalTime -> 2023-12-25 19:58:44
déc. 25 21:01:27 bookwyrm celery[258785]: Configuration ->
déc. 25 21:01:27 bookwyrm celery[258785]:     . broker -> redis://localhost:6379/0
déc. 25 21:01:27 bookwyrm celery[258785]:     . loader -> celery.loaders.app.AppLoader
déc. 25 21:01:27 bookwyrm celery[258785]:     . scheduler -> django_celery_beat.schedulers.DatabaseScheduler
déc. 25 21:01:27 bookwyrm celery[258785]:     . logfile -> [stderr]@%INFO
déc. 25 21:01:27 bookwyrm celery[258785]:     . maxinterval -> 5.00 seconds (5s)
déc. 25 21:01:27 bookwyrm systemd[1]: bookwyrm-scheduler.service: Deactivated successfully.
déc. 25 21:01:27 bookwyrm systemd[1]: Stopped bookwyrm-scheduler.service - BookWyrm scheduler.
déc. 25 21:01:27 bookwyrm systemd[1]: bookwyrm-scheduler.service: Consumed 3.323s CPU time.
déc. 25 21:01:27 bookwyrm systemd[1]: Started bookwyrm-scheduler.service - BookWyrm scheduler.
déc. 25 21:01:29 bookwyrm celery[258867]: [2023-12-25 20:01:29,718: INFO/MainProcess] beat: Starting...

You can find enclosed a screenshot from the celery admin page. Screenshot 2023-12-25 at 21-02-50 BookWyrm de Lovetux net

Someting might missing on my side however I can't find it.

Thanks.

mouse-reeve commented 9 months ago

Are there workers visible under the list of tasks? It may be that redis isn’t picking up the tasks that are being created

dato commented 9 months ago

@solimanhindy Please note there is also bookwyrm-worker.service unit (which spawns the actual workers).

Could you restart that one as well, if you haven’t already? When healthy, logs for bookwyrm-worker.service ought to show tasks being processed.

solimanhindy commented 9 months ago

Hi @mouse-reeve and @dato thanks for your help. I did restart both bookwyrm-worker.service and bookwyrm-scheduler.service.

root@bookwyrm:~# systemctl restart bookwyrm-worker.service 
root@bookwyrm:~# systemctl status bookwyrm-worker.service 
● bookwyrm-worker.service - BookWyrm worker
     Loaded: loaded (/etc/systemd/system/bookwyrm-worker.service; enabled; preset: enabled)
     Active: active (running) since Tue 2023-12-26 20:37:46 CET; 4s ago
   Main PID: 30939 (celery)
      Tasks: 2 (limit: 1706)
     Memory: 115.2M
        CPU: 2.008s
     CGroup: /system.slice/bookwyrm-worker.service
             ├─30939 /opt/bookwyrm/venv/bin/python3 /opt/bookwyrm/venv/bin/celery -A celerywyrm worker -l info -Q high_priority,medium_priority,low_priority,imports,broadcast
             └─30945 /opt/bookwyrm/venv/bin/python3 /opt/bookwyrm/venv/bin/celery -A celerywyrm worker -l info -Q high_priority,medium_priority,low_priority,imports,broadcast

déc. 26 20:37:48 bookwyrm celery[30939]:   . bookwyrm.suggested_users.remove_suggestion_task
déc. 26 20:37:48 bookwyrm celery[30939]:   . bookwyrm.suggested_users.remove_user_task
déc. 26 20:37:48 bookwyrm celery[30939]:   . bookwyrm.suggested_users.rerank_suggestions_task
déc. 26 20:37:48 bookwyrm celery[30939]:   . bookwyrm.suggested_users.rerank_user_task
déc. 26 20:37:48 bookwyrm celery[30939]:   . bookwyrm.views.inbox.activity_task
déc. 26 20:37:48 bookwyrm celery[30939]:   . imagekit.cachefiles.backends._generate_file
déc. 26 20:37:48 bookwyrm celery[30939]: [2023-12-26 19:37:48,124: INFO/MainProcess] Connected to redis://localhost:6379/0
déc. 26 20:37:48 bookwyrm celery[30939]: [2023-12-26 19:37:48,134: INFO/MainProcess] mingle: searching for neighbors
déc. 26 20:37:49 bookwyrm celery[30939]: [2023-12-26 19:37:49,143: INFO/MainProcess] mingle: all alone
déc. 26 20:37:49 bookwyrm celery[30939]: [2023-12-26 19:37:49,159: INFO/MainProcess] celery@bookwyrm ready.
root@bookwyrm:~#

root@bookwyrm:~# systemctl restart bookwyrm-scheduler.service 
root@bookwyrm:~# systemctl status bookwyrm-scheduler.service 
● bookwyrm-scheduler.service - BookWyrm scheduler
     Loaded: loaded (/etc/systemd/system/bookwyrm-scheduler.service; enabled; preset: enabled)
     Active: active (running) since Tue 2023-12-26 20:47:18 CET; 4s ago
   Main PID: 31141 (celery)
      Tasks: 1 (limit: 1706)
     Memory: 99.5M
        CPU: 1.773s
     CGroup: /system.slice/bookwyrm-scheduler.service
             └─31141 /opt/bookwyrm/venv/bin/python3 /opt/bookwyrm/venv/bin/celery -A celerywyrm beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler

déc. 26 20:47:18 bookwyrm systemd[1]: Started bookwyrm-scheduler.service - BookWyrm scheduler.
déc. 26 20:47:20 bookwyrm celery[31141]: [2023-12-26 19:47:20,266: INFO/MainProcess] beat: Starting...
root@bookwyrm:~#
`̀`

There is a worker connected see screenshot. 
![Screenshot 2023-12-26 at 20-42-37 BookWyrm de Lovetux net](https://github.com/bookwyrm-social/bookwyrm/assets/11487285/c959b59e-93e6-4c9e-8b87-baa00d8b51b3)

Are there other processes or log I could share with you?

Thanks!
marcusyoung commented 9 months ago

Are you dockerless by any chance? This looks like the new worker queues are not set up in the service. These were introduced an update or so ago. You would need to first stop the services(s), then copy the new service file(s):

cp contrib/systemd/*.service /etc/systemd/system/

and then

sudo systemctl daemon-reload

then start the services.

solimanhindy commented 9 months ago

Hi @marcusyoung, @dato and @mouse-reeve thank you so much for your help and expertise. Indeed I'm running a dockerless Bookwyrm instance.

@marcusyoung I followed what you said:

cp contrib/systemd/*.service /etc/systemd/system/

And

sudo systemctl daemon-reload

I restarted all the services and everything is back to normal! You Guys made my day!

@lhirlimann sorry about this mistake on my side and thanks for opening this issue.

As far as I am concerned we can close this issue.

Regards,