search

IgnisDa / ryot

Roll your own tracker!
https://ryot.io
GNU General Public License v3.0
1.94k stars 52 forks source link

High CPU usage #440

Closed ellsclytn closed 8 months ago

ellsclytn commented 1 year ago

I've noticed that my Ryot container enters a state of high CPU usage daily.

image

The above shows CPU usage over a 48h period. It seems to recover temporarily around 00:00, which leads me to believe it's something happening within one of the daily background jobs, but I'm just not sure which one as of yet.

The biggest spike seems to happen around 03:23-03:26.

image

I looked at the logs (where I've set RUST_LOG=ryot=trace,sea_orm=debug), but there doesn't seem to be much to look at beyond a lot of SQL queries. 

2023-10-25T03:23:39.161607Z DEBUG sea_orm::driver::sqlx_postgres: SELECT "collection_to_entity"."id", "collection_to_entity"."last_updated_on", "collection_to_entity"."collection_id", "collection_to_entity"."metadata_id", "collection_to_entity"."person_id", "collection_to_entity"."metadata_group_id", "collection_to_entity"."exercise_id" FROM "collection_to_entity" WHERE "collection_to_entity"."collection_id" = 2 AND "collection_to_entity"."metadata_id" = 479 LIMIT 1
2023-10-25T03:23:39.161894Z DEBUG sea_orm::driver::sqlx_postgres: UPDATE "collection_to_entity" SET "last_updated_on" = '2023-10-25 03:23:39 +00:00' WHERE "collection_to_entity"."id" = 10 RETURNING "id", "last_updated_on", "collection_id", "metadata_id", "person_id", "metadata_group_id", "exercise_id"
2023-10-25T03:23:39.163085Z DEBUG sea_orm::driver::sqlx_postgres: SELECT "user_to_entity"."id", "user_to_entity"."last_updated_on", "user_to_entity"."user_id", "user_to_entity"."num_times_interacted", "user_to_entity"."metadata_id", "user_to_entity"."exercise_id", "user_to_entity"."metadata_monitored", "user_to_entity"."metadata_reminder", "user_to_entity"."exercise_extra_information" FROM "user_to_entity" WHERE "user_to_entity"."user_id" = 1 AND "user_to_entity"."metadata_id" = 479 LIMIT 1
2023-10-25T03:23:39.164215Z TRACE ryot::background: Job: "AfterMediaSeen", Time Taken: 28ms, Successful = true
2023-10-25T04:00:00.003823Z TRACE ryot::background: Getting data from yanked integrations for all users
2023-10-25T04:00:00.004961Z DEBUG sea_orm::driver::sqlx_postgres: SELECT "user"."id" FROM "user" WHERE "user"."yank_integrations" IS NOT NULL

This is the last few lines of logs between 2023-10-25T03:23:00 to 2023-10-25T04:26:00.

I'm wondering if anyone is/has experienced similar, or might have some further leads I could explore.

IgnisDa commented 8 months ago

Try setting env var RUST_LOG=ryot=debug and try to regenerate summary.

ellsclytn commented 8 months ago

I realised I've been running an older version of Postgres. I've upgraded to Postgres 16. I have debug logs enabled on Ryot now, but manually kicking off a summary regeneration completed within seconds. I'll keep an eye on it tomorrow after the regular cron jobs start and report the results.

ellsclytn commented 8 months ago

Unfortunately it has continued even after the Postgres upgrade. Once the midnight jobs started CPU returned to ~10% and stayed there. Then when the next midnight rolled around it roughly doubled CPU usage.

I'd post the logs except that they're quite big. 120MB for the debug logs in total. ~500KB if I exclude any lines mentioning "Progress update". I'll wait for #640 to release and try a few things with it.

IgnisDa commented 8 months ago

@ellsclytn I released it as v4.2.5.

diogovalentte commented 8 months ago

I'm suffering from the same issue, at 01h25 in my timezone (UTC-3), the CPU usage of the Ryot container greatly increases. The graph below is my CPU usage:

image

This CPU usage continues for days. If I restart the container, it goes back to normal until the next 01h25.

ellsclytn commented 8 months ago

I tried removing the number of monitored shows. There were more than I realised, but not many. It's now down to three. That's certainly helped, but it does appear the problem persists (and gets consistently worse with a higher monitoring count). The CPU usage seems to "stack" each time the midnight cron job fires off.

I was able to confirm that setting DISABLE_UPDATE_WATCHLIST_MEDIA=true worked around the issue for me.

As for logs, I haven't seen a great deal that jumped out at me. The only thing that seemed a little different now was that some shows have quite a large log output which I think coincided with some more recent changes where the webhook path changed for the Jellyfin integration. Example of large output:

[backend] 2024-02-21T08:03:44.392967Z DEBUG ryot::miscellaneous::resolver: Progress update = Model { id: 10877, progress: 100, started_on: Some(2024-02-21), finished_on: Some(2024-02-21), user_id: 1, metadata_id: 479, state: Completed, updated_at: [2024-02-21T07:51:28.102148Z, 2024-02-21T07:51:29.076844Z, 2024-02-21T07:51:29.082814Z, 2024-02-21T07:51:30.083490Z, 2024-02-21T07:51:31.077929Z, 2024-02-21T07:51:32.057250Z, 2024-02-21T07:51:33.076630Z, 2024-02-21T07:51:34.072888Z, 2024-02-21T07:51:35.063937Z, 2024-02-21T07:51:36.078455Z, 2024-02-21T07:51:37.068737Z, 2024-02-21T07:51:38.082010Z, 2024-02-21T07:51:39.083784Z, 2024-02-21T07:51:40.088666Z, 2024-02-21T07:51:41.068606Z, 2024-02-21T07:51:42.077457Z, 2024-02-21T07:51:43.078821Z, 2024-02-21T07:51:44.079512Z, 2024-02-21T07:51:45.083775Z, 2024-02-21T07:51:46.072185Z, 2024-02-21T07:51:47.098788Z, 2024-02-21T07:51:48.101506Z, 2024-02-21T07:51:49.095454Z, 2024-02-21T07:51:50.084703Z, 2024-02-21T07:51:51.090321Z, 2024-02-21T07:51:52.091057Z, 2024-02-21T07:51:53.074288Z, 2024-02-21T07:51:54.083231Z, 2024-02-21T07:51:55.083420Z, 2024-02-21T07:51:56.069872Z, 2024-02-21T07:51:57.076592Z, 2024-02-21T07:51:58.077776Z, 2024-02-21T07:51:59.080866Z, 2024-02-21T07:52:00.086636Z, 2024-02-21T07:52:01.078323Z, 2024-02-21T07:52:02.080358Z, 2024-02-21T07:52:03.086206Z, 2024-02-21T07:52:04.086972Z, 2024-02-21T07:52:05.074869Z, 2024-02-21T07:52:06.087955Z, 2024-02-21T07:52:07.088170Z, 2024-02-21T07:52:08.082761Z, 2024-02-21T07:52:08.090154Z, 2024-02-21T07:52:09.087242Z, 2024-02-21T07:52:10.087337Z, 2024-02-21T07:52:11.072028Z, 2024-02-21T07:52:12.078831Z, 2024-02-21T07:52:13.079213Z, 2024-02-21T07:52:14.094454Z, 2024-02-21T07:52:15.099771Z, 2024-02-21T07:52:16.097773Z, 2024-02-21T07:52:17.076009Z, 2024-02-21T07:52:18.082507Z, 2024-02-21T07:52:19.083272Z, 2024-02-21T07:52:20.075004Z, 2024-02-21T07:52:21.083971Z, 2024-02-21T07:52:22.083075Z, 2024-02-21T07:52:23.066285Z, 2024-02-21T07:52:24.079564Z, 2024-02-21T07:52:25.080869Z, 2024-02-21T07:52:26.076148Z, 2024-02-21T07:52:27.088608Z, 2024-02-21T07:52:28.087441Z, 2024-02-21T07:52:29.090941Z, 2024-02-21T07:52:30.097319Z, 2024-02-21T07:52:31.085445Z, 2024-02-21T07:52:32.077071Z, 2024-02-21T07:52:33.084351Z, 2024-02-21T07:52:34.084610Z, 2024-02-21T07:52:35.081798Z, 2024-02-21T07:52:35.088777Z, 2024-02-21T07:52:36.088788Z, 2024-02-21T07:52:37.088170Z, 2024-02-21T07:52:38.113339Z, 2024-02-21T07:52:39.121880Z, 2024-02-21T07:52:40.120491Z, 2024-02-21T07:52:41.086818Z, 2024-02-21T07:52:42.096522Z, 2024-02-21T07:52:43.097243Z, 2024-02-21T07:52:44.087029Z, 2024-02-21T07:52:45.090137Z, 2024-02-21T07:52:46.081054Z, 2024-02-21T07:52:47.076876Z, 2024-02-21T07:52:48.089526Z, 2024-02-21T07:52:49.086563Z, 2024-02-21T07:52:50.081325Z, 2024-02-21T07:52:50.087416Z, 2024-02-21T07:52:51.086818Z, 2024-02-21T07:52:52.086535Z, 2024-02-21T07:52:53.093160Z, 2024-02-21T07:52:54.100368Z, 2024-02-21T07:52:55.099127Z, 2024-02-21T07:52:56.088632Z, 2024-02-21T07:52:57.097166Z, 2024-02-21T07:52:58.095666Z, 2024-02-21T07:52:59.083037Z, 2024-02-21T07:53:00.093291Z, 2024-02-21T07:53:01.081068Z, 2024-02-21T07:53:02.073270Z, 2024-02-21T07:53:03.085128Z, 2024-02-21T07:53:04.081277Z, 2024-02-21T07:53:05.094496Z, 2024-02-21T07:53:06.097820Z, 2024-02-21T07:53:07.098751Z, 2024-02-21T07:53:08.086611Z, 2024-02-21T07:53:09.098329Z, 2024-02-21T07:53:10.099019Z, 2024-02-21T07:53:11.086693Z, 2024-02-21T07:53:12.095324Z, 2024-02-21T07:53:13.125020Z, 2024-02-21T07:53:14.104731Z, 2024-02-21T07:53:14.126407Z, 2024-02-21T07:53:15.112457Z, 2024-02-21T07:53:16.100604Z, 2024-02-21T07:53:17.094569Z, 2024-02-21T07:53:18.099810Z, 2024-02-21T07:53:19.095504Z, 2024-02-21T07:53:20.100749Z, 2024-02-21T07:53:21.100462Z, 2024-02-21T07:53:22.100353Z, 2024-02-21T07:53:23.096486Z, 2024-02-21T07:53:23.101847Z, 2024-02-21T07:53:24.102366Z, 2024-02-21T07:53:25.102517Z, 2024-02-21T07:53:26.092127Z, 2024-02-21T07:53:27.099336Z, 2024-02-21T07:53:28.099684Z, 2024-02-21T07:53:29.106239Z, 2024-02-21T07:53:30.111700Z, 2024-02-21T07:53:31.100430Z, 2024-02-21T07:53:32.096400Z, 2024-02-21T07:53:33.102881Z, 2024-02-21T07:53:34.102642Z, 2024-02-21T07:53:35.105674Z, 2024-02-21T07:53:36.109173Z, 2024-02-21T07:53:37.107880Z, 2024-02-21T07:53:38.090264Z, 2024-02-21T07:53:39.104922Z, 2024-02-21T07:53:40.105080Z, 2024-02-21T07:53:41.095237Z, 2024-02-21T07:53:42.109958Z, 2024-02-21T07:53:43.107382Z, 2024-02-21T07:53:44.098979Z, 2024-02-21T07:53:45.110082Z, 2024-02-21T07:53:46.100274Z, 2024-02-21T07:53:47.099741Z, 2024-02-21T07:53:47.105273Z, 2024-02-21T07:53:48.105794Z, 2024-02-21T07:53:49.106203Z, 2024-02-21T07:53:50.095001Z, 2024-02-21T07:53:51.104611Z, 2024-02-21T07:53:52.102755Z, 2024-02-21T07:53:53.108729Z, 2024-02-21T07:53:54.114257Z, 2024-02-21T07:53:55.115604Z, 2024-02-21T07:53:56.101156Z, 2024-02-21T07:53:57.107531Z, 2024-02-21T07:53:58.108490Z, 2024-02-21T07:53:59.100859Z, 2024-02-21T07:54:00.109248Z, 2024-02-21T07:54:01.099810Z, 2024-02-21T07:54:02.109180Z, 2024-02-21T07:54:03.109076Z, 2024-02-21T07:54:04.111597Z, 2024-02-21T07:54:05.119111Z, 2024-02-21T07:54:06.124456Z, 2024-02-21T07:54:07.123606Z, 2024-02-21T07:54:08.093592Z, 2024-02-21T07:54:09.108363Z, 2024-02-21T07:54:10.109093Z, 2024-02-21T07:54:11.103030Z, 2024-02-21T07:54:11.109669Z, 2024-02-21T07:54:12.114131Z, 2024-02-21T07:54:13.100515Z, 2024-02-21T07:54:14.125554Z, 2024-02-21T07:54:15.131314Z, 2024-02-21T07:54:16.124546Z, 2024-02-21T07:54:17.104544Z, 2024-02-21T07:54:18.103453Z, 2024-02-21T07:54:19.114375Z, 2024-02-21T07:54:20.104054Z, 2024-02-21T07:54:21.113895Z, 2024-02-21T07:54:22.113874Z, 2024-02-21T07:54:23.119047Z, 2024-02-21T07:54:24.122575Z, 2024-02-21T07:54:25.124312Z, 2024-02-21T07:54:26.104330Z, 2024-02-21T07:54:27.110970Z, 2024-02-21T07:54:28.112567Z, 2024-02-21T07:54:29.114677Z, 2024-02-21T07:54:30.119563Z, 2024-02-21T07:54:31.108220Z, 2024-02-21T07:54:32.109399Z, 2024-02-21T07:54:33.113924Z, 2024-02-21T07:54:34.115213Z, 2024-02-21T07:54:35.117479Z, 2024-02-21T07:54:36.119892Z, 2024-02-21T07:54:37.118099Z, 2024-02-21T07:54:38.121913Z, 2024-02-21T07:54:39.120449Z, 2024-02-21T07:54:40.120100Z, 2024-02-21T07:54:41.107411Z, 2024-02-21T07:54:42.117291Z, 2024-02-21T07:54:43.117533Z, 2024-02-21T07:54:44.097872Z, 2024-02-21T07:54:45.108627Z, 2024-02-21T07:54:46.096928Z, 2024-02-21T07:54:47.093050Z, 2024-02-21T07:54:48.107771Z, 2024-02-21T07:54:49.108457Z, 2024-02-21T07:54:50.121509Z, 2024-02-21T07:54:51.124720Z, 2024-02-21T07:54:52.129748Z, 2024-02-21T07:54:53.106457Z, 2024-02-21T07:54:54.121895Z, 2024-02-21T07:54:55.118525Z, 2024-02-21T07:54:56.106250Z, 2024-02-21T07:54:57.121Z, 2024-02-21T07:54:58.119194Z, 2024-02-21T07:54:59.120866Z, 2024-02-21T07:55:00.125038Z, 2024-02-21T07:55:01.115063Z, 2024-02-21T07:55:02.127989Z, 2024-02-21T07:55:03.131691Z, 2024-02-21T07:55:04.122076Z, 2024-02-21T07:55:05.115644Z, 2024-02-21T07:55:06.113814Z, 2024-02-21T07:55:07.123356Z, 2024-02-21T07:55:08.131397Z, 2024-02-21T07:55:09.137045Z, 2024-02-21T07:55:10.136148Z, 2024-02-21T07:55:11.116573Z, 2024-02-21T07:55:12.124050Z, 2024-02-21T07:55:13.150396Z, 2024-02-21T07:55:14.142895Z, 2024-02-21T07:55:14.163561Z, 2024-02-21T07:55:15.151438Z, 2024-02-21T07:55:16.149553Z, 2024-02-21T07:55:17.118024Z, 2024-02-21T07:55:18.123997Z, 2024-02-21T07:55:19.121363Z, 2024-02-21T07:55:20.117542Z, 2024-02-21T07:55:21.117461Z, 2024-02-21T07:55:22.125604Z, 2024-02-21T07:55:23.131670Z, 2024-02-21T07:55:24.134450Z, 2024-02-21T07:55:25.134847Z, 2024-02-21T07:55:26.106049Z, 2024-02-21T07:55:27.119884Z, 2024-02-21T07:55:28.121282Z, 2024-02-21T07:55:29.131315Z, 2024-02-21T07:55:30.137290Z, 2024-02-21T07:55:31.126174Z, 2024-02-21T07:55:32.137006Z, 2024-02-21T07:55:33.141823Z, 2024-02-21T07:55:34.140804Z, 2024-02-21T07:55:35.118676Z, 2024-02-21T07:55:36.116744Z, 2024-02-21T07:55:37.127983Z, 2024-02-21T07:55:38.119534Z, 2024-02-21T07:55:39.128560Z, 2024-02-21T07:55:40.129013Z, 2024-02-21T07:55:41.120286Z, 2024-02-21T07:55:41.127493Z, 2024-02-21T07:55:42.126967Z, 2024-02-21T07:55:43.127991Z, 2024-02-21T07:55:44.140047Z, 2024-02-21T07:55:45.148597Z, 2024-02-21T07:55:46.143430Z, 2024-02-21T07:55:47.130758Z, 2024-02-21T07:55:48.139620Z, 2024-02-21T07:55:48.776280Z, 2024-02-21T07:56:03.782048Z, 2024-02-21T07:56:18.792306Z, 2024-02-21T07:56:33.788010Z, 2024-02-21T07:56:48.789741Z, 2024-02-21T07:57:03.791718Z, 2024-02-21T07:57:18.898643Z, 2024-02-21T07:57:24.195733Z, 2024-02-21T07:57:25.203801Z, 2024-02-21T07:57:26.204450Z, 2024-02-21T07:57:27.181800Z, 2024-02-21T07:57:28.194524Z, 2024-02-21T07:57:29.195198Z, 2024-02-21T07:57:30.237567Z, 2024-02-21T07:57:30.257977Z, 2024-02-21T07:57:31.234567Z, 2024-02-21T07:57:32.242482Z, 2024-02-21T07:57:33.229576Z, 2024-02-21T07:57:34.234987Z, 2024-02-21T07:57:35.235967Z, 2024-02-21T07:57:36.241036Z, 2024-02-21T07:57:37.245341Z, 2024-02-21T07:57:38.243356Z, 2024-02-21T07:57:39.246586Z, 2024-02-21T07:57:40.252569Z, 2024-02-21T07:57:41.252245Z, 2024-02-21T07:57:42.239014Z, 2024-02-21T07:57:43.248883Z, 2024-02-21T07:57:44.249901Z, 2024-02-21T07:57:45.243766Z, 2024-02-21T07:57:45.251250Z, 2024-02-21T07:57:46.242792Z, 2024-02-21T07:57:47.251818Z, 2024-02-21T07:57:48.240697Z, 2024-02-21T07:57:49.240845Z, 2024-02-21T07:57:50.249388Z, 2024-02-21T07:57:51.235601Z, 2024-02-21T07:57:52.243122Z, 2024-02-21T07:57:53.243766Z, 2024-02-21T07:57:54.250887Z, 2024-02-21T07:57:55.253777Z, 2024-02-21T07:57:56.251069Z, 2024-02-21T07:57:57.242929Z, 2024-02-21T07:57:58.251355Z, 2024-02-21T07:57:59.256175Z, 2024-02-21T07:58:00.236896Z, 2024-02-21T07:58:01.234858Z, 2024-02-21T07:58:02.243375Z, 2024-02-21T07:58:03.241424Z, 2024-02-21T07:58:03.246978Z, 2024-02-21T07:58:04.247844Z, 2024-02-21T07:58:05.246505Z, 2024-02-21T07:58:06.224565Z, 2024-02-21T07:58:07.235633Z, 2024-02-21T07:58:08.236146Z, 2024-02-21T07:58:09.250353Z, 2024-02-21T07:58:10.253152Z, 2024-02-21T07:58:11.251749Z, 2024-02-21T07:58:12.240191Z, 2024-02-21T07:58:13.249597Z, 2024-02-21T07:58:14.250060Z, 2024-02-21T07:58:15.249855Z, 2024-02-21T07:58:16.246819Z, 2024-02-21T07:58:17.255543Z, 2024-02-21T07:58:18.288947Z, 2024-02-21T07:58:19.291601Z, 2024-02-21T07:58:20.291495Z, 2024-02-21T07:58:21.255760Z, 2024-02-21T07:58:22.264299Z, 2024-02-21T07:58:23.264103Z, 2024-02-21T07:58:24.250704Z, 2024-02-21T07:58:25.260754Z, 2024-02-21T07:58:26.261345Z, 2024-02-21T07:58:27.250729Z, 2024-02-21T07:58:28.262581Z, 2024-02-21T07:58:29.261231Z, 2024-02-21T07:58:30.244972Z, 2024-02-21T07:58:31.246492Z, 2024-02-21T07:58:32.254347Z, 2024-02-21T07:58:33.261996Z, 2024-02-21T07:58:34.263890Z, 2024-02-21T07:58:35.262800Z, 2024-02-21T07:58:36.252209Z, 2024-02-21T07:58:37.258816Z, 2024-02-21T07:58:38.261397Z, 2024-02-21T07:58:39.255535Z, 2024-02-21T07:58:39.264775Z, 2024-02-21T07:58:40.262990Z, 2024-02-21T07:58:41.263709Z, 2024-02-21T07:58:42.254733Z, 2024-02-21T07:58:43.264564Z, 2024-02-21T07:58:44.266092Z, 2024-02-21T07:58:45.248007Z, 2024-02-21T07:58:46.255545Z, 2024-02-21T07:58:47.255912Z, 2024-02-21T07:58:48.247414Z, 2024-02-21T07:58:49.257003Z, 2024-02-21T07:58:50.258648Z, 2024-02-21T07:58:51.256348Z, 2024-02-21T07:58:52.263374Z, 2024-02-21T07:58:53.263479Z, 2024-02-21T07:58:54.266760Z, 2024-02-21T07:58:55.272424Z, 2024-02-21T07:58:56.271040Z, 2024-02-21T07:58:57.252297Z, 2024-02-21T07:58:58.264532Z, 2024-02-21T07:58:59.262287Z, 2024-02-21T07:59:00.273100Z, 2024-02-21T07:59:01.268727Z, 2024-02-21T07:59:02.280595Z, 2024-02-21T07:59:03.260433Z, 2024-02-21T07:59:04.266471Z, 2024-02-21T07:59:05.266941Z, 2024-02-21T07:59:06.268796Z, 2024-02-21T07:59:07.272228Z, 2024-02-21T07:59:08.272266Z, 2024-02-21T07:59:09.260737Z, 2024-02-21T07:59:10.269065Z, 2024-02-21T07:59:11.269332Z, 2024-02-21T07:59:12.263741Z, 2024-02-21T07:59:12.270187Z, 2024-02-21T07:59:13.273922Z, 2024-02-21T07:59:14.271657Z, 2024-02-21T07:59:15.253961Z, 2024-02-21T07:59:16.257856Z, 2024-02-21T07:59:17.268526Z, 2024-02-21T07:59:18.267182Z, 2024-02-21T07:59:19.294126Z, 2024-02-21T07:59:20.270704Z, 2024-02-21T07:59:21.276772Z, 2024-02-21T07:59:21.298943Z, 2024-02-21T07:59:22.284047Z, 2024-02-21T07:59:23.282334Z, 2024-02-21T07:59:24.265932Z, 2024-02-21T07:59:25.270905Z, 2024-02-21T07:59:26.270963Z, 2024-02-21T07:59:27.277282Z, 2024-02-21T07:59:28.284275Z, 2024-02-21T07:59:29.283685Z, 2024-02-21T07:59:30.249422Z, 2024-02-21T07:59:31.254651Z, 2024-02-21T07:59:32.258356Z, 2024-02-21T07:59:33.280537Z, 2024-02-21T07:59:34.283695Z, 2024-02-21T07:59:35.273125Z, 2024-02-21T07:59:36.263693Z, 2024-02-21T07:59:37.270818Z, 2024-02-21T07:59:38.270196Z, 2024-02-21T07:59:39.274114Z, 2024-02-21T07:59:40.265921Z, 2024-02-21T07:59:41.275628Z, 2024-02-21T07:59:42.264874Z, 2024-02-21T07:59:43.272402Z, 2024-02-21T07:59:44.270144Z, 2024-02-21T07:59:45.269557Z, 2024-02-21T07:59:46.283514Z, 2024-02-21T07:59:47.284941Z, 2024-02-21T07:59:48.270127Z, 2024-02-21T07:59:49.278477Z, 2024-02-21T07:59:50.267712Z, 2024-02-21T07:59:51.260208Z, 2024-02-21T07:59:52.267829Z, 2024-02-21T07:59:53.267996Z, 2024-02-21T07:59:54.277022Z, 2024-02-21T07:59:55.281830Z, 2024-02-21T07:59:56.280459Z, 2024-02-21T07:59:57.260705Z, 2024-02-21T07:59:58.277002Z, 2024-02-21T07:59:59.277748Z, 2024-02-21T08:00:00.268039Z, 2024-02-21T08:00:01.267973Z, 2024-02-21T08:00:02.274423Z, 2024-02-21T08:00:03.284237Z, 2024-02-21T08:00:04.288070Z, 2024-02-21T08:00:05.286527Z, 2024-02-21T08:00:06.286187Z, 2024-02-21T08:00:07.291022Z, 2024-02-21T08:00:08.292858Z, 2024-02-21T08:00:09.277284Z, 2024-02-21T08:00:10.285274Z, 2024-02-21T08:00:11.284851Z, 2024-02-21T08:00:12.270250Z, 2024-02-21T08:00:13.277903Z, 2024-02-21T08:00:14.274778Z, 2024-02-21T08:00:15.286739Z, 2024-02-21T08:00:16.293985Z, 2024-02-21T08:00:17.290689Z, 2024-02-21T08:00:18.276563Z, 2024-02-21T08:00:19.285817Z, 2024-02-21T08:00:20.286192Z, 2024-02-21T08:00:21.288304Z, 2024-02-21T08:00:22.290991Z, 2024-02-21T08:00:23.291920Z, 2024-02-21T08:00:24.282009Z, 2024-02-21T08:00:25.288901Z, 2024-02-21T08:00:26.289074Z, 2024-02-21T08:00:27.289957Z, 2024-02-21T08:00:28.294428Z, 2024-02-21T08:00:29.292258Z, 2024-02-21T08:00:30.275069Z, 2024-02-21T08:00:31.282083Z, 2024-02-21T08:00:32.284626Z, 2024-02-21T08:00:33.275830Z, 2024-02-21T08:00:34.288637Z, 2024-02-21T08:00:35.287189Z, 2024-02-21T08:00:36.290997Z, 2024-02-21T08:00:37.292564Z, 2024-02-21T08:00:38.291384Z, 2024-02-21T08:00:39.287308Z, 2024-02-21T08:00:39.295186Z, 2024-02-21T08:00:40.292077Z, 2024-02-21T08:00:41.292565Z, 2024-02-21T08:00:42.278611Z, 2024-02-21T08:00:43.285663Z, 2024-02-21T08:00:44.282690Z, 2024-02-21T08:00:45.284773Z, 2024-02-21T08:00:46.286492Z, 2024-02-21T08:00:47.288018Z, 2024-02-21T08:00:48.292832Z, 2024-02-21T08:00:49.291569Z, 2024-02-21T08:00:50.296654Z, 2024-02-21T08:00:51.281858Z, 2024-02-21T08:00:52.283830Z, 2024-02-21T08:00:53.286848Z, 2024-02-21T08:00:54.297129Z, 2024-02-21T08:00:55.303697Z, 2024-02-21T08:00:56.304277Z, 2024-02-21T08:00:57.284579Z, 2024-02-21T08:00:58.292739Z, 2024-02-21T08:00:59.293651Z, 2024-02-21T08:01:00.285164Z, 2024-02-21T08:01:01.291153Z, 2024-02-21T08:01:02.290430Z, 2024-02-21T08:01:03.292161Z, 2024-02-21T08:01:04.291251Z, 2024-02-21T08:01:05.292253Z, 2024-02-21T08:01:06.294871Z, 2024-02-21T08:01:07.297013Z, 2024-02-21T08:01:08.295646Z, 2024-02-21T08:01:09.299401Z, 2024-02-21T08:01:10.300152Z, 2024-02-21T08:01:11.299391Z, 2024-02-21T08:01:12.299256Z, 2024-02-21T08:01:13.299627Z, 2024-02-21T08:01:14.299285Z, 2024-02-21T08:01:15.300779Z, 2024-02-21T08:01:16.299526Z, 2024-02-21T08:01:17.300116Z, 2024-02-21T08:01:18.302746Z, 2024-02-21T08:01:19.327763Z, 2024-02-21T08:01:20.303231Z, 2024-02-21T08:01:21.296521Z, 2024-02-21T08:01:21.309756Z, 2024-02-21T08:01:22.308972Z, 2024-02-21T08:01:23.306708Z, 2024-02-21T08:01:24.296625Z, 2024-02-21T08:01:25.303714Z, 2024-02-21T08:01:26.304485Z, 2024-02-21T08:01:27.307624Z, 2024-02-21T08:01:28.313132Z, 2024-02-21T08:01:29.311219Z, 2024-02-21T08:01:30.287911Z, 2024-02-21T08:01:31.286856Z, 2024-02-21T08:01:32.296923Z, 2024-02-21T08:01:33.305499Z, 2024-02-21T08:01:34.313470Z, 2024-02-21T08:01:35.312839Z, 2024-02-21T08:01:36.277635Z, 2024-02-21T08:01:37.293178Z, 2024-02-21T08:01:38.293124Z, 2024-02-21T08:01:39.289481Z, 2024-02-21T08:01:40.301033Z, 2024-02-21T08:01:41.299248Z, 2024-02-21T08:01:42.305067Z, 2024-02-21T08:01:43.306277Z, 2024-02-21T08:01:44.308773Z, 2024-02-21T08:01:45.299660Z, 2024-02-21T08:01:46.308780Z, 2024-02-21T08:01:47.309346Z, 2024-02-21T08:01:48.300939Z, 2024-02-21T08:01:49.306829Z, 2024-02-21T08:01:50.307207Z, 2024-02-21T08:01:51.294527Z, 2024-02-21T08:01:52.299678Z, 2024-02-21T08:01:53.299981Z, 2024-02-21T08:01:54.314978Z, 2024-02-21T08:01:55.321036Z, 2024-02-21T08:01:56.319813Z, 2024-02-21T08:01:57.299793Z, 2024-02-21T08:01:58.308501Z, 2024-02-21T08:01:59.309298Z, 2024-02-21T08:02:00.287368Z, 2024-02-21T08:02:01.289217Z, 2024-02-21T08:02:02.299767Z, 2024-02-21T08:02:03.304518Z, 2024-02-21T08:02:04.309283Z, 2024-02-21T08:02:05.307966Z, 2024-02-21T08:02:06.296502Z, 2024-02-21T08:02:07.304684Z, 2024-02-21T08:02:08.302886Z, 2024-02-21T08:02:09.309604Z, 2024-02-21T08:02:10.314445Z, 2024-02-21T08:02:11.310930Z, 2024-02-21T08:02:12.314808Z, 2024-02-21T08:02:13.319508Z, 2024-02-21T08:02:14.318230Z, 2024-02-21T08:02:15.303998Z, 2024-02-21T08:02:16.302881Z, 2024-02-21T08:02:17.310767Z, 2024-02-21T08:02:18.312068Z, 2024-02-21T08:02:19.307879Z, 2024-02-21T08:02:20.312067Z, 2024-02-21T08:02:21.287875Z, 2024-02-21T08:02:22.299939Z, 2024-02-21T08:02:23.301433Z, 2024-02-21T08:02:24.302388Z, 2024-02-21T08:02:25.316099Z, 2024-02-21T08:02:26.302551Z, 2024-02-21T08:02:27.314173Z, 2024-02-21T08:02:28.319195Z, 2024-02-21T08:02:29.319867Z, 2024-02-21T08:02:30.291269Z, 2024-02-21T08:02:31.296852Z, 2024-02-21T08:02:32.304873Z, 2024-02-21T08:02:33.317328Z, 2024-02-21T08:02:34.322870Z, 2024-02-21T08:02:35.323871Z, 2024-02-21T08:02:36.306808Z, 2024-02-21T08:02:37.314149Z, 2024-02-21T08:02:38.302746Z, 2024-02-21T08:02:39.297075Z, 2024-02-21T08:02:40.313215Z, 2024-02-21T08:02:41.311860Z, 2024-02-21T08:02:42.303227Z, 2024-02-21T08:02:43.312870Z, 2024-02-21T08:02:44.310897Z, 2024-02-21T08:02:45.297110Z, 2024-02-21T08:02:46.298434Z, 2024-02-21T08:02:47.307188Z, 2024-02-21T08:02:48.320901Z, 2024-02-21T08:02:49.325133Z, 2024-02-21T08:02:50.323376Z, 2024-02-21T08:02:51.308433Z, 2024-02-21T08:02:52.316606Z, 2024-02-21T08:02:53.317410Z, 2024-02-21T08:02:54.329432Z, 2024-02-21T08:02:55.332361Z, 2024-02-21T08:02:56.330873Z, 2024-02-21T08:02:57.317949Z, 2024-02-21T08:02:58.322632Z, 2024-02-21T08:02:59.322914Z, 2024-02-21T08:03:00.316263Z, 2024-02-21T08:03:01.325420Z, 2024-02-21T08:03:02.321883Z, 2024-02-21T08:03:03.310896Z, 2024-02-21T08:03:04.319005Z, 2024-02-21T08:03:05.319568Z, 2024-02-21T08:03:06.325221Z, 2024-02-21T08:03:07.328459Z, 2024-02-21T08:03:08.328869Z, 2024-02-21T08:03:09.351578Z, 2024-02-21T08:03:10.361952Z, 2024-02-21T08:03:11.360464Z, 2024-02-21T08:03:12.344339Z, 2024-02-21T08:03:13.352898Z, 2024-02-21T08:03:14.351264Z, 2024-02-21T08:03:15.352210Z, 2024-02-21T08:03:16.343630Z, 2024-02-21T08:03:17.343574Z, 2024-02-21T08:03:18.358258Z, 2024-02-21T08:03:19.453335Z, 2024-02-21T08:03:20.359213Z, 2024-02-21T08:03:21.361006Z, 2024-02-21T08:03:21.384404Z, 2024-02-21T08:03:22.368183Z, 2024-02-21T08:03:23.366537Z, 2024-02-21T08:03:24.355065Z, 2024-02-21T08:03:25.363222Z, 2024-02-21T08:03:26.363548Z, 2024-02-21T08:03:27.364691Z, 2024-02-21T08:03:28.373064Z, 2024-02-21T08:03:29.371607Z, 2024-02-21T08:03:30.357086Z, 2024-02-21T08:03:31.363181Z, 2024-02-21T08:03:32.363472Z, 2024-02-21T08:03:33.374042Z, 2024-02-21T08:03:34.375779Z, 2024-02-21T08:03:35.375096Z, 2024-02-21T08:03:36.359369Z, 2024-02-21T08:03:37.368009Z, 2024-02-21T08:03:38.368055Z, 2024-02-21T08:03:39.354172Z, 2024-02-21T08:03:40.368468Z, 2024-02-21T08:03:41.368538Z, 2024-02-21T08:03:42.379286Z, 2024-02-21T08:03:43.385206Z, 2024-02-21T08:03:44.384502Z], show_extra_information: Some(SeenShowExtraInformation { season: 5, episode: 9 }), podcast_extra_information: None, anime_extra_information: None, manga_extra_information: None, last_updated_on: 2024-02-21T08:03:44.384502Z, num_times_updated: 668 }

I suspect this is unrelated though as the issue has been going on longer than this sort of data has been present.

For each of the monitored shows, they all have Updating metadata for <ID> and Updated metadata for <ID> within seconds, so at first glance it doesn't seem as though it's getting stuck on any shows.

I also don't have any notification systems configured (intentionally). Would I be right in saying Ryot will avoid trying to send any notifications if there is no notification system configured?

IgnisDa commented 8 months ago

I was able to confirm that setting DISABLE_UPDATE_WATCHLIST_MEDIA=true worked around the issue for me.

Yep, you're right. I believe it is not actual metadata update that is causing the problem but the associated person data being updated that is the problem. Shows have a lot of people associated with them, the problem compounds with more of them being monitored.

I haven't been able to come up with a satisfactory way to deal with this problem. But I will be focusing on this in the coming months.

I also don't have any notification systems configured (intentionally). Would I be right in saying Ryot will avoid trying to send any notifications if there is no notification system configured?

Sending notifications is just a for-loop, so if your server has none configured, it is effectively a no-op. I don't think that is related to the over-arching issue here at all.

ellsclytn commented 8 months ago

I wonder, would it be a complex change to allow the person tracking functionality to be disabled? Mainly as a workaround, but I'll also admit that I don't personally have much use for the person tracking stuff. Understandable if that's easier said than done though.

IgnisDa commented 8 months ago

I have thought of a solution that will enable Ryot to not be so aggressive with getting person/group details. This will also enable me to finally work #580 and #444.

IgnisDa commented 8 months ago

@ellsclytn I think I finally got it. The idle CPU usage is still a bit high (~8-15%) for my liking, but the spike seems to be no longer happening. Can you confirm on your instance too?

ellsclytn commented 8 months ago

Sure thing, I've just switched to 4.2.10. I'll leave it for 2 days and let you know how it goes.

IgnisDa commented 8 months ago

Note to self:

There still seems to be a huge spike every minute: image

This is probably due to the usage of apalis.

EDIT: This was not being caused by Ryot. It was being caused by other containers.

Here is the CPU usage by Ryot:

image

ellsclytn commented 8 months ago

Last two days have been looking a lot better :tada:

image

IgnisDa commented 8 months ago

I think this can be finally closed. Thanks everyone who helped me debug it!

If anyone is interested in knowing what the problem was, I wrote a small write up on my blog.

IgnisDa commented 4 months ago

Another small update: after v6, idle CPU usage should pretty much drop to 0%.

image

(Made changes to the scheduler backend so that it reads new jobs directly from memory instead of sqlite).