Closed ellsclytn closed 8 months ago
Try setting env var RUST_LOG=ryot=debug
and try to regenerate summary.
I realised I've been running an older version of Postgres. I've upgraded to Postgres 16. I have debug
logs enabled on Ryot now, but manually kicking off a summary regeneration completed within seconds. I'll keep an eye on it tomorrow after the regular cron jobs start and report the results.
Unfortunately it has continued even after the Postgres upgrade. Once the midnight jobs started CPU returned to ~10% and stayed there. Then when the next midnight rolled around it roughly doubled CPU usage.
I'd post the logs except that they're quite big. 120MB for the debug logs in total. ~500KB if I exclude any lines mentioning "Progress update". I'll wait for #640 to release and try a few things with it.
@ellsclytn I released it as v4.2.5
.
I'm suffering from the same issue, at 01h25 in my timezone (UTC-3), the CPU usage of the Ryot container greatly increases. The graph below is my CPU usage:
This CPU usage continues for days. If I restart the container, it goes back to normal until the next 01h25.
I tried removing the number of monitored shows. There were more than I realised, but not many. It's now down to three. That's certainly helped, but it does appear the problem persists (and gets consistently worse with a higher monitoring count). The CPU usage seems to "stack" each time the midnight cron job fires off.
I was able to confirm that setting DISABLE_UPDATE_WATCHLIST_MEDIA=true
worked around the issue for me.
As for logs, I haven't seen a great deal that jumped out at me. The only thing that seemed a little different now was that some shows have quite a large log output which I think coincided with some more recent changes where the webhook path changed for the Jellyfin integration. Example of large output:
[backend] 2024-02-21T08:03:44.392967Z DEBUG ryot::miscellaneous::resolver: Progress update = Model { id: 10877, progress: 100, started_on: Some(2024-02-21), finished_on: Some(2024-02-21), user_id: 1, metadata_id: 479, state: Completed, updated_at: [2024-02-21T07:51:28.102148Z, 2024-02-21T07:51:29.076844Z, 2024-02-21T07:51:29.082814Z, 2024-02-21T07:51:30.083490Z, 2024-02-21T07:51:31.077929Z, 2024-02-21T07:51:32.057250Z, 2024-02-21T07:51:33.076630Z, 2024-02-21T07:51:34.072888Z, 2024-02-21T07:51:35.063937Z, 2024-02-21T07:51:36.078455Z, 2024-02-21T07:51:37.068737Z, 2024-02-21T07:51:38.082010Z, 2024-02-21T07:51:39.083784Z, 2024-02-21T07:51:40.088666Z, 2024-02-21T07:51:41.068606Z, 2024-02-21T07:51:42.077457Z, 2024-02-21T07:51:43.078821Z, 2024-02-21T07:51:44.079512Z, 2024-02-21T07:51:45.083775Z, 2024-02-21T07:51:46.072185Z, 2024-02-21T07:51:47.098788Z, 2024-02-21T07:51:48.101506Z, 2024-02-21T07:51:49.095454Z, 2024-02-21T07:51:50.084703Z, 2024-02-21T07:51:51.090321Z, 2024-02-21T07:51:52.091057Z, 2024-02-21T07:51:53.074288Z, 2024-02-21T07:51:54.083231Z, 2024-02-21T07:51:55.083420Z, 2024-02-21T07:51:56.069872Z, 2024-02-21T07:51:57.076592Z, 2024-02-21T07:51:58.077776Z, 2024-02-21T07:51:59.080866Z, 2024-02-21T07:52:00.086636Z, 2024-02-21T07:52:01.078323Z, 2024-02-21T07:52:02.080358Z, 2024-02-21T07:52:03.086206Z, 2024-02-21T07:52:04.086972Z, 2024-02-21T07:52:05.074869Z, 2024-02-21T07:52:06.087955Z, 2024-02-21T07:52:07.088170Z, 2024-02-21T07:52:08.082761Z, 2024-02-21T07:52:08.090154Z, 2024-02-21T07:52:09.087242Z, 2024-02-21T07:52:10.087337Z, 2024-02-21T07:52:11.072028Z, 2024-02-21T07:52:12.078831Z, 2024-02-21T07:52:13.079213Z, 2024-02-21T07:52:14.094454Z, 2024-02-21T07:52:15.099771Z, 2024-02-21T07:52:16.097773Z, 2024-02-21T07:52:17.076009Z, 2024-02-21T07:52:18.082507Z, 2024-02-21T07:52:19.083272Z, 2024-02-21T07:52:20.075004Z, 2024-02-21T07:52:21.083971Z, 2024-02-21T07:52:22.083075Z, 2024-02-21T07:52:23.066285Z, 2024-02-21T07:52:24.079564Z, 2024-02-21T07:52:25.080869Z, 2024-02-21T07:52:26.076148Z, 2024-02-21T07:52:27.088608Z, 2024-02-21T07:52:28.087441Z, 2024-02-21T07:52:29.090941Z, 2024-02-21T07:52:30.097319Z, 2024-02-21T07:52:31.085445Z, 2024-02-21T07:52:32.077071Z, 2024-02-21T07:52:33.084351Z, 2024-02-21T07:52:34.084610Z, 2024-02-21T07:52:35.081798Z, 2024-02-21T07:52:35.088777Z, 2024-02-21T07:52:36.088788Z, 2024-02-21T07:52:37.088170Z, 2024-02-21T07:52:38.113339Z, 2024-02-21T07:52:39.121880Z, 2024-02-21T07:52:40.120491Z, 2024-02-21T07:52:41.086818Z, 2024-02-21T07:52:42.096522Z, 2024-02-21T07:52:43.097243Z, 2024-02-21T07:52:44.087029Z, 2024-02-21T07:52:45.090137Z, 2024-02-21T07:52:46.081054Z, 2024-02-21T07:52:47.076876Z, 2024-02-21T07:52:48.089526Z, 2024-02-21T07:52:49.086563Z, 2024-02-21T07:52:50.081325Z, 2024-02-21T07:52:50.087416Z, 2024-02-21T07:52:51.086818Z, 2024-02-21T07:52:52.086535Z, 2024-02-21T07:52:53.093160Z, 2024-02-21T07:52:54.100368Z, 2024-02-21T07:52:55.099127Z, 2024-02-21T07:52:56.088632Z, 2024-02-21T07:52:57.097166Z, 2024-02-21T07:52:58.095666Z, 2024-02-21T07:52:59.083037Z, 2024-02-21T07:53:00.093291Z, 2024-02-21T07:53:01.081068Z, 2024-02-21T07:53:02.073270Z, 2024-02-21T07:53:03.085128Z, 2024-02-21T07:53:04.081277Z, 2024-02-21T07:53:05.094496Z, 2024-02-21T07:53:06.097820Z, 2024-02-21T07:53:07.098751Z, 2024-02-21T07:53:08.086611Z, 2024-02-21T07:53:09.098329Z, 2024-02-21T07:53:10.099019Z, 2024-02-21T07:53:11.086693Z, 2024-02-21T07:53:12.095324Z, 2024-02-21T07:53:13.125020Z, 2024-02-21T07:53:14.104731Z, 2024-02-21T07:53:14.126407Z, 2024-02-21T07:53:15.112457Z, 2024-02-21T07:53:16.100604Z, 2024-02-21T07:53:17.094569Z, 2024-02-21T07:53:18.099810Z, 2024-02-21T07:53:19.095504Z, 2024-02-21T07:53:20.100749Z, 2024-02-21T07:53:21.100462Z, 2024-02-21T07:53:22.100353Z, 2024-02-21T07:53:23.096486Z, 2024-02-21T07:53:23.101847Z, 2024-02-21T07:53:24.102366Z, 2024-02-21T07:53:25.102517Z, 2024-02-21T07:53:26.092127Z, 2024-02-21T07:53:27.099336Z, 2024-02-21T07:53:28.099684Z, 2024-02-21T07:53:29.106239Z, 2024-02-21T07:53:30.111700Z, 2024-02-21T07:53:31.100430Z, 2024-02-21T07:53:32.096400Z, 2024-02-21T07:53:33.102881Z, 2024-02-21T07:53:34.102642Z, 2024-02-21T07:53:35.105674Z, 2024-02-21T07:53:36.109173Z, 2024-02-21T07:53:37.107880Z, 2024-02-21T07:53:38.090264Z, 2024-02-21T07:53:39.104922Z, 2024-02-21T07:53:40.105080Z, 2024-02-21T07:53:41.095237Z, 2024-02-21T07:53:42.109958Z, 2024-02-21T07:53:43.107382Z, 2024-02-21T07:53:44.098979Z, 2024-02-21T07:53:45.110082Z, 2024-02-21T07:53:46.100274Z, 2024-02-21T07:53:47.099741Z, 2024-02-21T07:53:47.105273Z, 2024-02-21T07:53:48.105794Z, 2024-02-21T07:53:49.106203Z, 2024-02-21T07:53:50.095001Z, 2024-02-21T07:53:51.104611Z, 2024-02-21T07:53:52.102755Z, 2024-02-21T07:53:53.108729Z, 2024-02-21T07:53:54.114257Z, 2024-02-21T07:53:55.115604Z, 2024-02-21T07:53:56.101156Z, 2024-02-21T07:53:57.107531Z, 2024-02-21T07:53:58.108490Z, 2024-02-21T07:53:59.100859Z, 2024-02-21T07:54:00.109248Z, 2024-02-21T07:54:01.099810Z, 2024-02-21T07:54:02.109180Z, 2024-02-21T07:54:03.109076Z, 2024-02-21T07:54:04.111597Z, 2024-02-21T07:54:05.119111Z, 2024-02-21T07:54:06.124456Z, 2024-02-21T07:54:07.123606Z, 2024-02-21T07:54:08.093592Z, 2024-02-21T07:54:09.108363Z, 2024-02-21T07:54:10.109093Z, 2024-02-21T07:54:11.103030Z, 2024-02-21T07:54:11.109669Z, 2024-02-21T07:54:12.114131Z, 2024-02-21T07:54:13.100515Z, 2024-02-21T07:54:14.125554Z, 2024-02-21T07:54:15.131314Z, 2024-02-21T07:54:16.124546Z, 2024-02-21T07:54:17.104544Z, 2024-02-21T07:54:18.103453Z, 2024-02-21T07:54:19.114375Z, 2024-02-21T07:54:20.104054Z, 2024-02-21T07:54:21.113895Z, 2024-02-21T07:54:22.113874Z, 2024-02-21T07:54:23.119047Z, 2024-02-21T07:54:24.122575Z, 2024-02-21T07:54:25.124312Z, 2024-02-21T07:54:26.104330Z, 2024-02-21T07:54:27.110970Z, 2024-02-21T07:54:28.112567Z, 2024-02-21T07:54:29.114677Z, 2024-02-21T07:54:30.119563Z, 2024-02-21T07:54:31.108220Z, 2024-02-21T07:54:32.109399Z, 2024-02-21T07:54:33.113924Z, 2024-02-21T07:54:34.115213Z, 2024-02-21T07:54:35.117479Z, 2024-02-21T07:54:36.119892Z, 2024-02-21T07:54:37.118099Z, 2024-02-21T07:54:38.121913Z, 2024-02-21T07:54:39.120449Z, 2024-02-21T07:54:40.120100Z, 2024-02-21T07:54:41.107411Z, 2024-02-21T07:54:42.117291Z, 2024-02-21T07:54:43.117533Z, 2024-02-21T07:54:44.097872Z, 2024-02-21T07:54:45.108627Z, 2024-02-21T07:54:46.096928Z, 2024-02-21T07:54:47.093050Z, 2024-02-21T07:54:48.107771Z, 2024-02-21T07:54:49.108457Z, 2024-02-21T07:54:50.121509Z, 2024-02-21T07:54:51.124720Z, 2024-02-21T07:54:52.129748Z, 2024-02-21T07:54:53.106457Z, 2024-02-21T07:54:54.121895Z, 2024-02-21T07:54:55.118525Z, 2024-02-21T07:54:56.106250Z, 2024-02-21T07:54:57.121Z, 2024-02-21T07:54:58.119194Z, 2024-02-21T07:54:59.120866Z, 2024-02-21T07:55:00.125038Z, 2024-02-21T07:55:01.115063Z, 2024-02-21T07:55:02.127989Z, 2024-02-21T07:55:03.131691Z, 2024-02-21T07:55:04.122076Z, 2024-02-21T07:55:05.115644Z, 2024-02-21T07:55:06.113814Z, 2024-02-21T07:55:07.123356Z, 2024-02-21T07:55:08.131397Z, 2024-02-21T07:55:09.137045Z, 2024-02-21T07:55:10.136148Z, 2024-02-21T07:55:11.116573Z, 2024-02-21T07:55:12.124050Z, 2024-02-21T07:55:13.150396Z, 2024-02-21T07:55:14.142895Z, 2024-02-21T07:55:14.163561Z, 2024-02-21T07:55:15.151438Z, 2024-02-21T07:55:16.149553Z, 2024-02-21T07:55:17.118024Z, 2024-02-21T07:55:18.123997Z, 2024-02-21T07:55:19.121363Z, 2024-02-21T07:55:20.117542Z, 2024-02-21T07:55:21.117461Z, 2024-02-21T07:55:22.125604Z, 2024-02-21T07:55:23.131670Z, 2024-02-21T07:55:24.134450Z, 2024-02-21T07:55:25.134847Z, 2024-02-21T07:55:26.106049Z, 2024-02-21T07:55:27.119884Z, 2024-02-21T07:55:28.121282Z, 2024-02-21T07:55:29.131315Z, 2024-02-21T07:55:30.137290Z, 2024-02-21T07:55:31.126174Z, 2024-02-21T07:55:32.137006Z, 2024-02-21T07:55:33.141823Z, 2024-02-21T07:55:34.140804Z, 2024-02-21T07:55:35.118676Z, 2024-02-21T07:55:36.116744Z, 2024-02-21T07:55:37.127983Z, 2024-02-21T07:55:38.119534Z, 2024-02-21T07:55:39.128560Z, 2024-02-21T07:55:40.129013Z, 2024-02-21T07:55:41.120286Z, 2024-02-21T07:55:41.127493Z, 2024-02-21T07:55:42.126967Z, 2024-02-21T07:55:43.127991Z, 2024-02-21T07:55:44.140047Z, 2024-02-21T07:55:45.148597Z, 2024-02-21T07:55:46.143430Z, 2024-02-21T07:55:47.130758Z, 2024-02-21T07:55:48.139620Z, 2024-02-21T07:55:48.776280Z, 2024-02-21T07:56:03.782048Z, 2024-02-21T07:56:18.792306Z, 2024-02-21T07:56:33.788010Z, 2024-02-21T07:56:48.789741Z, 2024-02-21T07:57:03.791718Z, 2024-02-21T07:57:18.898643Z, 2024-02-21T07:57:24.195733Z, 2024-02-21T07:57:25.203801Z, 2024-02-21T07:57:26.204450Z, 2024-02-21T07:57:27.181800Z, 2024-02-21T07:57:28.194524Z, 2024-02-21T07:57:29.195198Z, 2024-02-21T07:57:30.237567Z, 2024-02-21T07:57:30.257977Z, 2024-02-21T07:57:31.234567Z, 2024-02-21T07:57:32.242482Z, 2024-02-21T07:57:33.229576Z, 2024-02-21T07:57:34.234987Z, 2024-02-21T07:57:35.235967Z, 2024-02-21T07:57:36.241036Z, 2024-02-21T07:57:37.245341Z, 2024-02-21T07:57:38.243356Z, 2024-02-21T07:57:39.246586Z, 2024-02-21T07:57:40.252569Z, 2024-02-21T07:57:41.252245Z, 2024-02-21T07:57:42.239014Z, 2024-02-21T07:57:43.248883Z, 2024-02-21T07:57:44.249901Z, 2024-02-21T07:57:45.243766Z, 2024-02-21T07:57:45.251250Z, 2024-02-21T07:57:46.242792Z, 2024-02-21T07:57:47.251818Z, 2024-02-21T07:57:48.240697Z, 2024-02-21T07:57:49.240845Z, 2024-02-21T07:57:50.249388Z, 2024-02-21T07:57:51.235601Z, 2024-02-21T07:57:52.243122Z, 2024-02-21T07:57:53.243766Z, 2024-02-21T07:57:54.250887Z, 2024-02-21T07:57:55.253777Z, 2024-02-21T07:57:56.251069Z, 2024-02-21T07:57:57.242929Z, 2024-02-21T07:57:58.251355Z, 2024-02-21T07:57:59.256175Z, 2024-02-21T07:58:00.236896Z, 2024-02-21T07:58:01.234858Z, 2024-02-21T07:58:02.243375Z, 2024-02-21T07:58:03.241424Z, 2024-02-21T07:58:03.246978Z, 2024-02-21T07:58:04.247844Z, 2024-02-21T07:58:05.246505Z, 2024-02-21T07:58:06.224565Z, 2024-02-21T07:58:07.235633Z, 2024-02-21T07:58:08.236146Z, 2024-02-21T07:58:09.250353Z, 2024-02-21T07:58:10.253152Z, 2024-02-21T07:58:11.251749Z, 2024-02-21T07:58:12.240191Z, 2024-02-21T07:58:13.249597Z, 2024-02-21T07:58:14.250060Z, 2024-02-21T07:58:15.249855Z, 2024-02-21T07:58:16.246819Z, 2024-02-21T07:58:17.255543Z, 2024-02-21T07:58:18.288947Z, 2024-02-21T07:58:19.291601Z, 2024-02-21T07:58:20.291495Z, 2024-02-21T07:58:21.255760Z, 2024-02-21T07:58:22.264299Z, 2024-02-21T07:58:23.264103Z, 2024-02-21T07:58:24.250704Z, 2024-02-21T07:58:25.260754Z, 2024-02-21T07:58:26.261345Z, 2024-02-21T07:58:27.250729Z, 2024-02-21T07:58:28.262581Z, 2024-02-21T07:58:29.261231Z, 2024-02-21T07:58:30.244972Z, 2024-02-21T07:58:31.246492Z, 2024-02-21T07:58:32.254347Z, 2024-02-21T07:58:33.261996Z, 2024-02-21T07:58:34.263890Z, 2024-02-21T07:58:35.262800Z, 2024-02-21T07:58:36.252209Z, 2024-02-21T07:58:37.258816Z, 2024-02-21T07:58:38.261397Z, 2024-02-21T07:58:39.255535Z, 2024-02-21T07:58:39.264775Z, 2024-02-21T07:58:40.262990Z, 2024-02-21T07:58:41.263709Z, 2024-02-21T07:58:42.254733Z, 2024-02-21T07:58:43.264564Z, 2024-02-21T07:58:44.266092Z, 2024-02-21T07:58:45.248007Z, 2024-02-21T07:58:46.255545Z, 2024-02-21T07:58:47.255912Z, 2024-02-21T07:58:48.247414Z, 2024-02-21T07:58:49.257003Z, 2024-02-21T07:58:50.258648Z, 2024-02-21T07:58:51.256348Z, 2024-02-21T07:58:52.263374Z, 2024-02-21T07:58:53.263479Z, 2024-02-21T07:58:54.266760Z, 2024-02-21T07:58:55.272424Z, 2024-02-21T07:58:56.271040Z, 2024-02-21T07:58:57.252297Z, 2024-02-21T07:58:58.264532Z, 2024-02-21T07:58:59.262287Z, 2024-02-21T07:59:00.273100Z, 2024-02-21T07:59:01.268727Z, 2024-02-21T07:59:02.280595Z, 2024-02-21T07:59:03.260433Z, 2024-02-21T07:59:04.266471Z, 2024-02-21T07:59:05.266941Z, 2024-02-21T07:59:06.268796Z, 2024-02-21T07:59:07.272228Z, 2024-02-21T07:59:08.272266Z, 2024-02-21T07:59:09.260737Z, 2024-02-21T07:59:10.269065Z, 2024-02-21T07:59:11.269332Z, 2024-02-21T07:59:12.263741Z, 2024-02-21T07:59:12.270187Z, 2024-02-21T07:59:13.273922Z, 2024-02-21T07:59:14.271657Z, 2024-02-21T07:59:15.253961Z, 2024-02-21T07:59:16.257856Z, 2024-02-21T07:59:17.268526Z, 2024-02-21T07:59:18.267182Z, 2024-02-21T07:59:19.294126Z, 2024-02-21T07:59:20.270704Z, 2024-02-21T07:59:21.276772Z, 2024-02-21T07:59:21.298943Z, 2024-02-21T07:59:22.284047Z, 2024-02-21T07:59:23.282334Z, 2024-02-21T07:59:24.265932Z, 2024-02-21T07:59:25.270905Z, 2024-02-21T07:59:26.270963Z, 2024-02-21T07:59:27.277282Z, 2024-02-21T07:59:28.284275Z, 2024-02-21T07:59:29.283685Z, 2024-02-21T07:59:30.249422Z, 2024-02-21T07:59:31.254651Z, 2024-02-21T07:59:32.258356Z, 2024-02-21T07:59:33.280537Z, 2024-02-21T07:59:34.283695Z, 2024-02-21T07:59:35.273125Z, 2024-02-21T07:59:36.263693Z, 2024-02-21T07:59:37.270818Z, 2024-02-21T07:59:38.270196Z, 2024-02-21T07:59:39.274114Z, 2024-02-21T07:59:40.265921Z, 2024-02-21T07:59:41.275628Z, 2024-02-21T07:59:42.264874Z, 2024-02-21T07:59:43.272402Z, 2024-02-21T07:59:44.270144Z, 2024-02-21T07:59:45.269557Z, 2024-02-21T07:59:46.283514Z, 2024-02-21T07:59:47.284941Z, 2024-02-21T07:59:48.270127Z, 2024-02-21T07:59:49.278477Z, 2024-02-21T07:59:50.267712Z, 2024-02-21T07:59:51.260208Z, 2024-02-21T07:59:52.267829Z, 2024-02-21T07:59:53.267996Z, 2024-02-21T07:59:54.277022Z, 2024-02-21T07:59:55.281830Z, 2024-02-21T07:59:56.280459Z, 2024-02-21T07:59:57.260705Z, 2024-02-21T07:59:58.277002Z, 2024-02-21T07:59:59.277748Z, 2024-02-21T08:00:00.268039Z, 2024-02-21T08:00:01.267973Z, 2024-02-21T08:00:02.274423Z, 2024-02-21T08:00:03.284237Z, 2024-02-21T08:00:04.288070Z, 2024-02-21T08:00:05.286527Z, 2024-02-21T08:00:06.286187Z, 2024-02-21T08:00:07.291022Z, 2024-02-21T08:00:08.292858Z, 2024-02-21T08:00:09.277284Z, 2024-02-21T08:00:10.285274Z, 2024-02-21T08:00:11.284851Z, 2024-02-21T08:00:12.270250Z, 2024-02-21T08:00:13.277903Z, 2024-02-21T08:00:14.274778Z, 2024-02-21T08:00:15.286739Z, 2024-02-21T08:00:16.293985Z, 2024-02-21T08:00:17.290689Z, 2024-02-21T08:00:18.276563Z, 2024-02-21T08:00:19.285817Z, 2024-02-21T08:00:20.286192Z, 2024-02-21T08:00:21.288304Z, 2024-02-21T08:00:22.290991Z, 2024-02-21T08:00:23.291920Z, 2024-02-21T08:00:24.282009Z, 2024-02-21T08:00:25.288901Z, 2024-02-21T08:00:26.289074Z, 2024-02-21T08:00:27.289957Z, 2024-02-21T08:00:28.294428Z, 2024-02-21T08:00:29.292258Z, 2024-02-21T08:00:30.275069Z, 2024-02-21T08:00:31.282083Z, 2024-02-21T08:00:32.284626Z, 2024-02-21T08:00:33.275830Z, 2024-02-21T08:00:34.288637Z, 2024-02-21T08:00:35.287189Z, 2024-02-21T08:00:36.290997Z, 2024-02-21T08:00:37.292564Z, 2024-02-21T08:00:38.291384Z, 2024-02-21T08:00:39.287308Z, 2024-02-21T08:00:39.295186Z, 2024-02-21T08:00:40.292077Z, 2024-02-21T08:00:41.292565Z, 2024-02-21T08:00:42.278611Z, 2024-02-21T08:00:43.285663Z, 2024-02-21T08:00:44.282690Z, 2024-02-21T08:00:45.284773Z, 2024-02-21T08:00:46.286492Z, 2024-02-21T08:00:47.288018Z, 2024-02-21T08:00:48.292832Z, 2024-02-21T08:00:49.291569Z, 2024-02-21T08:00:50.296654Z, 2024-02-21T08:00:51.281858Z, 2024-02-21T08:00:52.283830Z, 2024-02-21T08:00:53.286848Z, 2024-02-21T08:00:54.297129Z, 2024-02-21T08:00:55.303697Z, 2024-02-21T08:00:56.304277Z, 2024-02-21T08:00:57.284579Z, 2024-02-21T08:00:58.292739Z, 2024-02-21T08:00:59.293651Z, 2024-02-21T08:01:00.285164Z, 2024-02-21T08:01:01.291153Z, 2024-02-21T08:01:02.290430Z, 2024-02-21T08:01:03.292161Z, 2024-02-21T08:01:04.291251Z, 2024-02-21T08:01:05.292253Z, 2024-02-21T08:01:06.294871Z, 2024-02-21T08:01:07.297013Z, 2024-02-21T08:01:08.295646Z, 2024-02-21T08:01:09.299401Z, 2024-02-21T08:01:10.300152Z, 2024-02-21T08:01:11.299391Z, 2024-02-21T08:01:12.299256Z, 2024-02-21T08:01:13.299627Z, 2024-02-21T08:01:14.299285Z, 2024-02-21T08:01:15.300779Z, 2024-02-21T08:01:16.299526Z, 2024-02-21T08:01:17.300116Z, 2024-02-21T08:01:18.302746Z, 2024-02-21T08:01:19.327763Z, 2024-02-21T08:01:20.303231Z, 2024-02-21T08:01:21.296521Z, 2024-02-21T08:01:21.309756Z, 2024-02-21T08:01:22.308972Z, 2024-02-21T08:01:23.306708Z, 2024-02-21T08:01:24.296625Z, 2024-02-21T08:01:25.303714Z, 2024-02-21T08:01:26.304485Z, 2024-02-21T08:01:27.307624Z, 2024-02-21T08:01:28.313132Z, 2024-02-21T08:01:29.311219Z, 2024-02-21T08:01:30.287911Z, 2024-02-21T08:01:31.286856Z, 2024-02-21T08:01:32.296923Z, 2024-02-21T08:01:33.305499Z, 2024-02-21T08:01:34.313470Z, 2024-02-21T08:01:35.312839Z, 2024-02-21T08:01:36.277635Z, 2024-02-21T08:01:37.293178Z, 2024-02-21T08:01:38.293124Z, 2024-02-21T08:01:39.289481Z, 2024-02-21T08:01:40.301033Z, 2024-02-21T08:01:41.299248Z, 2024-02-21T08:01:42.305067Z, 2024-02-21T08:01:43.306277Z, 2024-02-21T08:01:44.308773Z, 2024-02-21T08:01:45.299660Z, 2024-02-21T08:01:46.308780Z, 2024-02-21T08:01:47.309346Z, 2024-02-21T08:01:48.300939Z, 2024-02-21T08:01:49.306829Z, 2024-02-21T08:01:50.307207Z, 2024-02-21T08:01:51.294527Z, 2024-02-21T08:01:52.299678Z, 2024-02-21T08:01:53.299981Z, 2024-02-21T08:01:54.314978Z, 2024-02-21T08:01:55.321036Z, 2024-02-21T08:01:56.319813Z, 2024-02-21T08:01:57.299793Z, 2024-02-21T08:01:58.308501Z, 2024-02-21T08:01:59.309298Z, 2024-02-21T08:02:00.287368Z, 2024-02-21T08:02:01.289217Z, 2024-02-21T08:02:02.299767Z, 2024-02-21T08:02:03.304518Z, 2024-02-21T08:02:04.309283Z, 2024-02-21T08:02:05.307966Z, 2024-02-21T08:02:06.296502Z, 2024-02-21T08:02:07.304684Z, 2024-02-21T08:02:08.302886Z, 2024-02-21T08:02:09.309604Z, 2024-02-21T08:02:10.314445Z, 2024-02-21T08:02:11.310930Z, 2024-02-21T08:02:12.314808Z, 2024-02-21T08:02:13.319508Z, 2024-02-21T08:02:14.318230Z, 2024-02-21T08:02:15.303998Z, 2024-02-21T08:02:16.302881Z, 2024-02-21T08:02:17.310767Z, 2024-02-21T08:02:18.312068Z, 2024-02-21T08:02:19.307879Z, 2024-02-21T08:02:20.312067Z, 2024-02-21T08:02:21.287875Z, 2024-02-21T08:02:22.299939Z, 2024-02-21T08:02:23.301433Z, 2024-02-21T08:02:24.302388Z, 2024-02-21T08:02:25.316099Z, 2024-02-21T08:02:26.302551Z, 2024-02-21T08:02:27.314173Z, 2024-02-21T08:02:28.319195Z, 2024-02-21T08:02:29.319867Z, 2024-02-21T08:02:30.291269Z, 2024-02-21T08:02:31.296852Z, 2024-02-21T08:02:32.304873Z, 2024-02-21T08:02:33.317328Z, 2024-02-21T08:02:34.322870Z, 2024-02-21T08:02:35.323871Z, 2024-02-21T08:02:36.306808Z, 2024-02-21T08:02:37.314149Z, 2024-02-21T08:02:38.302746Z, 2024-02-21T08:02:39.297075Z, 2024-02-21T08:02:40.313215Z, 2024-02-21T08:02:41.311860Z, 2024-02-21T08:02:42.303227Z, 2024-02-21T08:02:43.312870Z, 2024-02-21T08:02:44.310897Z, 2024-02-21T08:02:45.297110Z, 2024-02-21T08:02:46.298434Z, 2024-02-21T08:02:47.307188Z, 2024-02-21T08:02:48.320901Z, 2024-02-21T08:02:49.325133Z, 2024-02-21T08:02:50.323376Z, 2024-02-21T08:02:51.308433Z, 2024-02-21T08:02:52.316606Z, 2024-02-21T08:02:53.317410Z, 2024-02-21T08:02:54.329432Z, 2024-02-21T08:02:55.332361Z, 2024-02-21T08:02:56.330873Z, 2024-02-21T08:02:57.317949Z, 2024-02-21T08:02:58.322632Z, 2024-02-21T08:02:59.322914Z, 2024-02-21T08:03:00.316263Z, 2024-02-21T08:03:01.325420Z, 2024-02-21T08:03:02.321883Z, 2024-02-21T08:03:03.310896Z, 2024-02-21T08:03:04.319005Z, 2024-02-21T08:03:05.319568Z, 2024-02-21T08:03:06.325221Z, 2024-02-21T08:03:07.328459Z, 2024-02-21T08:03:08.328869Z, 2024-02-21T08:03:09.351578Z, 2024-02-21T08:03:10.361952Z, 2024-02-21T08:03:11.360464Z, 2024-02-21T08:03:12.344339Z, 2024-02-21T08:03:13.352898Z, 2024-02-21T08:03:14.351264Z, 2024-02-21T08:03:15.352210Z, 2024-02-21T08:03:16.343630Z, 2024-02-21T08:03:17.343574Z, 2024-02-21T08:03:18.358258Z, 2024-02-21T08:03:19.453335Z, 2024-02-21T08:03:20.359213Z, 2024-02-21T08:03:21.361006Z, 2024-02-21T08:03:21.384404Z, 2024-02-21T08:03:22.368183Z, 2024-02-21T08:03:23.366537Z, 2024-02-21T08:03:24.355065Z, 2024-02-21T08:03:25.363222Z, 2024-02-21T08:03:26.363548Z, 2024-02-21T08:03:27.364691Z, 2024-02-21T08:03:28.373064Z, 2024-02-21T08:03:29.371607Z, 2024-02-21T08:03:30.357086Z, 2024-02-21T08:03:31.363181Z, 2024-02-21T08:03:32.363472Z, 2024-02-21T08:03:33.374042Z, 2024-02-21T08:03:34.375779Z, 2024-02-21T08:03:35.375096Z, 2024-02-21T08:03:36.359369Z, 2024-02-21T08:03:37.368009Z, 2024-02-21T08:03:38.368055Z, 2024-02-21T08:03:39.354172Z, 2024-02-21T08:03:40.368468Z, 2024-02-21T08:03:41.368538Z, 2024-02-21T08:03:42.379286Z, 2024-02-21T08:03:43.385206Z, 2024-02-21T08:03:44.384502Z], show_extra_information: Some(SeenShowExtraInformation { season: 5, episode: 9 }), podcast_extra_information: None, anime_extra_information: None, manga_extra_information: None, last_updated_on: 2024-02-21T08:03:44.384502Z, num_times_updated: 668 }
I suspect this is unrelated though as the issue has been going on longer than this sort of data has been present.
For each of the monitored shows, they all have Updating metadata for <ID>
and Updated metadata for <ID>
within seconds, so at first glance it doesn't seem as though it's getting stuck on any shows.
I also don't have any notification systems configured (intentionally). Would I be right in saying Ryot will avoid trying to send any notifications if there is no notification system configured?
I was able to confirm that setting DISABLE_UPDATE_WATCHLIST_MEDIA=true worked around the issue for me.
Yep, you're right. I believe it is not actual metadata update that is causing the problem but the associated person
data being updated that is the problem. Shows have a lot of people associated with them, the problem compounds with more of them being monitored.
I haven't been able to come up with a satisfactory way to deal with this problem. But I will be focusing on this in the coming months.
I also don't have any notification systems configured (intentionally). Would I be right in saying Ryot will avoid trying to send any notifications if there is no notification system configured?
Sending notifications is just a for-loop
, so if your server has none configured, it is effectively a no-op. I don't think that is related to the over-arching issue here at all.
I wonder, would it be a complex change to allow the person
tracking functionality to be disabled? Mainly as a workaround, but I'll also admit that I don't personally have much use for the person tracking stuff. Understandable if that's easier said than done though.
I have thought of a solution that will enable Ryot to not be so aggressive with getting person/group details. This will also enable me to finally work #580 and #444.
@ellsclytn I think I finally got it. The idle CPU usage is still a bit high (~8-15%) for my liking, but the spike seems to be no longer happening. Can you confirm on your instance too?
Sure thing, I've just switched to 4.2.10
. I'll leave it for 2 days and let you know how it goes.
Note to self:
There still seems to be a huge spike every minute:
This is probably due to the usage of apalis.
EDIT: This was not being caused by Ryot. It was being caused by other containers.
Here is the CPU usage by Ryot:
Last two days have been looking a lot better :tada:
I think this can be finally closed. Thanks everyone who helped me debug it!
If anyone is interested in knowing what the problem was, I wrote a small write up on my blog.
Another small update: after v6
, idle CPU usage should pretty much drop to 0%.
(Made changes to the scheduler backend so that it reads new jobs directly from memory instead of sqlite).
I've noticed that my Ryot container enters a state of high CPU usage daily.
The above shows CPU usage over a 48h period. It seems to recover temporarily around 00:00, which leads me to believe it's something happening within one of the daily background jobs, but I'm just not sure which one as of yet.
The biggest spike seems to happen around 03:23-03:26.
I looked at the logs (where I've set
RUST_LOG=ryot=trace,sea_orm=debug
), but there doesn't seem to be much to look at beyond a lot of SQL queries.This is the last few lines of logs between
2023-10-25T03:23:00
to2023-10-25T04:26:00
.I'm wondering if anyone is/has experienced similar, or might have some further leads I could explore.