search

dan-online / autopulse

💫 automated scanning tool that bridges media organizers such as Sonarr and Radarr with media servers such as Plex and Jellyfin
https://autopulse.dancodes.online
MIT License
127 stars 5 forks source link

bug: Redundant calls to refresh endpoint of plex #45

Closed mvanbaak closed 1 month ago

mvanbaak commented 1 month ago

Description of the bug

So my current setup is:

The manual url bazarr uses: http://192.168.10.240:2875/triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01'

Logs of autopulse registering the triggers:

2024-09-25T15:48:21.245219Z DEBUG autopulse::routes::triggers: added 1 file from manual trigger
2024-09-25T15:48:21.245311Z  INFO actix_web::middleware::logger: 192.168.10.50 "GET /triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01 HTTP/1.1" 200 429 "-" "curl/8.7.1" 0.010673
2024-09-25T15:48:28.622728Z DEBUG autopulse::routes::triggers: added 1 file from manual trigger
2024-09-25T15:48:28.622806Z  INFO actix_web::middleware::logger: 192.168.10.50 "GET /triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01 HTTP/1.1" 200 429 "-" "curl/8.7.1" 0.003517
2024-09-25T15:48:37.469985Z DEBUG autopulse::routes::triggers: added 1 file from manual trigger
2024-09-25T15:48:37.470066Z  INFO actix_web::middleware::logger: 192.168.10.50 "GET /triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01 HTTP/1.1" 200 429 "-" "curl/8.7.1" 0.004094
2024-09-25T15:48:55.249892Z DEBUG autopulse::routes::triggers: added 1 file from manual trigger
2024-09-25T15:48:55.249986Z  INFO actix_web::middleware::logger: 192.168.10.50 "GET /triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01 HTTP/1.1" 200 429 "-" "curl/8.7.1" 0.003927

All triggers are like 1 or 2 minutes from each other. The trigger timers thing works ok as autopulse worker only sends the files to the targets after X minutes after the last trigger for a directory/file.

BUT ... autopulse sends multiple refresh requests for the same directory to plex. While I would expect it to do only 1 per directory:

logs from the nginx that is running as reverse proxy of my plex instance:

2a02:a443:381e:1::240 - - [25/Sep/2024:17:58:04 +0200] "GET /library/sections HTTP/1.1" 200 6316 "-" "-" "-"
2a02:a443:381e:1::240 - - [25/Sep/2024:17:58:04 +0200] "GET /library/sections/13/refresh?path=%2Fmnt%2Fmvanbaaknas%2FSeries%2FPeda%C3%A7o+de+Mim+%282024%29+%5Bimdb-tt27505298%5D HTTP/1.1" 200 0 "-" "-" "-"
2a02:a443:381e:1::240 - - [25/Sep/2024:17:58:04 +0200] "GET /library/sections/13/refresh?path=%2Fmnt%2Fmvanbaaknas%2FSeries%2FPeda%C3%A7o+de+Mim+%282024%29+%5Bimdb-tt27505298%5D HTTP/1.1" 200 0 "-" "-" "-"
2a02:a443:381e:1::240 - - [25/Sep/2024:17:58:04 +0200] "GET /library/sections/13/refresh?path=%2Fmnt%2Fmvanbaaknas%2FSeries%2FPeda%C3%A7o+de+Mim+%282024%29+%5Bimdb-tt27505298%5D HTTP/1.1" 200 0 "-" "-" "-"
2a02:a443:381e:1::240 - - [25/Sep/2024:17:58:04 +0200] "GET /library/sections/13/refresh?path=%2Fmnt%2Fmvanbaaknas%2FSeries%2FPeda%C3%A7o+de+Mim+%282024%29+%5Bimdb-tt27505298%5D HTTP/1.1" 200 0 "-" "-" "-"
2a02:a443:381e:1::240 - - [25/Sep/2024:17:58:04 +0200] "GET /library/sections/13/refresh?path=%2Fmnt%2Fmvanbaaknas%2FSeries%2FPeda%C3%A7o+de+Mim+%282024%29+%5Bimdb-tt27505298%5D HTTP/1.1" 200 0 "-" "-" "-"

with autoscan this results in only 1 HTTP call to the refresh endpoint with that directory.

Steps To Reproduce

Contact the autopulse daemon with the same path several times, each within the timeout period from the other. For example, run the following command 2 or more times in a couple of seconds and watch the outgoing http calls to plex. Expected: only one per directory Actual: every trigger results in a http request to plex, even if the directory/file is the same

curl --basic --user admin:password 'http://192.168.10.240:2875/triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01'

Additional Information

Running autopulse 1.0.1 with the following configuration:

app:
  hostname: 0.0.0.0
  port: 2875
  database_url: sqlite:///var/db/autopulse/autopulse.db
  log_level: debug

auth:
  username: admin
  password: password

opts:
  check_path: false
  max_retries: 5
  default_timer_wait: 300
  cleanup_days: 10

triggers:
  manual:
    type: manual

targets:
  plex:
    type: plex
    url: https://my.plex.server
    token: SUPERSECRITNOTGOINGTOSHAREHERE

webhooks: {}
dan-online commented 1 month ago

I think the best way to solve this is to just check that the path is not already in the database in a pending process state. I'll add that in a PR

mvanbaak commented 1 month ago

I think the best way to solve this is to just check that the path is not already in the database in a pending process state. I'll add that in a PR

That would indeed be the best option I guess. As long as it does reset the timer though ;)

dan-online commented 1 month ago

Yep, it should still reset the timer since it will just return the existing event

dan-online commented 1 month ago

Feel free to check out pr-46 once this finishes

mvanbaak commented 1 month ago

Feel free to check out pr-46 once this finishes

I dont use docker. But I can check later today.

mvanbaak commented 1 month ago

Feel free to check out pr-46 once this finishes

I can confirm that it now sends only 1 refresh:

2024-09-27T08:46:30.183013Z  INFO autopulse: 💫 autopulse starting up...
2024-09-27T08:46:30.186775Z  INFO actix_server::builder: starting 12 workers
2024-09-27T08:46:30.186790Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
2024-09-27T08:46:30.186798Z  INFO actix_server::server: starting service: "actix-web-service-0.0.0.0:2875", workers: 12, listening on: 0.0.0.0:2875
2024-09-27T08:46:37.667117Z DEBUG autopulse::routes::triggers: added 1 file from manual trigger
2024-09-27T08:46:37.667204Z  INFO actix_web::middleware::logger: 192.168.10.50 "GET /triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01 HTTP/1.1" 200 429 "-" "curl/8.7.1" 0.008685
2024-09-27T08:46:50.701050Z DEBUG autopulse::routes::triggers: added 1 file from manual trigger
2024-09-27T08:46:50.701148Z  INFO actix_web::middleware::logger: 192.168.10.50 "GET /triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01 HTTP/1.1" 200 429 "-" "curl/8.7.1" 0.000375
2024-09-27T08:47:20.030644Z DEBUG autopulse::routes::triggers: added 1 file from manual trigger
2024-09-27T08:47:20.030714Z  INFO actix_web::middleware::logger: 192.168.10.50 "GET /triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01 HTTP/1.1" 200 429 "-" "curl/8.7.1" 0.000290
2024-09-27T08:47:32.948891Z DEBUG autopulse::routes::triggers: added 1 file from manual trigger
2024-09-27T08:47:32.948959Z  INFO actix_web::middleware::logger: 192.168.10.50 "GET /triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01 HTTP/1.1" 200 429 "-" "curl/8.7.1" 0.000280
2024-09-27T08:48:03.330211Z  INFO autopulse::service::runner: sent 1 file to targets
bonswouar commented 3 weeks ago

@mvanbaak Sorry to ping you on a fixed issue, but I was wondering, how did you configure Bazarr to trigger notifications to Autopulse? (i.e http://192.168.10.240:2875/triggers/manual?path=%2fmnt%2fmvanbaaknas%2fSeries%2fPeda%c3%a7o+de+Mim+%282024%29+%5bimdb-tt27505298%5d%2fSeason+01) You used Bazarr Notifications (with apprise), right? (I don't see any other native way)

The closest I got was using the Form notification type with the following URL: form://admin:password@autopulse_host:2875/triggers/my_bazarr?:title&:type&:version&:message&method=get&attach-as=path

But it seems attach-as doesn't work as I hoped, as path is always missing ("Query deserialize error: missing field path"). Tried also with JSON type, but same issue, no file attached, no way to extract the path.

Looking at the default payloads in Bazarr notifications, it seems to pretty much always be some plain text (i.e "MyMovie : English subtitles manually downloaded from embeddedsubtitles with a score of 100.0%."), Thus I don't really see how to extract the file path, any tips would be much appreciated!

mvanbaak commented 3 weeks ago

You used Bazarr Notifications (with apprise), right? (I don't see any other native way)

No, I use Custom Post-Processing which is under Settings -> Subtitles. I use http there to send a http request :)

The closest I got was using the Form notification type with the following URL: form://admin:password@autopulse_host:2875/triggers/my_bazarr?:title&:type&:version&:message&method=get&attach-as=path

But it seems attach-as doesn't work as I hoped, as path is always missing ("Query deserialize error: missing field path"). Tried also with JSON type, but same issue, no file attached, no way to extract the path.

Looking at the default payloads in Bazarr notifications, it seems to pretty much always be some plain text (i.e "MyMovie : English subtitles manually downloaded from embeddedsubtitles with a score of 100.0%."), Thus I don't really see how to extract the file path, any tips would be much appreciated!

bonswouar commented 3 weeks ago

Much easier!! I didn't know about this setting, thanks for your help