umputun / feed-master

Pulls multiple podcast feeds (RSS) and republishes as a common feed, properly sorted and podcast-client friendly.
https://feed-master.umputun.dev
MIT License
116 stars 26 forks source link

Total number of files limited to 256 #125

Open mokkosid opened 6 months ago

mokkosid commented 6 months ago

Hi, apologies, I'm not very technical. No matter if feed limit is set, or total value - I always get 256 max files. Is there any other setting to tune it? Here's snippet from my config:

`youtube:
  base_url: http://192.168.1.137:8080/yt/media
  dl_template: yt-dlp --extract-audio --audio-format=mp3 --audio-quality=0 -f m4a/bestaudio "https://www.youtube.com/watch?v={{.ID}}" --no-progress -o {{.FileName}}
  base_chan_url: "https://www.youtube.com/feeds/videos.xml?channel_id="
  base_playlist_url: "https://www.youtube.com/feeds/videos.xml?playlist_id="
  update: 60s
  max_per_channel: 2
  files_location: ./var/yt
  rss_location: ./var/rss
  channels:
  - {id: PLZVQqcKxEn_6YaOniJmxATjODSVUbbMkd, name: "Точка", type: "playlist", lang: "ru-ru"}
  - {id: PLeIdlMruOHyn32oLDjhfN8eso9TI7JMTs, name: "Бессмертный доктор", type: "playlist", lang: "ru-ru", "keep": 20}
  - {id: PL2GGJKocg5o6eGEzD1_D3tlXS3tu01QAq, name: "ГКЧП", type: "playlist", lang: "ru-ru"}
  - {id: PLvJR9G7c0HbaEORIasjhhfn7QxAAPRXF4, name: "Арт и Потапенко", type: "playlist", lang: "ru-ru"}
  - {id: PLr0vtm6PM7z04N3CBo5aTlAEcT6zQs4nU, name: "Как все навернулось", type: "playlist", lang: "ru-ru"}  
  - {id: PL2GGJKocg5o7hyH5lnDFugIoDfpDPHhxU, name: "Зомбоящик", type: "playlist", lang: "ru-ru"}
  - {id: PLNJFUkCCIG7waDlYjC-G6W12pdZijbLFU, name: "Машины", type: "playlist", lang: "ru-ru"} 
  - {id: PLNJFUkCCIG7xGO_j2O3Vwn29RTZXI7nAI, name: "Арсенал", type: "playlist", lang: "ru-ru"} 
  - {id: UCyYl1mXWoe4z_FcxzMaDHCw, name: "Шендерович", type: "channel", lang: "ru-ru"} 
  - {id: PL3s9UGpg5TNWELPe8VIxlu-M5HiLEmKXE, name: "Перст Фомы", type: "playlist", lang: "ru-ru", "keep": 15} 
  - {id: PLZVQqcKxEn_5Bm8fzIZIuCdRHcXhw5Nnd, name: "Ключевые события", type: "playlist", lang: "ru-ru"}
  - {id: PLZVQqcKxEn_4a-Ep4FbSwr1mquUR2U0Ik, name: "Classic news", type: "playlist", lang: "ru-ru"}
  - {id: PLZVQqcKxEn_5Iq5d-oAh-TFifAc-F-1J7, name: "Плющев Стримы", type: "playlist", lang: "ru-ru", "keep": 5}
  - {id: PLZVQqcKxEn_5kBobhQoB-dYlI-M2Es_aQ, name: "Кто такой", type: "playlist", lang: "ru-ru", "keep": 5}
  - {id: PLWhOMsxksihbgmpwiHZzDCx9PXxtpZsns, name: "История Кузнецов", type: "playlist", lang: "ru-ru"}  
  - {id: PLWhOMsxksihbjDvVBEud4b5HCvs5cwwRj, name: "Курс Потапенко", type: "playlist", lang: "ru-ru"}  
  - {id: PLWhOMsxksihYqUZjvJuerSc9gbx5TjggS, name: "Будем наблюдать", type: "playlist", lang: "ru-ru"} 
  - {id: UC5wRw_R3kzwV5zGe7kzec8A, name: "Пархоменко", type: "channel", lang: "ru-ru"}
  - {id: PLXkT3vA5uPhNsjrMhWiYCLLnqmWdvagT5, name: "Шепелин Стримы", type: "playlist", lang: "ru-ru"}

system:
  update: 1m
  max_per_feed: 25
  max_total: 1000
  max_keep: 1000
  base_url: http://192.168.1.137:8080`

Here's log message: 2024/02/29 06:32:10.925 [INFO] all channels processed - channels: 19, entries: 260, processed: 256, updated: 0, removed: 0, ignored: 2, skipped: 256, lifetime: 256, feed size: 256

Most of the feeds got 15 files as maximum in this configuration. Could you please help to understand how to raise this limit?

mokkosid commented 6 months ago

Adding update. Tested with simple config with 1 youtube playlist:

`feeds: Point: title: Точка description: link: http://localhost:8080/feed/Point language: "ru-ru" image: images/yt-example.png sources:

youtube: base_url: http://192.168.1.137:8080/yt/media dl_template: yt-dlp --extract-audio --audio-format=mp3 --audio-quality=0 -f m4a/bestaudio "https://www.youtube.com/watch?v={{.ID}}" --no-progress -o {{.FileName}} base_chan_url: "https://www.youtube.com/feeds/videos.xml?channel_id=" base_playlist_url: "https://www.youtube.com/feeds/videos.xml?playlist_id=" update: 60s max_per_channel: 20 files_location: ./var/yt rss_location: ./var/rss channels:

system: update: 1m max_per_feed: 35 max_total: 10000 max_keep: 10000 base_url: http://192.168.1.137:8080`

Can only get 15 records: 2024-03-12 19:21:34 2024/03/12 11:21:34.612 [INFO] GET - /yt/rss/PLZVQqcKxEn_6YaOniJmxATjODSVUbbMkd - 192.168.1.137 - 192.168.65.0 - 200 (50587) - 1.114278ms 2024-03-12 19:22:31 2024/03/12 11:22:31.374 [INFO] got 15 entries for Точка, limit to 35 2024-03-12 19:22:31 2024/03/12 11:22:31.375 [INFO] all channels processed - channels: 1, entries: 15, processed: 15, updated: 0, removed: 0, ignored: 0, skipped: 15, lifetime: 15, feed size: 15