search

ben-xo / dir2cast

Turn a directory of MP3s into a podcast - automatically.
http://www.ben-xo.com/dir2cast
BSD 3-Clause "New" or "Revised" License
152 stars 20 forks source link

New install doesn't update if dir2cast is added after the media files #51

Closed gityopr closed 2 years ago

gityopr commented 2 years ago

Hi, I can't get dir2cast to update the feed when a new file is added to its directory. Deleting a file from the directory also doesn't cause any changes.

If I delete the contents of /temp then it refreshes appropriately but I'd prefer not to have to do this every time, if possible.

Details: I'm running dir2cast on a Raspberry Pi 4 in the /www folder of a SWAG docker container. Otherwise my .ini file is stock.

ben-xo commented 2 years ago

Can you give me the output of ls -l ? it updates based on the modification timestamps of the file, so that needs to be newer than dir2cast

gityopr commented 2 years ago

Thanks for the quick reply!

pi@rpi:~/podcast-server/kdbookecast $ ls -l
total 2164536
-rw-rw-r-- 1 pi pi      1203 Dec 21 18:34  502.html
-rwxr--r-- 1 pi pi   3767269 Dec 21 18:44 'A Full Life by Paulo Bacigalupi.m4a'
-rw-r--r-- 1 pi pi    295773 Dec 21 18:39 'Carl Sagan - The Demon-Haunted World (Cary Elwes, Seth MacFarlane).jpg'
-rwxrw-r-- 1 pi pi 498811479 Dec 21 18:34 'Carl Sagan - The Demon-Haunted World (Cary Elwes, Seth MacFarlane).mp3'
-rwxrw-r-- 1 pi pi 544397046 Dec 21 18:35  colorless-tsukuru-tazaki-and.mp3
-rw-rw-r-- 1 pi pi        96 Dec 21 18:34  composer.json
-rwxrwxr-x 1 pi pi      8908 Dec 21 18:35  dir2cast.ini
-rw-rw-r-- 1 pi pi     66709 Dec 21 18:34  dir2cast.php
drwxrwxr-x 2 pi pi      4096 Dec 21 18:34  getID3
-rwx-w---- 1 pi pi 406827669 Dec 21 18:35 'How Beautiful We Were.mp3'
-rwxrw-r-- 1 pi pi     22074 Dec 21 18:35  image.jpg
-rw-rw-r-- 1 pi pi       780 Dec 21 18:35  index.html
-rw-r--r-- 1 pi pi     31116 Dec 21 18:39 'Klara and the Sun.jpg'
-rwxrw-r-- 1 pi pi 587253078 Dec 21 18:36 'Klara and the Sun.m4a'
-rw-r--r-- 1 pi pi    115349 Dec 21 18:39 'Love Is an Ex-Country.jpg'
-rwxrw-r-- 1 pi pi 174817435 Dec 21 18:36 'Love Is an Ex-Country.m4a'
drwxr-xr-x 2 pi pi      4096 Dec 21 18:39  temp

I touch'd "A Full Life by Paulo Bacigalupi.m4a" since that was the one I was testing. That caused the script to pick it up and it appeared in my client apps shortly after. However, pasting a new file which has a newer modification date into the directory didn't update it. 

pi@rpi:~/podcast-server/kdbookecast $ ls -la
total 2171220
drwxrwxr-x  4 pi pi      4096 Dec 22 08:19  .
drwxr-xr-x 14 pi pi      4096 Dec 20 22:20  ..
-rw-rw-r--  1 pi pi      1203 Dec 21 18:34  502.html
-rwxr--r--  1 pi pi   3767269 Dec 22 08:13 'A Full Life by Paulo Bacigalupi.m4a'
-rw-r--r--  1 pi pi   6824161 Dec 22 08:19 'Alien (Preview).mp3'
-rw-r--r--  1 pi pi    295773 Dec 21 18:39 'Carl Sagan - The Demon-Haunted World (Cary Elwes, Seth MacFarlane).jpg'
-rwxrw-r--  1 pi pi 498811479 Dec 21 18:34 'Carl Sagan - The Demon-Haunted World (Cary Elwes, Seth MacFarlane).mp3'
-rwxrw-r--  1 pi pi 544397046 Dec 21 18:35  colorless-tsukuru-tazaki-and.mp3
-rw-rw-r--  1 pi pi        96 Dec 21 18:34  composer.json
-rwxrwxr-x  1 pi pi      8908 Dec 21 18:35  dir2cast.ini
-rw-rw-r--  1 pi pi     66709 Dec 21 18:34  dir2cast.php
-rwxrw-r--  1 pi pi      6148 Dec 21 18:39  .DS_Store
drwxrwxr-x  2 pi pi      4096 Dec 21 18:34  getID3
-rwx-w----  1 pi pi 406827669 Dec 21 18:35 'How Beautiful We Were.mp3'
-rwxrw-r--  1 pi pi     22074 Dec 21 18:35  image.jpg
-rw-rw-r--  1 pi pi       780 Dec 21 18:35  index.html
-rw-r--r--  1 pi pi     31116 Dec 21 18:39 'Klara and the Sun.jpg'
-rwxrw-r--  1 pi pi 587253078 Dec 21 18:36 'Klara and the Sun.m4a'
-rw-r--r--  1 pi pi    115349 Dec 21 18:39 'Love Is an Ex-Country.jpg'
-rwxrw-r--  1 pi pi 174817435 Dec 21 18:36 'Love Is an Ex-Country.m4a'
drwxr-xr-x  2 pi pi      4096 Dec 22 08:14  temp

This is the situation with the folder now, the server lists "A Full Life....m4a" but not "Alien (Preview).mp3". Here's the output of ls -l inside the temp folder:

pi@rpi:~/podcast-server/kdbookecast $ ls -l temp/
total 40
-rw-r--r-- 1 pi pi  268 Dec 21 18:39 '1205de5de8d7ad1fc2ac4079e146bfc1__Carl Sagan - The Demon-Haunted World (Cary Elwes, Seth MacFarlane).mp3__data'
-rw-r--r-- 1 pi pi  261 Dec 21 18:39 '29fb0b475b5c5980f6afcd5e08ce3f22__Klara and the Sun.m4a__data'
-rw-r--r-- 1 pi pi  104 Dec 22 08:14 '513726ad29a49b5aa9324880afb8c615__A Full Life by Paulo Bacigalupi.m4a__data'
-rw-r--r-- 1 pi pi  109 Dec 21 18:39 '93ec60a54c683cd5d310584ff36cc430__How Beautiful We Were.mp3__data'
-rw-r--r-- 1 pi pi 1241 Dec 21 18:39 '95822048accbb8e847e963dd005f5508__Love Is an Ex-Country.m4a__data'
-rw-r--r-- 1 pi pi   77 Dec 22 08:14 'ae5150c0a97439f7a4327499bf2aea36__._A Full Life by Paulo Bacigalupi.m4a__data'
-rw-r--r-- 1 pi pi 9793 Dec 22 08:27  bf49df1dd863708a9101bb03671c5556__config_www.xml
-rw-r--r-- 1 pi pi  313 Dec 21 18:39  cfb467a3fad5b73a1e9808c74ac9b5ed__colorless-tsukuru-tazaki-and.mp3__data

Thanks for your help!

Edit: Thought the code button would make it prettier, fixed that!

ben-xo commented 2 years ago

Try touching all of them in sequence, as they're mostly older than dir2cast. I wonder if copying them is preserving the date from the source folder? Then wait. Few mins before adding more

ben-xo commented 2 years ago

I think i see a separate bug, it shouldn't be including the . file for A Full Life

the config_www.xml temp file (i assume that's the feed itself) looks like it contains everything, is it just being browser cached? do you need to do a hard refresh in the browser?

dir2cast is designed to wait a bit after you update so that it isn't adding broken MP3s to the feed (as you copy them in), but waiting 5 mins solves that - or you can fiddle with that in the config

ben-xo commented 2 years ago

basically if that .xml temp file updates (and contains all the files) then the problem is probably caching in your browser.

the following two settings apply to your use case

; *** THESE SHOULD BE LEFT AS THEY ARE - unless you have a good reason. ***

; Number of seconds for which the cache file is guaranteed valid. You can 
; only specify this in the dir2cast.ini that is in the same folder as 
; dir2cast.php. Defaults to 5
;MIN_CACHE_TIME = 5

; Time-to-live (Expiry time) of the feed
; This defaults to 60 minutes
;TTL = 60

a new feed is valid of a minimum of 5 seconds before it will even try to regenerate. Also, it declares the feed as being current for an hour so feed readers may not refresh it for an hour unless you force them to. This includes HTTP caches and browsers, potentially also nginx depending how it's configured. (tacking ?saidsa junk on the end usually bypasses this if you want to test.)

Anyway, you can reduce these if you want.

there is also a URL ?force=password where you can make it refresh anyway, that's if the cache files are definitely not right.

; The password to use on the 'force=<password>' part of the URL in order to 
; force the RSS feed to be regenerated (bypassing and replacing the cached 
; copy). If this is empty then this feature is disabled. 
;FORCE_PASSWORD =
gityopr commented 2 years ago

Thanks, touching all the media files and confirming the .xml in that folder did the trick. I knew it would be something silly! Thank you.

ben-xo commented 2 years ago

Still technically a bug though (I didn't intend for it not to work properly if you copied dir2cast in AFTER your media files),, so I will try to improve the experience here!

gityopr commented 2 years ago

Yeah something is definitely up with the stock swag docker container and dir2cast. Tried spinning up a new webserver on a Debian VM instead of a Raspberry Pi. I also waited about an hour to add files to the directory.

The server works great but dir2cast needs that reset link or clearing the temp folder to update. Does it rely on some PHP extensions or anything like that?

ben-xo commented 2 years ago

Yeah something is definitely up with the stock swag docker container and dir2cast. Tried spinning up a new webserver on a Debian VM instead of a Raspberry Pi. I also waited about an hour to add files to the directory.

The server works great but dir2cast needs that reset link or clearing the temp folder to update. Does it rely on some PHP extensions or anything like that?

Only the XML extension which you must have if it worked at all. It does rely on the file modification times so if your file system is FAT32 or something instead of ext3 or ext4 it might not work. I’ve never tried it on fat32

ben-xo commented 2 years ago

I have been working on some more tests to see if I can reproduce and fix the problem. Thanks for reporting it.

ben-xo commented 2 years ago

i have done some work on the caching which may solve your problem, would love to know if they do

ben-xo commented 2 years ago

would also be useful if you could run the test suite for me on your Pi

gityopr commented 2 years ago

Sorry for the late reply, I've been away from this project because of the holidays. How do I test the test suite?

ben-xo commented 2 years ago

To run the test suite,

ben-xo commented 2 years ago

Hi @gityopr hoping to hear back from you, or perhaps give instructions so I can have a go :) but I have a hunch it's a twist specific to your particular combination and I might not be able to replicate it

gityopr commented 2 years ago

I'm sorry for the delay on this! I believe part of the issue is that I'm running my webserver out of a docker container. SWAG by Linuxserver. Last time I looked at this I had a lot of trouble figuring out how to get composer running in that container. I'll give it a bit more work this weekend.

Thanks for not closing this issue!

ben-xo commented 2 years ago

Last time I looked at this I had a lot of trouble figuring out how to get composer running in that container.

I would have thought you'd do something like this:

helmut72 commented 2 years ago

I have the same problem. Downloading a video with youtube-dl-server use the date of video. Also touch the file doesn't help. Even deleting a video still shows the video in my Podcast app. Solution is deleting dir2casts temp folder. Fow now I will run a 3rd container with a cronjob deleting the temp folder, if file count has been changed.

ben-xo commented 2 years ago

@helmut72 can you try setting MIN_CACHE_TIME=0 and MIN_FILE_AGE=0 in dir2cast.ini, it should effectively disable the cache altogether

ben-xo commented 2 years ago

I would still like to know more about the filesystem and the file dates that you have which are causing this behaviour - ideally a test case that I can work with so that I can reproduce the problem?

helmut72 commented 2 years ago

Hi, I use a standard Ubuntu 22.04 with Docker installed from docker.com, Apache with PHP. Nothng special with one exception: to reduce wear on a Raspberry I have set noatime option in fstab for ages, so I do it also on my server with SSD. I have moved my collection from one folder to the video folder of dir2cast. File dates are different. From 2019 to may 2022. Then I have tested if dir2cast recognize them. Can test your suggested settings after work.

ben-xo commented 2 years ago

Thank you, I appreciate the feedback. I have an idea for a fix which will enable this to work with your use cases.

I did not realise how popular it would be to copy "old" files into a podcast folder (I don't do this personally), but I will add some code to refresh the feed if the number of files changes.

helmut72 commented 2 years ago

can you try setting MIN_CACHE_TIME=0 and MIN_FILE_AGE=0 in dir2cast.ini

This fix one problem, using touch file.mp4 add the file in my Podcast app. But deleting a file it is still shown in my Podcast app. Also files with old dates won't be added.

I did not realise how popular it would be to copy "old" files into a podcast folder

When someone uploaded a video on Youtube in 2019, file date becomes 2019 on my harddrive. I use youtube-dl-server to get my content easily.

Also try wget: wget https://jsoncompare.org/LearningContainer/SampleFiles/Video/MP4/sample-mp4-file.mp4

File date is 2021 on the harddrive. dir2cast never find it.

ben-xo commented 2 years ago

Thanks. I understand your situation a lot better now. I really appreciate you taking the time to add details to the bug report.

I will come up with a solution for this.

ben-xo commented 2 years ago

@gityopr or @helmut72 if you'd care to try the #58 to see if it fixes your problem? if so, i'll merge this and add it to the official release

helmut72 commented 2 years ago

Looks good to me. Adding, deleting, adding again, deleting again, using touch before adding a 3rd time... all behaves like expected. Great, thanks!

ben-xo commented 2 years ago

v1.33 has been released which incorporates these changes. Thanks for your help!