podget aborts with 'sed' error

stomar commented 8 years ago

podget stops while going through my feeds, with

sed: -e expression #1, char 28: unknown command: `A'

podget completely aborts, all servers that follow in the serverlist are not checked.

Example serverlist (problem lies in the first feed):

$ cat .podget/serverlist
http://www.radiotux.de/index.php?/feeds/index.rss2 RadioTux
http://thelinuxlink.net/tllts/tllts_ogg.rss Linux_Link

Output:

$ podget -vvvv
[snip]

Already downloaded http://archiv.radiotux.de/sendungen/weeklynews/2011-11-01.RadioTux.Ubuntu-Weekly-News-KW44.mp3.
sed: -e expression #1, char 28: unknown command: `A'

dvehrs commented 8 years ago

OK, looking at Podget, there are no more sed statements in the end of the loop after the "Already downloaded..." statement. So this error is most likely generated with the filename modification filters on the next item on the list.

I downloaded feed, and this may be the issue.

When we extract the enclosure tags from index.rss2, this is what I get for the last one downloaded and the next in line...

<enclosure url="http://archiv.radiotux.de/sendungen/weeklynews/2011-11-01.RadioTux.Ubuntu-Weekly-News-KW44.mp3" type="audio/mpeg" length='6382838' /> <enclosure url="http://a.blip.tv/api.swf#AYLbuQMA" type="" />

The second one appears to be a flash ad. That is where I believe the error is happening.

I've added the feed to my test feeds list and I'm currently downloading it to see what happens.

dvehrs commented 8 years ago

Interesting tidbits, blip.tv was acquired by Maker Studios in 2013 and shut down by them on August 20, 2015.

dvehrs commented 8 years ago

OK, I am not able to replicate the error you describe. The first time I ran podget over this feed, it downloaded every item but the one api.swf file which gave a series of "connection timed out" messages and then podget continued with the next item on the feed.

When I ran it a second time to get the 'Already downloaded..' messages, it looked like: `$ podget -vvvv podget

Parsing Config file. Config directory: /home/lowkey/.podget Config file: podgetrc Server List: serverlist

Loading temporary FILENAME_BADCHARS to test base configuration file and directory names. Clearing temporary FILENAME_BADCHARS, will read configured version from podgetrc

Library Directory: /home/lowkey/NoBackups/POD Session Directory: /tmp/user/1000/podget Log Directory: /home/lowkey/NoBackups/POD/.LOG WGet options: -c -nH --progress=bar Filename Bad Characters: `~!#$^&=+{}*[]:;"'<>?|\ Filename Replace Character: _ Debug Disabled - Delete temp files.

Session file not found. Creating podget.20026 .

Main loop. SERVER LIST FILE: serverlist WGET OPTIONS: -c -nH --progress=bar Playlist Creation Enabled. Playlist name: New-2016-03-27.r2.m3u

UTF-8 Loop running.

[...SNIP...]

Serverlist Line --> http://www.radiotux.de/index.php?/feeds/index.rss2 Reset WGET_OPTIONS to remove loop specific options -- WGET_COMMON_OPTIONS: -c -nH --progress=bar Reset WGET_OPTION_FILENAME_LOCATION Reset WGET_OPTION_DISPOSITION_FAIL

Category: GERMAN Name: RadioTux
Loop Specific WGET_OPTIONS: -c -nH --progress=bar WGET OPTIONS: -c -nH --progress=bar

Downloading feed index from http://www.radiotux.de/index.php?/feeds/index.rss2 --2016-03-27 19:25:26-- http://www.radiotux.de/index.php?/feeds/index.rss2 Resolving www.radiotux.de (www.radiotux.de)... 89.238.66.242 Connecting to www.radiotux.de (www.radiotux.de)|89.238.66.242|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/xml] Saving to: ‘STDOUT’

[ <=> ] 526.67K 534KB/s in 1.0s

2016-03-27 19:25:31 (534 KB/s) - written to stdout [539310]

Already downloaded http://archiv.radiotux.de/sendungen/radiotux/2016-02-26.RadioTux.Magazin.Februar2016.mp3.

Already downloaded http://archiv.radiotux.de/sendungen/radiotux/2016-01-31.RadioTux.Magazin.Januar2016.mp3.

[ ... SNIP ... ]

Already downloaded http://archiv.radiotux.de/sendungen/weeklynews/2011-11-01.RadioTux.Ubuntu-Weekly-News-KW44.mp3.

ORIGINAL FILENAME: api.swf#AYLbuQMA MODIFIED FILENAME: api.swf_AYLbuQMA Downloading api.swf_AYLbuQMA from http://a.blip.tv --2016-03-27 19:25:39-- http://a.blip.tv/api.swf Resolving a.blip.tv (a.blip.tv)... 74.122.172.235 Connecting to a.blip.tv (a.blip.tv)|74.122.172.235|:80... failed: Connection timed out. Retrying.

--2016-03-27 19:27:47-- (try: 2) http://a.blip.tv/api.swf Connecting to a.blip.tv (a.blip.tv)|74.122.172.235|:80... failed: Connection timed out. Retrying.

--2016-03-27 19:29:56-- (try: 3) http://a.blip.tv/api.swf Connecting to a.blip.tv (a.blip.tv)|74.122.172.235|:80... failed: Connection timed out. Retrying.

[ ... SNIP ... ]

--2016-03-27 20:06:04-- (try:19) http://a.blip.tv/api.swf Connecting to a.blip.tv (a.blip.tv)|74.122.172.235|:80... failed: Connection timed out. Retrying.

--2016-03-27 20:08:21-- (try:20) http://a.blip.tv/api.swf Connecting to a.blip.tv (a.blip.tv)|74.122.172.235|:80... failed: Connection timed out. Giving up.

Already downloaded http://archiv.radiotux.de/sendungen/techview-podcast/2011-10-30.Techview-Podcast.128.mp3.

Already downloaded http://archiv.radiotux.de/sendungen/talk/2011-10-27.RadioTux.Talk.145.mp3.

[ ... SNIP ... ] `

And podget did this without or with the small changes you submitted for the man pages and help text. So I'm not sure it's a problem with Podget. We had a similar problem recently that we ended up tracing to a associated program. So lets compare environments.

I'm testing on a Debian Sid system with: $ sed --version sed (GNU sed) 4.2.2

$ bash --version GNU bash, version 4.3.42(1)-release (x86_64-pc-linux-gnu)

$ uname -a Linux homeserver 4.4.0-1-amd64 #1 SMP Debian 4.4.6-1 (2016-03-17) x86_64 GNU/Linux

What are you running?

stomar commented 8 years ago

sed 4.2.2 GNU bash, Version 4.3.11(1)-release (x86_64-pc-linux-gnu) Linux 3.19.0-56-generic #62~14.04.1-Ubuntu SMP Fri Mar 11 11:03:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

ghost commented 8 years ago

I cannot reproduce this error under Fedora 23.

sed (GNU sed) 4.2.2
GNU bash, version 4.3.42(1)-release (x86_64-redhat-linux-gnu)
Linux 4.4.6-300.fc23.x86_64 #1 SMP Wed Mar 16 22:10:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

ghost commented 8 years ago

One more thing I also get the api.swf "Connection timed out".

dvehrs commented 8 years ago

OK, basically it appears we have similar environments, but given that Filip and I are getting similar results, I wonder if the difference is in how you have podget configured in podgetrc. Now I use a basically default configuration with one change to modify where the Library is stored.

What is your configuration? Do you have any options enabled that are not part of the default?

If you would like to compare to default, you can build one to check against with:

podget --create-config default-testrc

Then compare it with the default file with:

cd ${HOME}/.podget diff podgetrc default-testrc

Or if you prefer a little more visual a diff then do: vimdiff podgetrc default-testrc

My diff would look like:

$ diff podgetrc testrc 
18c18
< DIR_LIBRARY=/home/dvehrs/NoBackups/POD
---
> DIR_LIBRARY=/home/dvehrs/POD

stomar commented 8 years ago

Maybe it's not an environment thing after all... I actually can also reproduce your result; however, my initial podget run was different, and that seems to change the behaviour of later invocations.

Please try again with:

$ podget -r 1
$ podget

(Starting from an empty POD directory.)

This adds http://a.blip.tv/api.swf#AYLbuQMA to POD/.LOG/done instead of to POD/.LOG/errors, which at least for me triggers the 'sed' error in the next podget run.

BTW. I used both a default config and a config with all (switchable) filename cleanups switched off.

stomar commented 8 years ago

I did the same for a modified feed with a different enclosure url:

modified URL: http://a.blip.tv/api.swf#KLMN
results in:   sed: -e expression #1, char 28: unknown command: `K'

So, it's definitely the #K... part that causes the problem.

dvehrs commented 8 years ago

OK, your first example:

$ podget -r 1
$ podget

Is a way to avoid this problem. The '-r' flag is short for '--recent' and the count after it is how many items from the feed to download. Every item after that will simply be marked as if it were already downloaded. So this does not prove that the problem doesn't exist.

As for your modified URL, you need to learn to post more information. The tiny tidbits you post make it almost impossible to determine what is happening. For example, "results in:" is not part of podget and so your posting doesn't give me any context to work with. Remember, I cannot see your screen so I need to see more of what I can work with.

And we need to backup here. As I said before, given that Filip and I are getting very similar results what we need to do is determine why your environment or configuration is different. Please compare your podgetrc to one built with the --create-config flag so that I can make my configuration match yours for testing. Until I do that it will be almost impossible to figure out what is causing the error on your end.

dvehrs commented 8 years ago

And quick question, your version statement for sed is different that what Filip or I posted. Did you simply leave out the '(GNU sed)' or are you running a non-GNU version of sed? Is it the busybox version or something like it?

What is your output from ldd for sed?

$ ldd /bin/sed
        linux-vdso.so.1 (0x00007ffd2d7cf000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f6227c28000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6227884000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f6227613000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f622740f000)
        /lib64/ld-linux-x86-64.so.2 (0x00005592798e0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f62271f2000)

stomar commented 8 years ago

I'm not a native speaker, but I wasn't aware that I expressed myself that incomprehensibly. So here another try:

As I wrote, I was able to reproduce the behaviour you described previously, where no error occurs, using my environment, by invoking podget two times without any switches (thereby downloading all the podcasts, except the problematic one due to timeouts).

But originally I did something different: I used the -r switch (I know what it does) on the very first run of podget, to avoid downloading of old podcasts. Then, on the second invocation of podget (without any switches), I got the error.

So I assume it's probably not an environment issue.

Please start from a clean POD directory and try the following two runs of podget in that order:

podget -r 1
podget

The first command adds the problematic podcast URL to the done file, which seems to cause the error in the next run.

As already stated, I both used a custom podgetrc (with filename cleanup turned off) and the default one.

In order to further analyze the produced error message, I locally served a slightly modified RSS file and did the same again. For example, I changed the podcast URL from http://a.blip.tv/api.swf#AYLbuQMA to http://a.blip.tv/api.swf#KLMN. The respective error messages were

sed: -e expression #1, char 28: unknown command: `A'

and

sed: -e expression #1, char 28: unknown command: `K'

So obviously the "unknown command" reported by 'sed' directly corresponds to the character following the # in the podcast URL.

I hope this is clear enough now.

stomar commented 8 years ago

PS. It just occurred to me: while there are no replies to my posts I tend to edit them. So in case you only read the initial emails you might have missed some information.

dvehrs commented 8 years ago

You do not need to use the '-r' flag to avoid downloading old podcasts. Podget records each file it successfully downloads in the 'done' log. When you run Podget again, it compares every enclosure URL with those in that file. If it finds it, it proceeds to the next item. If the URL is not found, then it downloads it.

The purpose of the '-r' flag is to allow you to join a podcast feed without having to download all the old items. This is good for podcasts with date sensitive content like news feeds.

As for your latest actions, I'm afraid they aren't clear. I cannot help with the limited information you are giving me.

Please post your podgetrc file so I can make mine match to be able to test as you are running.

Second, when you post an error, do not post just the last line but rather give me a few lines for context. For an example, compare what you post to what I posted above. I cannot debug this issue remotely with just one error message. Yes it tells me that sed failed but not which of the dozens of sed commands in Podget caused the problem. To determine that I need your config and more text pasted than a single line.

PS: I read your replies on Github so unless you are editing them as I am reading them its unlikely I missed anything.

dvehrs commented 8 years ago

OK I now understand how you were using the '-r' flag. A little unconventional but sure enough it caused the error you described. I'm diving into it.

dvehrs commented 8 years ago

OK, I've found the offending line.

URL_CLEAN=$(echo "${URL_INPUT}" | sed -e "s/([^])/\1\/g")

Not sure how to fix it yet but I need to play with it....

dvehrs commented 8 years ago

OK, I do believe I fixed the problem. I've uploaded my proposed fix to the dev branch on Github.

It turns out the offending line needed to be moved down in the file about 12 lines and changed to:

URL_CLEAN=$(echo "${URL_INPUT}" | sed -e "s${TEST_CHAR}([^])${TEST_CHAR}\1\${TEST_CHAR}g")

The problem was caused because we were using the / character as our sed delimiter and it's common use in URLs. Combine that with the # character and we hit a bug. However, we test for characters not in the URL for the another sed statement and I was able to use that same check to determine a safe character to use as a delimiter in this statement as well.

Like I said the changes have been uploaded to the dev branch for testing.

Thanks for the report and sorry I was so slow understanding how to replicate the error here.

stomar commented 8 years ago

The purpose of the '-r' flag is to allow you to join a podcast feed without having to download all the old items.

(Yes, that's exactly why I used it... to avoid downloading of hundreds of old podcasts that I would never have the time to listen to... too busy filing issues).

Tested the dev branch, works for me :fireworks:

Thanks for the fast fix.

dvehrs commented 8 years ago

OK, fix has been merged into the master branch and tagged as release 0.7.13

I've also posted updated files on Sourceforge and submitted them to Debian for consideration. With any luck they will be included in the repositories soon and make it down to Ubuntu. As for the other distributions, well that's up to their package maintainers. Here's hoping it gets out to all quickly!

Thanks again!

dvehrs / podget

podget aborts with 'sed' error #6