Can't add podcast from soundcloud

[Zastin]

I try add three different soundcloud podcast and none work. Year ago one of them work gpodder app in Jolla 1. Now I have sailfish X.

Here is terminal log where I try to add one of them:

2017-11-04 07:06:05,201 [gpodder.util] DEBUG: run_in_background: <function run_in_background_thread..wrapper.<l ocals>. at 0xef386660> (False) 2017-11-04 07:06:05,203 [gpodder.model] INFO: Expanding prefix sc:kaarle -> http://soundcloud.com/kaarle 2017-11-04 07:06:05,204 [gpodder.util] DEBUG: run_in_background: <function run_in_background_thread..wrapper.<l ocals>. at 0xec5f2f60> (False) 2017-11-04 07:06:05,206 [gpodder.model] INFO: Updating download_folder of http://soundcloud.com/kaarle to http_soundc loud.com_kaarle 2017-11-04 07:06:05,208 [minidb] DEBUG: INSERT INTO PodcastChannel (title, url, link, description, cover_url, auth_user name, auth_password, http_last_modified, http_etag, auto_archive_episodes, download_folder, pause_subscription, section , payment_url, downloadstrategy) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ['', 'http://soundcloud.com/kaar le', '', '', None, '', '', None, None, None, 'httpsoundcloud.com_kaarle', None, 'other', None, 0] 2017-11-04 07:06:05,216 [minidb] DEBUG: SELECT FROM PodcastChannel WHERE id = ? [4] 2017-11-04 07:06:05,220 [gpodder.registry] INFO: feed_handler resolved by soundcloud_feed_handler from gpodder.plugins. soundcloud: None -> <gpodder.plugins.soundcloud.SoundcloudFeed object at 0xecd46990> 2017-11-04 07:06:05,400 [minidb] DEBUG: SELECT FROM PodcastChannel WHERE id = ? [4] 2017-11-04 07:06:05,401 [minidb] DEBUG: PodcastChannel(id=4) title: -> kaarle on Soundcloud 2017-11-04 07:06:05,403 [minidb] DEBUG: PodcastChannel(id=4) link: -> http://soundcloud.com/kaarle 2017-11-04 07:06:05,404 [minidb] DEBUG: PodcastChannel(id=4) description: -> Tracks published by kaarle on Soundcloud. 2017-11-04 07:06:05,405 [minidb] DEBUG: PodcastChannel(id=4) cover_url: None -> https://i1.sndcdn.com/avatars-000271856 218-4j5icc-large.jpg 2017-11-04 07:06:05,406 [minidb] DEBUG: UPDATE PodcastChannel SET title=?, link=?, description=?, coverurl=? WHERE id = ? ['kaarle on Soundcloud', 'http://soundcloud.com/kaarle', 'Tracks published by kaarle on Soundcloud.', 'https://i1.s ndcdn.com/avatars-000271856218-4j5icc-large.jpg', 4] 2017-11-04 07:06:05,538 [gpodder.model] DEBUG: Fetch failed. Removing buggy feed. Traceback (most recent call last): File "/usr/share/harbour-org.gpodder.sailfish/gpodder/model.py", line 619, in load tmp.update() File "/usr/share/harbour-org.gpodder.sailfish/gpodder/model.py", line 743, in update self._consume_custom_feed(result) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/model.py", line 684, in _consume_custom_feed new_episodes, seen_guids = custom_feed.get_new_episodes(self) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/plugins/soundcloud.py", line 163, in get_new_episodes return self._get_new_episodes(channel, 'tracks') File "/usr/share/harbour-org.gpodder.sailfish/gpodder/plugins/soundcloud.py", line 166, in _get_new_episodes tracks = [t for t in self.sc_user.get_tracks(track_type)] File "/usr/share/harbour-org.gpodder.sailfish/gpodder/plugins/soundcloud.py", line 166, in tracks = [t for t in self.sc_user.get_tracks(track_type)] File "/usr/share/harbour-org.gpodder.sailfish/gpodder/plugins/soundcloud.py", line 113, in get_tracks tracks = (track for track in util.read_json(json_url) if track['downloadable']) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/util.py", line 856, in read_json return json.loads(urlopen(url).read().decode('utf-8')) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/util.py", line 690, in urlopen return opener.open(request) File "/usr/lib/python3.4/urllib/request.py", line 469, in open response = meth(req, response) File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.4/urllib/request.py", line 507, in error return self._call_chain(args) File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain result = func(args) File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found 2017-11-04 07:06:05,545 [minidb] DEBUG: DELETE FROM PodcastEpisode WHERE podcast_id = ? [4] 2017-11-04 07:06:05,551 [minidb] DEBUG: DELETE FROM PodcastChannel WHERE id = ? [4] Exception in thread Thread-5: Traceback (most recent call last): File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner self.run() File "/usr/lib/python3.4/threading.py", line 868, in run self._target(*self._args, *self._kwargs) File "/usr/share/harbour-org.gpodder.sailfish/main.py", line 56, in util.run_in_background(lambda: f(args)) File "/usr/share/harbour-org.gpodder.sailfish/main.py", line 233, in subscribe self.core.model.load_podcast(url, create=True) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/model.py", line 942, in loadpodcast return self.PodcastClass.load(self, url, create, authenticationtokens) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/model.py", line 619, in load tmp.update() File "/usr/share/harbour-org.gpodder.sailfish/gpodder/model.py", line 743, in update self._consume_custom_feed(result) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/model.py", line 684, in _consume_custom_feed new_episodes, seen_guids = custom_feed.get_new_episodes(self) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/plugins/soundcloud.py", line 163, in get_new_episodes return self._get_new_episodes(channel, 'tracks') File "/usr/share/harbour-org.gpodder.sailfish/gpodder/plugins/soundcloud.py", line 166, in _get_new_episodes tracks = [t for t in self.sc_user.get_tracks(track_type)] File "/usr/share/harbour-org.gpodder.sailfish/gpodder/plugins/soundcloud.py", line 166, in tracks = [t for t in self.sc_user.get_tracks(track_type)] File "/usr/share/harbour-org.gpodder.sailfish/gpodder/plugins/soundcloud.py", line 113, in get_tracks tracks = (track for track in util.read_json(json_url) if track['downloadable']) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/util.py", line 856, in read_json return json.loads(urlopen(url).read().decode('utf-8')) File "/usr/share/harbour-org.gpodder.sailfish/gpodder/util.py", line 690, in urlopen return opener.open(request) File "/usr/lib/python3.4/urllib/request.py", line 469, in open response = meth(req, response) File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.4/urllib/request.py", line 507, in error return self._call_chain(args) File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain result = func(args) File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found

[Keeper-of-the-Keys]

I am playing a but with this, I updated all API references in soundcloud.py to be to https but that does not solve it.

As far as I can tell the permalink that is returned is not to an RSS/Atom feed which is why podcastparser blows up on this, for my feeds I usually just subscribe through their iTunes address but of course it would be nicer if this worked.

I also checked and the soundcloud plugin in the main gpodder (3.10) certainly functions properly, I have tried just replacing soundcloud.py with the newer version but that did not work now I am trying to go over the diff.

[Keeper-of-the-Keys]

@elelay Is this functionality working in the main gpodder? can you point me to what function I likely need to backport?

[elelay]

As you might already have found it's https://github.com/gpodder/gpodder/blob/master/src/gpodder/plugins/soundcloud.py Track lists are json. First retrieve the uri from 'https://api.soundcloud.com/users/transalp.json?consumer_key=zrweghtEtnZLpXf3mlm8mQ' Then track list from 'https://api.soundcloud.com/users/2237/tracks.json?filter=downloadable&consumer_key=zrweghtEtnZLpXf3mlm8mQ&limit=200'

[Keeper-of-the-Keys]

Minor update - I have partially back ported the newer version of the plugin, the result at the moment is a plugin that loads some but not all episodes and sets some fields wrong, but it is promising.

I currently have only one podcast hosted there, so any suggestions for other test cases are very welcome.

[carmenfdezb]

Minor update - I have partially back ported the newer version of the plugin, the result at the moment is a plugin that loads some but not all episodes and sets some fields wrong, but it is promising.

I currently have only one podcast hosted there, so any suggestions for other test cases are very welcome.

Hi @Keeper-of-the-Keys, how can I test updated plugin?

[Keeper-of-the-Keys]

Haven't pushed it yet b/c it's kind too ugly right now. Hope that it's good enough to be pushed today.

[Keeper-of-the-Keys]

Still ugly but initial efforts can be seen here: https://github.com/Keeper-of-the-Keys/gpodder-core/tree/soundcloud-update

ATM it only loads 7/35 episodes, I'm not sure why

Also it would probably be good to implement the cacheing seen upstream.

[sfbg]

@Keeper-of-the-Keys just an uneducated idea, but wouldn't it be most efficient to try to backport the working Soundcloud implementation from gpodder/gpodder (i.e. 3.10.9) into gpodder/gpodder-core (4.x)

Not only for soundcloud but foor the entire core. There were quite a few fixes and changes in the 3.x branch, that are not present in the practically abandoned till now 4.x branch..

Edit: OK, this is exactly what you seem to be doing, sorry... :/

[Keeper-of-the-Keys]

ATM I'm trying not to backport all core changes, that seems like too big a taks (especially considering I am still familiarizing myzelf with the codebase).

So I'm just tryingto get the changes in the soundcloud plugin backported

[sfbg]

Absolutely, you are doing it the right way. Step by step. Sorry for speaking without reading first on my side. Anyway, I am really happy that gPodder for Sailfish is resurrected. It's been a long winter sleep ;)

[carmenfdezb]

Hi @Keeper-of-the-Keys! I was testing with kaarle podcast: https://soundcloud.com/kaarle. For me, it works fine, I get the same episodes than gpodder desktop (87 episodes), and I don't get any errors.

[Keeper-of-the-Keys]

Weird for Medic Mindset it keeps giving me only 7 of the 36 episodes that the API returns...

Edit - the problem seems to be the "downloadable" flag, what's weird about that is that the API ignores the filter and returns 36 but I guess the plugin is still checking for it and not all episodes are flagged downloadable. Will need to check how this affects play ability.

[sfbg]

Tested Medic Mindset on the latest GPodder Desktop release. It also lists just 7 episodes. So, I'd say your backport has been successful. If there is an issue, it is not Sailfish (or 4.x branch) specific.

[Keeper-of-the-Keys]

OK, so I think if that is the case then the whether or not the downloadable flag really matters and other optimization of the plugin should probably be a separate issue and a separate PR?

Everything was behaving as expected in all your tests?

(I ask because I only backported a part of the newer plugin so it seems highly likely that stuff may not work properly...)

[sfbg]

I'd say push it to upstream. It is definitely much better then not working at all. For the more general issue, this has to be dealt with within gpodder/gpodder

[elelay]

the problem seems to be the "downloadable" flag, what's weird about that is that the API ignores the filter and returns 36 but I guess the plugin is still checking for it and not all episodes are flagged downloadable. Will need to check how this affects play ability.

indeed, even if a track is marked downloadable we prefer to stream it (128k mp3). Maybe we should check streamable instead of downloadable (tried on kaarle tracks as well as https://soundcloud.com/thisissomenoise-893392236).

Reference: https://developers.soundcloud.com/docs/api/reference#tracks not that explicit

[Keeper-of-the-Keys]

So it seems that the actual url used is always the stream URL and that regardless of the downloadable flag we can download episodes. Also the performance of using the Soundcloud RSS feed is way better then the soundcloud.py (this is how I was/am subscribed to Medic Mindset it turns out).

Considering this maybe the soundcloud plugin should just search for the user and then have the rss feed be parsed by podcast parser?

[Keeper-of-the-Keys]

The RSS feed used by Medic Mindset: http://feeds.soundcloud.com/users/soundcloud:users:246565416/sounds.rss

This would suggest that once the search plugin knows the ID of whatever stream the user searched it should be able to hand off http://feeds.soundcloud.com/users/soundcloud:users:${id}/sounds.rss to podcastparser.

Of course this may be considered "misbehavior", I don't know what the conventions are in podcast consumer land...

[sfbg]

Considering this maybe the soundcloud plugin should just search for the user and then have the rss feed be parsed by podcast parser?

This makes perfect sense and is the solution closest to the standard - using RSS to subscribe to and download podcasts.

[elelay]

soundcloud feeds are great! I didn't know they existed. It seems it's something the content producer enables manually

[elelay]

Medic Mindset and Kaarle (http://feeds.soundcloud.com/users/soundcloud:users:4624400/sounds.rss) did it but https://soundcloud.com/ssshheee didn't: http://feeds.soundcloud.com/users/soundcloud:users:4022380/sounds.rss

[Keeper-of-the-Keys]

@elelay Now I finally understand what was going on - Soundcloud didn't work so I used iTunes links instead to add podcasts, these were parsed and the RSS address was extracted from them, the RSS actually points to wherever the podcast owner told it to point.

Last time I checked iTunes stopped working (somewhere shortly after they switched to podcasts.apple), another thing to look into at some point.

[Keeper-of-the-Keys]

Sadly there does not seem to be a field in the API that indicates whether or not the RSS feed is active.

[elelay]

We prefer the stream url to the download url, so we should check if either the downloadable or the streamable flag is set. User can disable streaming like this: https://help.soundcloud.com/hc/en-us/articles/115003564608-Disabling-app-playback-for-your-track

[sfbg]

Just tested it after the PR was merged. Finding Soundcloud podcasts works super fast, but...

Updating for new episodes is unfortunately way too slow. The podcast I tested it with has 103 episodes and it took full 2 minutes to check for new episodes for that podcast alone. It seems to go through each episode from the first one and takes more than a second per episode:

2019-08-18 23:11:10,919 [gpodder.plugins.soundcloud] DEBUG: get_user_info url: https://api.soundcloud.com/users/thekanoha.json?consumer_key=zrweghtEtnZLpXf3mlm8mQ
2019-08-18 23:11:11,110 [gpodder.plugins.soundcloud] DEBUG: get_tracks url: https://api.soundcloud.com/users/299747716/tracks.json?filter=downloadable&consumer_key=zrweghtEtnZLpXf3mlm8mQ&limit=200
2019-08-18 23:11:11,626 [gpodder.plugins.soundcloud] DEBUG: track in tracks url: https://api.soundcloud.com/tracks/658688249/stream?consumer_key=zrweghtEtnZLpXf3mlm8mQ
2019-08-18 23:11:12,849 [gpodder.plugins.soundcloud] DEBUG: track in tracks url: https://api.soundcloud.com/tracks/655696373/stream?consumer_key=zrweghtEtnZLpXf3mlm8mQ
....
2019-08-18 23:13:14,431 [gpodder.plugins.soundcloud] DEBUG: track in tracks url: https://api.soundcloud.com/tracks/318329433/stream?consumer_key=zrweghtEtnZLpXf3mlm8mQ
2019-08-18 23:13:15,653 [gpodder.plugins.soundcloud] DEBUG: track in tracks url: https://api.soundcloud.com/tracks/316369793/stream?consumer_key=zrweghtEtnZLpXf3mlm8mQ

It is practically unusable for that podcast. I still think that using the RSS feed is the cleaner solution. If the author has not activated it, then it is not really a podcast but a collection on Soundcloud.

[Keeper-of-the-Keys]

@sfbg I agree but that also places us in a weird position, should the plugin initially offer the option to us one or the other option? Should it run a detection on both methods and then report to the user RSS shows X episodes while API shows Y episodes if X!=Y and let the user decide? And then they decided to use RSS but the publisher decides (or makes a mistake) not to publish something over the RSS now what?

This is why I left it like this for now plus that I still need to look at how the other code works to see how an RSS feed could be handed off and lastly if a decision about RSS/API needs to be made here this again places UI code in a place where UI code is not desirable.

[elelay]

In gPodder desktop we use a cache (a simple json file) to make it faster to retrieve metadata on seen episodes. This speeds up a lot + saves api calls.

See https://github.com/gpodder/gpodder/blob/master/src/gpodder/plugins/soundcloud.py#L99

[Keeper-of-the-Keys]

If the podcasts are already in the local db shouldn't that be available as 'cache'?

[elelay]

Good point: it is totally doable for episodes: one could match episodes to tracks by url, then reuse the filetype and length from the episode. Now what about userinfo (esp. user_id), that is not saved in the channel in database? At least it should be saved in the SoundcloudUser object so that it's not fetched when we ask for the cover art then the title (it's cheaper when you have the json cache: a straight string concatenation and dict lookup)...

[Keeper-of-the-Keys]

This was now pushed to Jolla harbour and will be pushed to OpenRepos later this week so closing :)