search

KSP-CKAN / CKAN

The Comprehensive Kerbal Archive Network
https://forum.kerbalspaceprogram.com/index.php?/topic/197082-*
Other
1.99k stars 348 forks source link

SourceForge kref #4172

Closed HebaruSan closed 2 months ago

HebaruSan commented 2 months ago

Motivation

Discord user "dan" pointed out that KSP2 mod KSRe (frozen in KSP-CKAN/KSP2-NetKAN#174 after its author deleted it from SpaceDock) has been posting updates at https://sourceforge.net/projects/ksre/:

image

https://sourceforge.net/projects/ksre/files/

Currently Netkan doesn't have the ability to crawl SourceForge, but while its API documentation looks quite inadequate, it does feature RSS-format (Really Simple Syndication, not Real Solar System) feeds for each mod's files:

https://sourceforge.net/projects/ksre/rss

Changes

Now a netkan with $kref: '#/ckan/sourceforge/ksre' will be indexed from SourceForge based on its RSS feed. Each <item> with a title ending in .zip is taken to be a download, and the first one is treated as most recent.

Retrieving and parsing the RSS feed went smoothly thanks to System.ServiceModel.Syndication.SyndicationFeed, but getting a download URL was tricky because SourceForge thought our user agent string looked too browser-like and so wasn't returning redirects, so I had to pass in an alternate user agent string.

Only a few fields can be set based on the minimal metadata in the feed, but luckily the swinfo.json parser is able to handle the rest.

HebaruSan commented 2 months ago

Of course Mono doesn't want to play nice. Of course it doesn't.

image

HebaruSan commented 2 months ago

Apparently SourceForge's download link redirection does dynamic load balancing, so we potentially get a different URL each time:

image

Should figure out a way to stabilize that before we use this more widely...