Song embeds from Bandcamp - Githubissues

VocaDB / vocadb

VocaDB is a Vocaloid Database with translated artists, albums, music videos and more.

https://vocadb.net

Other

335 stars 40 forks source link

Song embeds from Bandcamp #18

Closed riipah closed 5 years ago

riipah commented 9 years ago

Many artists, especially western artists share full songs on bandcamp.

Songs (and albums) from bandcamp can be embedded, but the embed code requires song ID. Bandcamp had an API for retrieving ID from song URL, but they closed that, so probably the only possibility is some ugly HTML parsing similar to piapro, or requiring the user to paste bandcamp embed code instead of song URL. Would also need to make sure that artist information and length are parsed, if possible.

Otherwise there is nothing preventing us from implementing this.

disqonnected commented 5 years ago

Hmm, this is a blocker for adding many songs which would be really nice to be able to add. While scraping is far from optimal, it's worth noting that bandcamp has that information both in HTML elements as well as in the form of data in <script> elements (see var TralbumData).

riipah commented 5 years ago

There are still problems with scraping:

Will Bandcamp allow that? Some websites will ban your IP if they notice that you're scraping their pages.
That TralbumData exists there now, but will it still be there in the same format 6 months from now? I'm already updating the Piapro parser every few months when they make changes. Most of that is HTML parsing, but there's also some values that are from script tags.

I want to emphasize that I want to avoid having more components that might just break on their own because of external factors, unless it's absolutely necessary. I have very limited time to maintain those components.

disqonnected commented 5 years ago

It might not hurt to just ask bandcamp if they mind, or what method they propose. There is in fact a comment in that variable:
```
// For the curious:
// http://bandcamp.com/help/audio_basics#steal
// http://bandcamp.com/terms_of_use
```
They seem to be down-to-earth about it and vocadb is interested in promoting the artists, which is what they are hoping to achieve.
That's understandable.. There is no guarantee that this will keep working. One way is to at least outsource the problem: Applications like youtube-dl seem to put a lot of effort to keep their scrapers up-to-date. (youtube-dl --dump-json [some-bandcamp-song-url] | jq). But of course this would introduce an entire new language to vocadb, which is likely even more problematic.

riipah commented 5 years ago

I guess we could use the NYoutubeDL library, which encapsulates youtube-dl into nuget. I'll see if that can be used.

riipah commented 5 years ago

It seems that the library works, but it has an issue because the duration is set to integer, while it's actually a decimal/floating point. I fixed it in a local build, but I will report the issue.

riipah commented 5 years ago

I reported the problem. The author fixed it, and now it's working fine. I should be able to add Bandcamp embeds with this

riipah commented 5 years ago

Note: only enabled for single songs (/track/) posts for now. Albums require a separate embed code.

riipah commented 5 years ago

Unfortunately it seems that the library is glitchier than I expected. Communicating with a separate executable is apparently difficult. It's working on dev machine, but hanging on the server indefinitely. Gonna take a while longer.

disqonnected commented 5 years ago

Thanks a lot for working on this!

I would had never thought of using a wrapper for something that already spits out json, but that's likely my different language background. Hopefully NYoutubeDL remains maintained as enthusiastically as it is now.

riipah commented 5 years ago

The library isn't working and I've exhausted all my options. Sorry, but if you want this done, you'll have to do it yourself.

disqonnected commented 5 years ago

I appreciate it still. I understand. Is the current problem the fact that it hangs waiting for user input of some sort? Do you know how to reproduce it?

riipah commented 5 years ago

I was able to reproduce it by replacing youtube-dl.exe with another .exe/.cmd file that expects an input. And from the code I can see that it will wait for the process to exit indefinitely. There is an command for manual cancellation, but I think that's only feasible for interactive scenarios. A timeout should be added at least. The main problem though is that I don't know why it's waiting for input.

riipah commented 5 years ago

All I can see is the exit code is 255. Could be related to https://github.com/ytdl-org/youtube-dl/issues/10766 or https://github.com/norkunas/youtube-dl-php/issues/39 . NYoutubeDL by itself is surprisingly complicated library, not to mention youtube-dl. I don't think it's feasible to spend time making it work. In this time I would've written the parser myself, but again, I don't want to be the one to maintain it. It's just frustrating because I wanted to have those Bandcamp embeds too, but it seems more work than it's worth.

riipah commented 5 years ago

It was the Python issue. I reported it, and the maintainer of the NYoutubeDL library added support for python version of youtube-dl, which seems to work for now, although it's very slow.

disqonnected commented 5 years ago

Thanks for the details. I'm afraid I've just wasted too many hours just trying VStudio to work. I'd like to get back to this at some later point, but I am also having trouble finding the time.

disqonnected commented 5 years ago

Oh, never mind, I had not seen the last comment. I hope this doesn't cause any more pain. Thanks a lot for all your time. It's nice to see a long-standing issue be resolved.