johanneszab / TumblThree

A Tumblr Blog Backup Application
https://www.jzab.de/content/tumblthree
MIT License
922 stars 130 forks source link

Some videos are differently serialized in the tumblr api data. #239

Open ghost opened 6 years ago

ghost commented 6 years ago

Within the TumblThree application, new videos don't seem to be downloading. They're resolved as just .mp4

More details here: https://pastebin.com/eNe0T2PD

johanneszab commented 6 years ago

Perfectly downloads for me (except that the video count isn't right). The three videos you mention in the pastebin seem to be downloaded for me.

Default settings, I only turned off photo/image downloading to speed up the download.

Edit: Maybe it's a locale issue because you mentioned that for you the files are saved only as ".mp4". I don't know. I guess this will be hard for me to fix because I cannot reproduce it. Can you download any videos in any other blog at all?

johanneszab commented 6 years ago

Regarding your concern about the blog and not posting it here:

I think I can clean up all the post with NSFW material a week after the issues are resolved. Or maybe we should mark it in the title or add a tag? I don't know. But sometimes it's kinda annoying to ask again and again for error examples, even though I can understand that people don't want to post this publicly. But thanks for sharing your issue and the way you did it was perfectly fine for me.

ghost commented 6 years ago

Er.. there ARE six videos. The video count is correct. None of those three videos are the one I posted in the pastebin.

If you watch the video in the link, and the three videos you downloaded, you'll notice the three you downloaded do not correspond to the video in the link.

johanneszab commented 6 years ago

Ah, I see. I'll check the data from the api manually then. It's only 76 posts, shouldn't be too much work. Thanks for the clarification!

johanneszab commented 6 years ago

It's as usual with the messy data from Tumblr.

For some reason the necessary data is not in video-source but in video-player. It looks like this the case if the video is originally from twitter (or linked from twitter). It's still hosted on tumblr as well, thus we can grab it.

One video seems to be hosted on youtube. There is currently nothing implemented to get videos from youtube. I'm sure there is a library for that we could use though.

ghost commented 6 years ago

youtube-dl is public domain: https://github.com/rg3/youtube-dl/blob/master/LICENSE

johanneszab commented 6 years ago

Sure, I do know youtube-dl, but that's written in Python. We'd need something in C#. There is a library, I just haven't had time to test it yet.

Hrxn commented 6 years ago

I can't say how feasible this idea is, but given that youtube-dl is distributed in form of standalone executable binaries, it should be possible to utilize it via command-line interface, C# should support PowerShell out of the box, I assume..

johanneszab commented 6 years ago

That's not the point. The point is that everybody would have to install a python interpreter. That's something I don't want to have in this project. For some people is already to hard to extract the .zip file. I've implemented a whole lot of stuff by my own in this project, hence the binary size is only about ~300kb. Sure, you can add everything from everybody, deploy a 15MB file that has to download python, node.js, etc., but that's not something I'm keen of. You'd also have to take care of updates of each of those libraries and constantly check for important updates.

And as I've said, there is a youtube library written in C#. Someone just has to test it, figure out how to use it, grab the video urls, and implement everything. And that surely takes some days of spare time.

Hrxn commented 6 years ago

Not sure what you mean with Python interpreter here.. The youtube-dl standalone executable for Windows, for example, is a ~ 7.5 MiB exefile that already has a Python runtime baked in. I'm just saying that if you have youtube-dl (or similar) in your PATH, you can easily access them from everywhere, other programs included, and you don't have to include update checks or something, because that is the responsibility of the system, not your program. But you're right, there are libs for Youtube in C#, of course, so this would work as well..

ghost commented 6 years ago

jeez what on earth did this discussion turn into?

youtube-dl is public domain. That means you can just use the already compiled binary, you don't have to take snippets out from the code. It's a single executable and it's only 1.59MB.

When TumblThree needs to use it, simply call the binary.

ghost commented 6 years ago

I mean, unless you want to do a bunch of work for no reason when there's an Unlicense tool that has already done all the work for you.

And I really recommend you just use it because youtube changes their backend a LOT

Hrxn commented 6 years ago

it's only 1.59MB.

Not including Python runtime environment..

johanneszab commented 6 years ago

jeez what on earth did this discussion turn into?

@Aidolii: I don't know what exactly your problem is, but I've said here now three times that there is a youtube C# library. If you look in my repositories, I've a project that is licensed as public domain.

I mean, unless you want to do a bunch of work for no reason when there's an Unlicense tool that has already done all the work for you.

I think I don't need a lecture about open source by someone having contributed 0 to this project, whereas I've already spent several weeks/months into exactly this, an open source project.

ghost commented 6 years ago

Not including Python runtime environment..

youtube-dl's compiled binary does not require python

I don't know what exactly your problem is

I feel like I should be asking you that question, I don't know why you think I'm attacking you. I was trying to help for goodness sake.

Hrxn commented 6 years ago

youtube-dl's compiled binary (for Windows) has the Python interpreter built in... (The size difference makes it kinda obvious)

ghost commented 6 years ago

Yeah that's what I meant. Anyway. I'm out, was just trying to help and all that.