search

hikikomoriphoenix / Beedio

Android app that lets you find downloadable videos as you browse the web. Allows queuing downloads. Also includes bookmarking and ad-blocking features for easier browsing experience.
GNU General Public License v2.0
148 stars 44 forks source link

not downloading ts and mp3 files #45

Open dragnishan opened 5 years ago

dragnishan commented 5 years ago

not downloading dailmotion videos and facebook videos

hikikomoriphoenix commented 5 years ago

Hi, thank you for letting me know. I'll work on it as soon as I can.

dragnishan commented 5 years ago

i am developing related projects and i copied from ur code videocontentsearch,class files are not supporting other type of videos Eg;- mpeg,ts,mp3 ` try { String host; host = new URL(page).getHost(); if (host.contains("twitter.com") || host.contains("metacafe.com") || host.contains ("myspace.com")) { InputStream in = uCon.getInputStream(); InputStreamReader inReader = new InputStreamReader(in); BufferedReader buffReader = new BufferedReader(inReader); String line; String prefix = null; String type = null; String name = "video"; String website = null; if (title != null) { name = title; } if (host.contains("twitter.com")) { prefix = "https://video.twimg.com"; type = "ts"; website = "twitter.com"; } else if (host.contains("metacafe.com")) { String link = uCon.getURL().toString(); prefix = link.substring(0, link.lastIndexOf("/") + 1); website = "metacafe.com"; type = "mp4"; } else if (host.contains("myspace.com")) { String link = uCon.getURL().toString(); website = "myspace.com"; type = "ts";

                onVideoFound(null, type, link, name, page, true, website);
                String videoFound = "name:" + name + "\n" +
                        "link:" + link + "\n" +
                        "type:" + contentType + "\n" +
                        "size: null";
                Log.i(TAG, videoFound);
                return;
            }
            while ((line = buffReader.readLine()) != null) {
                if (line.endsWith(".m3u8")) {
                    String link = prefix + line;
                    onVideoFound(null, type, link, name, page, true, website);
                    String videoFound = "name:" + name + "\n" +
                            "link:" + link + "\n" +
                            "type:" + contentType + "\n" +
                            "size: null";
                    Log.i(TAG, videoFound);
                }
            }
        } else {
            Log.i("loremarTest", "Content type is " + contentType + " but site is not " +
                    "supported");
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}`

please help to automatically detect the mp3 and other types

hikikomoriphoenix commented 5 years ago

Try this for mp3 after getting content type from response headers:

if (contentType.equals("audio/mpeg"))

Or try audio/mpeg3 or whatever you see in this link -> https://stackoverflow.com/questions/12017694/content-type-for-mp3-download-response

For other formats, basically you'll have to google "content type for (whatever format you need)". Also you need to play the media that you are trying to download on the web page.

As for dailymotion, I tried it and it works? I was able to download 'ts' video files. As for facebook, probably needs some update. I'm planning on developing version 2. I'll include updates on that version. Also I am not aware that users can play mp3 audio files on this sites. Can you confirm that its true?

dragnishan commented 5 years ago

yes mp3 files also can play this site. and mp3 also cannot download its segment more parts like daily motion .ts files it can't be download one files its segmented. do u have any idea about this. and can u send ur any online chat information like gmail or whats app or Skype. its easy to clear our problems.

waiting for good responce

hikikomoriphoenix commented 5 years ago

What I would usually do is log the urls from onLoadResource like this:

webView.setWebViewClient(new WebViewClient() {
    @Override
    public void onLoadResource(final WebView view, final String url) {
        Log.v("MyTag", url);
    }
}

Look for the suspected urls for the mp3. Find the part of the url which constantly changes in each url. Usually it indicates which chunk of the file is being streamed. Delete that part and try to download it. Perhaps you can download the whole file this way. You can use replaceAll() for deleting like this:

urlString.replaceAll(regexPatternOfChunk, "");

If not, you'll have to download each chunk and append to target file. Make sure you set true for append in FileOutputStream constructor.

FileOutputStream out = new FileOutputStream(videoFile, true);
out.write(byteArrayOfChunkYouReadFromInputstream);

Also, you can just save one url from onLoadResource by selecting one url and then replace the changing part with something like "CHUNK" or "SEGMENT". When you download, restore it to original, replacing the number with the number of the first chunk, download the chunk , and then increment the number. Repeat until you get a FileNotFoundException when attempting to open connection with url. See how I did it in handleChunkedDownload.

You'll have to delete the Log command of course. You should capture the url using a keyword like "audio" or "mp3" or whatever that allows you to skip other unimportant urls.

dragnishan commented 5 years ago

Screenshot_2019-05-06-01-15-31 This is the problem for twiter videos , same as facebook do u have any ideas , i try so many ways. and original videos is segmented. how to solved

hikikomoriphoenix commented 5 years ago

I fixed it. New Release Here. Changes in the code:

// Skip twitter video chunks.
if (host.contains("twitter.com") && contentType.equals("video/mp2t")) {
    return;
}

and

            } else if (host.contains("facebook.com") && link.contains("bytestart")) {
                int b = link.lastIndexOf("&bytestart");
                int f = link.indexOf("fbcdn");
                if (b > 0) {
                    link = "https://video.xx." + link.substring(f, b);
                }
                URLConnection fbCon;
                fbCon = new URL(link).openConnection();
                fbCon.connect();
                size = fbCon.getHeaderField("content-length");
                website = "facebook.com";
            }

Twitter uses m3u8 file which lists the segments it will play. I fixed it so that it will only detect the m3u8 file and skip detecting segments. The link without indicated total size is the correct link to download whole video file. To download the whole file, you have to download each segment listed in the m3u8 file.

As for facebook, it seems like they changed to streaming segmented videos which was not the case before if I remember correctly. With the help of a video downloader in my desktop, I was able to deduce that I should replace the subdomain and remove the bytestart and byteend query fields in the link to download the whole file. Unfortunately, other videos in the page are still detected. So It requires a bit of luck to find the correct video.

dragnishan commented 5 years ago

ohh thanku but same problem occur to other website , i would like to download gaana.com mp3 songs but same segment , can i know the detail for the this code for skip the chunks

   } else if (host.contains("facebook.com") && link.contains("bytestart")) {
                int b = link.lastIndexOf("&bytestart");
                int f = link.indexOf("fbcdn");
                if (b > 0) {
                    link = "https://video.xx." + link.substring(f, b);
                }
                URLConnection fbCon;
                fbCon = new URL(link).openConnection();
                fbCon.connect();
                size = fbCon.getHeaderField("content-length");
                website = "facebook.com";
            }

because i need to download the ganaa.com

looking for the helping and any way thanks for the code

hikikomoriphoenix commented 5 years ago

So what I did was, I used the Flash Video Downloader extension in Chrome Browser. Went to a facebook page with a streaming video. Copied the link address of the video from Flash Video Downloader and pasted it in a text editor. I ran my app in android studio, and went to facebook page with video. I had my onLoadResource in webview client log all the urls. I saw that the urls for segments are similar to the link address I had from Flash Video Downloader except for the subdomain. So to get the link to the full video, I changed the subdomain to the one similar to link from Flash Video Downloader and removed the bytestart and byteend query fields that was placed at the end of each links.

because i need to download the ganaa.com

Please paste the logs you get from onLoadResource. Also it would help if you log the content type of each. To do this setup a webview and have your webview client override onLoadResource. Log the url. Open connection. Get content type and log it.

webView.setWebViewClient(new WebViewClient() {
    @Override
    public void onLoadResource(final WebView view, final String url) {
        Log.v("MyTag", "URL = " + url);
        URLConnection uCon = null;
        try {
            uCon = new URL(url).openConnection();
            uCon.connect();
        } catch (IOException e) {
            e.printStackTrace();
        }
        if (uCon != null) {
            String contentType = uCon.getHeaderField("content-type");
            if (contentType != null) {
                Log.v("MyTag", "Content type = " + contentType);
            } else {
                Log.v("MyTag", "Content type = unknown");
            }
        } else {
            Log.v("MyTag", "No connection");
        }
    }
}

Run the app. Go to ganaa.com and play an mp3. Go to logcat. Filter with your tag. Copy all logs and paste it here. Hopefully we get the links to these segments and see what can be done to get the url for full mp3. Or wether we need to download each segment one by one.

Also, can you also list all the other video streaming sites you're using. No porn sites pls. Thank u.

dragnishan commented 5 years ago 
   if (host.contains("gaana.com") && contentType.equals("audio/mpeg")) {
                return;
            }
05-09 02:04:24.660 22828-24721/smarttube.downloader I/loremarTest: name:Download Latest MP3 Songs Online: Play Old & New MP3 Music Online Free on Gaana.com
    link:https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment1_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
    type:video/mp2t
    size:132916
05-09 02:04:25.890 22828-24728/smarttube.downloader I/loremarTest: retreiving headers from https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment1_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
05-09 02:04:26.040 22828-24731/smarttube.downloader I/loremarTest: retreiving headers from https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment1_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
05-09 02:04:26.420 22828-24728/smarttube.downloader I/loremarTest: name:Download Latest MP3 Songs Online: Play Old & New MP3 Music Online Free on Gaana.com
    link:https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment1_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
    type:video/mp2t
    size:132916
05-09 02:04:26.800 22828-24731/smarttube.downloader I/loremarTest: name:Download Latest MP3 Songs Online: Play Old & New MP3 Music Online Free on Gaana.com
    link:https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment1_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
    type:video/mp2t
    size:132916
05-09 02:04:27.400 22828-24745/smarttube.downloader I/loremarTest: retreiving headers from https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment1_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
05-09 02:04:28.240 22828-24753/smarttube.downloader I/loremarTest: retreiving headers from https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment2_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
05-09 02:04:28.360 22828-24745/smarttube.downloader I/loremarTest: name:Download Latest MP3 Songs Online: Play Old & New MP3 Music Online Free on Gaana.com
    link:https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment1_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
    type:video/mp2t
    size:132916
05-09 02:04:29.300 22828-24753/smarttube.downloader I/loremarTest: name:Download Latest MP3 Songs Online: Play Old & New MP3 Music Online Free on Gaana.com
    link:https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment2_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
    type:video/mp2t
    size:153972
05-09 02:04:34.440 22828-24811/smarttube.downloader I/loremarTest: retreiving headers from https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment3_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
05-09 02:04:34.760 22828-24820/smarttube.downloader I/loremarTest: retreiving headers from https://cdn.ampproject.org/rtv/011905021827420/amp4ads-v0.js
05-09 02:04:34.780 22828-24826/smarttube.downloader I/loremarTest: retreiving headers from https://fonts.googleapis.com/css?family=Roboto:300|Roboto:400,500&lang=en
05-09 02:04:34.790 22828-24827/smarttube.downloader I/loremarTest: retreiving headers from https://fonts.googleapis.com/css?family=Roboto:400,500&text=
05-09 02:04:34.830 22828-24830/smarttube.downloader I/loremarTest: retreiving headers from https://tpc.googlesyndication.com/simgad/15397497175856570510/downsize_200k_v1?sqp=4sqPyQSTAUKQAQgAEhQNzczMPhUAAABAHQAAAAAlAAAAABgAIgoNAACAPxUAAIA_Kk8ISxABHQAAtEIgACgBMAY4A0CAwtcvSAFQAFgAYEtwAngAgAEAiAEAkAEAnQEAAIA_oAEAqAEAsAGAreIEuAH___________8BxQEtsp0-MhcIigEQSBgBIAEtAAAAPzBcODBFAADAPw&rs=AOga4qltAA9hOYWTBsNRF1bXfCzAMLEnuA
05-09 02:04:34.840 22828-24831/smarttube.downloader I/loremarTest: retreiving headers from https://securepubads.g.doubleclick.net/pagead/adview?ai=CRCjuWT3TXNa7NJeLvgTlrLCIDJuVqqlW5pL-rZcJxPChlt0UEAEgodOsaWCRgewEoAG79ar0AsgBBqkCvt3-9ebNTj7gAgCoAwHIAwqqBMkBT9C62U9mi7bSdjguyySidKvp6CW5sZuQ8vsyHfPTr6IXKsh83z2A1f9mxlQ31jRGleNAXNmR8lvn2cke2dCHiNXPFrul8K2M2N2FNFbA0TWrQP5SkgsouoWnQ7OJXBtz2WMSYbCkG57cLpUzOtBav1mm1IAN9x1A4vakgRXbrgw1k9KtZI_2xm9mbQmN1_GwD37cjPI3LuEM3nLseafDsivT_u3NnMksxULiJrzKvSN--E_KeT4uLdmQhb0BGVeg0OcIFHFo7SBcwATHjei3gwLgBAGIBcbexuQGkgUECAQYAZIFBAgFGASSBQQIBRgYkgUFCAUYqAGgBjfYBgKAB97m1YsBqAeOzhuoB9XJG6gH4NMbqAe6BqgH2csbqAfPzBuoB6a-G9gHAfIHBBCKpASgCP_RPbAIAtIIBwiAYRABGA2ACgPYEwKCFAsaCWdhYW5hLmNvbQ&sigh=RArXFw31_Lw&template_id=492&tpd=AGWhJmu5Rd-Gos1QmnRMSBY7VMNfRRfJiXAH6sX7tsm7b2i_Qw
05-09 02:04:35.220 22828-24811/smarttube.downloader I/loremarTest: name:Download Latest MP3 Songs Online: Play Old & New MP3 Music Online Free on Gaana.com
    link:https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment3_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
    type:video/mp2t
    size:190068
05-09 02:04:35.740 22828-24820/smarttube.downloader I/loremarTest: Not a video. Content type = text/javascript
05-09 02:04:36.210 22828-24830/smarttube.downloader I/loremarTest: Not a video. Content type = image/jpeg
05-09 02:04:36.280 22828-24826/smarttube.downloader I/loremarTest: Not a video. Content type = text/css; charset=utf-8
05-09 02:04:36.280 22828-24827/smarttube.downloader I/loremarTest: Not a video. Content type = text/css; charset=utf-8
05-09 02:04:36.740 22828-24831/smarttube.downloader I/loremarTest: Not a video. Content type = text/html; charset=iso-8859-1
05-09 02:04:44.450 22828-24884/smarttube.downloader I/loremarTest: retreiving headers from https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment4_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
05-09 02:04:45.270 22828-24884/smarttube.downloader I/loremarTest: name:Download Latest MP3 Songs Online: Play Old & New MP3 Music Online Free on Gaana.com
    link:https://vodhls-vh.akamaihd.net/i/songs/88/1821988/20947109/20947109_128.mp4/segment4_0_a.ts?set-akamai-hls-revision=5&hdntl=exp=1557434064~acl=/i/songs/88/1821988/20947109/20947109_128.mp4/*~data=hdntl~hmac=c67e17de84ebad457aa4d456ed2846eb5e7279de0327c392471f0f30e0fe0ad6
    type:video/mp2t
    size:153972

this logcat i try for the ganna but not working , any idea that content type type:video/mp2t but that not mp3 any idea

hikikomoriphoenix commented 5 years ago

You can try to add more keywords in res/values/videourl_filters.xml.

<resources>
    <string-array name="videourl_filters">
        <item>mp4</item>
        <item>video</item>
        <item>googleusercontent</item>
        <item>embed</item>
        <item>mp3</item>
        <item>audio</item>
        <item>mpeg</item>
<!-- Insert more filters here -->
    </string-array>
</resources>

Make sure you have this code:

        String urlLowerCase = url.toLowerCase();
        String[] filters = context.getResources().getStringArray(R.array.videourl_filters);
        boolean urlMightBeVideo = false;
        for (String filter : filters) {
            if (urlLowerCase.contains(filter)) {
                urlMightBeVideo = true;
                break;
            }
        }
        if (urlMightBeVideo) {
            // Do something here.
        }

Also add more content type:

if (
    host.contains("gaana.com") 
    && (
        contentType.equals("audio/mpeg")
        || contentType.equals("audio/mpeg3")
        || contentType.equals("audio/x-mpeg-3")
   )
)

Also, like I said before you could try to log all urls from onLoadResource() to get more idea how to fetch url for mp3.

Also , do not forget to play the mp3 itself in the page for anything of these to even work.

If nothing works, then I can not help you since my knowledge is also limited as well.