Is there any way to accelerate the downloads?

RobbieArmz commented 6 years ago

I was wondering if theres any way to speed up the download, cause I've been using it on my app, but the speed hardly goes past 300kb/s and I have 5mb/s link. Also its not me, users have reported the same thing. But when they use apps such as ADM they download it at maximum speed.

Just wondering how to force the download to use maximum speeds.(if there is such thing)

tonyofrancis commented 6 years ago

@RobbieArmz What version of Fetch are you using?. What does your Fetch Builder configuration look like? Are you using the OkHttpDownloader or HttpUrlConnectionDownloader. How many downloads are being downloaded in parallel? Once I have this information I can better help.

Also if you try the sample app. Do you get faster speeds there?

RobbieArmz commented 6 years ago

The build im using is this: com.tonyodev.fetch2:fetch2:2.0.0-RC14

My Builder looks like the following:

final Downloader okHttpDownloader = new OkHttpDownloader(new OkHttpClient.Builder().followRedirects(true).followSslRedirects(true).build());
Fetch mainFetch = new Fetch.Builder(getApplicationContext(), "Main").setDownloadConcurrentLimit(999).setDownloader(okHttpDownloader).build();

I do use OkHttpDownloader
Usually even when I try with one single download it gives me extremely low speeds
And the downloads links are all like these: http://www.blogger.com/video-play.mp4?contentId=c3e4bdcad249489f PS: This file is very small so it instantly completes, but when we're talking about a 30+mb download it takes a while(as I mentioned, something close to 200~300kb/s)

I never tried using the sample app, will do and report back.

tonyofrancis commented 6 years ago

@RobbieArmz Your configuration looks fine so far. The only thing that could slow the network down is several downloads going on at once. Another issue that can cause low performance is if you are also querying Fetch in the FetchListener. Especially the onProgress method. If you are querying Fetch in the onProgress method of the listener, and several downloads is ongoing, the downloads are fighting for access to the database(store information) and the query is also fighting for access to the database(Get information). So each in turn have to wait their turn for database access.

Do you mind sharing an example of your Fetch Listener? Also have you tried the latest release of Fetch? RC-17. Waiting to hear back on the performance of the sample app.

RobbieArmz commented 6 years ago

Okay so I ran tests, the thing is this, when I use the direct link which is .mp4 I get maximum speeds, I peaked at 6mb/s, when I use the blogger link I get 260kb/s. I don't believe this is related to the blogger servers because when I use browser or external downloaders like ADM for example to download the video I get around 4mb/s speeds.

I'll put my entire activity here and the logs I get:

package com.example.main.myapplication;

import android.os.Environment;
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.util.Log;

import com.tonyodev.fetch2.Download;
import com.tonyodev.fetch2.Downloader;
import com.tonyodev.fetch2.Error;
import com.tonyodev.fetch2.Fetch;
import com.tonyodev.fetch2.FetchListener;
import com.tonyodev.fetch2.Func;
import com.tonyodev.fetch2.NetworkType;
import com.tonyodev.fetch2.Priority;
import com.tonyodev.fetch2.Request;
import com.tonyodev.fetch2downloaders.OkHttpDownloader;

import org.jetbrains.annotations.NotNull;

import okhttp3.OkHttpClient;

public class MainActivity extends AppCompatActivity {

    String bbbDirect = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4";
    String bbbBlogger = "http://www.blogger.com/video-play.mp4?contentId=e878bb42c4ba79ae";
    private Fetch mainFetch;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        String link = bbbBlogger;
        String file = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS).toString() + "/file.mp4";

        final Downloader okHttpDownloader = new OkHttpDownloader(new OkHttpClient.Builder().followRedirects(true).followSslRedirects(true).build());
        mainFetch = new Fetch.Builder(getApplicationContext(), "Main").setDownloadConcurrentLimit(999).setDownloader(okHttpDownloader).build();
        mainFetch.removeAll();
        mainFetch.deleteAll();

        mainFetch.addListener(new FetchListener() {
            @Override
            public void onQueued(@NotNull Download download) {
                Log.d("PROG-D", "Download enqueued!");
            }

            @Override
            public void onCompleted(@NotNull Download download) {
                Log.d("PROG-D", "Download completed!");
            }

            @Override
            public void onError(@NotNull Download download) {
                Log.d("PROG-D", "Error on downloading: " + download.getError());
            }

            @Override
            public void onProgress(@NotNull Download download, long etaInMilliSeconds, long downloadedBytesPerSecond) {
                final int progress = download.getProgress();

                Log.d("PROG-D", "Progress Completed :" + progress);
                Log.d("PROG-D", "Speed :" + downloadedBytesPerSecond);
            }

            @Override
            public void onPaused(@NotNull Download download) {

            }

            @Override
            public void onResumed(@NotNull Download download) {

            }

            @Override
            public void onCancelled(@NotNull Download download) {
                Log.d("PROG-D", "Download canceled!");
            }

            @Override
            public void onRemoved(@NotNull Download download) {

            }

            @Override
            public void onDeleted(@NotNull Download download) {
                Log.d("PROG-D", "Download deleted!");
            }
        });

        downloadFetch(link, file);
    }

    public void downloadFetch(String url, String file){
        final Request request = new Request(url, file);
        request.setPriority(Priority.HIGH);
        request.setNetworkType(NetworkType.ALL);

        mainFetch.enqueue(request, new Func<Download>() {
            @Override
            public void call(Download download) {
                Log.v("PROG-D", "Item queued");
            }
        }, new Func<Error>() {
            @Override
            public void call(Error error) {
                Log.v("PROG-D", "Error: " + error);
            }
        });
    }
}

Logs with the direct link:

04-27 15:55:59.142 28220-28220/com.example.main.myapplication V/PROG-D: Item queued
04-27 15:55:59.142 28220-28220/com.example.main.myapplication D/PROG-D: Download enqueued!
04-27 15:56:00.334 28220-28220/com.example.main.myapplication D/PROG-D: Progress Completed :0
04-27 15:56:00.334 28220-28220/com.example.main.myapplication D/PROG-D: Speed :0
04-27 15:56:02.335 28220-28220/com.example.main.myapplication D/PROG-D: Progress Completed :6
04-27 15:56:02.335 28220-28220/com.example.main.myapplication D/PROG-D: Speed :5155808
04-27 15:56:04.335 28220-28220/com.example.main.myapplication D/PROG-D: Progress Completed :14
04-27 15:56:04.335 28220-28220/com.example.main.myapplication D/PROG-D: Speed :5695120
04-27 15:56:06.336 28220-28220/com.example.main.myapplication D/PROG-D: Progress Completed :21
04-27 15:56:06.336 28220-28220/com.example.main.myapplication D/PROG-D: Speed :5745320
04-27 15:56:08.336 28220-28220/com.example.main.myapplication D/PROG-D: Progress Completed :28
04-27 15:56:08.336 28220-28220/com.example.main.myapplication D/PROG-D: Speed :5653780
04-27 15:56:10.336 28220-28220/com.example.main.myapplication D/PROG-D: Progress Completed :35
04-27 15:56:10.336 28220-28220/com.example.main.myapplication D/PROG-D: Speed :5648931
04-27 15:56:12.337 28220-28220/com.example.main.myapplication D/PROG-D: Progress Completed :41
04-27 15:56:12.337 28220-28220/com.example.main.myapplication D/PROG-D: Speed :5170402
04-27 15:56:14.337 28220-28220/com.example.main.myapplication D/PROG-D: Progress Completed :48
04-27 15:56:14.337 28220-28220/com.example.main.myapplication D/PROG-D: Speed :5357400

Logs using the blogger link:

04-27 16:00:14.007 28694-28694/com.example.main.myapplication V/PROG-D: Item queued
04-27 16:00:14.007 28694-28694/com.example.main.myapplication D/PROG-D: Download enqueued!
04-27 16:00:17.205 28694-28694/com.example.main.myapplication D/PROG-D: Progress Completed :0
04-27 16:00:17.205 28694-28694/com.example.main.myapplication D/PROG-D: Speed :0
04-27 16:00:19.862 28694-28694/com.example.main.myapplication D/PROG-D: Progress Completed :8
04-27 16:00:19.863 28694-28694/com.example.main.myapplication D/PROG-D: Speed :1145689
04-27 16:00:22.712 28694-28694/com.example.main.myapplication D/PROG-D: Progress Completed :9
04-27 16:00:22.713 28694-28694/com.example.main.myapplication D/PROG-D: Speed :703917
04-27 16:00:25.481 28694-28694/com.example.main.myapplication D/PROG-D: Progress Completed :10
04-27 16:00:25.481 28694-28694/com.example.main.myapplication D/PROG-D: Speed :402809
04-27 16:00:28.360 28694-28694/com.example.main.myapplication D/PROG-D: Progress Completed :12
04-27 16:00:28.360 28694-28694/com.example.main.myapplication D/PROG-D: Speed :243576
04-27 16:00:31.100 28694-28694/com.example.main.myapplication D/PROG-D: Progress Completed :13
04-27 16:00:31.101 28694-28694/com.example.main.myapplication D/PROG-D: Speed :252314
04-27 16:00:33.983 28694-28694/com.example.main.myapplication D/PROG-D: Progress Completed :14
04-27 16:00:33.983 28694-28694/com.example.main.myapplication D/PROG-D: Speed :262139

As you can tell I'm not doing any work out of the ordinary inside those listeners, still somehow the speeds are low like that.

tonyofrancis commented 6 years ago

@RobbieArmz Just finished testing. I add a network interceptor to the okHttpClient to see what the actual url the download is happening from after the redirects. The redirected url returned by Blogger is not the direct url. It is returning some other url that is slow both on the App and the different web browsers I tested on. My OkHttpClient:

   final OkHttpClient client = new OkHttpClient.Builder()
                .followRedirects(true)
                .followSslRedirects(true)
                .addNetworkInterceptor(new Interceptor() {
                    @Override
                    public Response intercept(Chain chain) throws IOException {
                        System.out.println("url: " + chain.request().url());
                        return chain.proceed(chain.request());
                    }
                })
                .build();

Logs:

04-28 08:58:48.491 7916-7948/com.tonyodev.speedapp I/System.out: url: http://www.blogger.com/video-play.mp4?contentId=e878bb42c4ba79ae
04-28 08:58:48.842 7916-7948/com.tonyodev.speedapp I/System.out: url: https://www.blogger.com/video-play.mp4?contentId=e878bb42c4ba79ae
04-28 08:58:49.140 7916-7948/com.tonyodev.speedapp I/System.out: url: https://redirector.googlevideo.com/videoplayback?requiressl=yes&id=e878bb42c4ba79ae&itag=18&source=blogger&app=blogger&cmo=secure_transport%3Dyes&cmo=sensitive_content%3Dyes&ip=0.0.0.0&ipbits=0&expire=1527067811&sparams=requiressl,id,itag,source,ip,ipbits,expire&signature=23C9E6A7C52CBB6AC10F395D0C2DE3DC796A9B3A.36F95B5886EB922739F7DF60D1A0C51B876E8537&key=ck2
04-28 08:58:49.595 7916-7948/com.tonyodev.speedapp I/System.out: url: https://r2---sn-ab5l6n67.googlevideo.com/videoplayback?requiressl=yes&id=e878bb42c4ba79ae&itag=18&source=blogger&app=blogger&ip=0.0.0.0&ipbits=0&expire=1527067811&sparams=expire,id,ip,ipbits,itag,mip,mm,mn,ms,mv,pl,requiressl,sc,source&signature=1898D347A9315B8EA9FAA6BC560A7140C01D5805.1A4CC11A2AE6CBB57FEF4C1F2AA785FDFBD491C1&key=cms1&cms_redirect=yes&mip=2604:2000:d04f:aa00:b15a:2cde:358d:aadc&mm=30&mn=sn-ab5l6n67&ms=nxu&mt=1524920240&mv=m&pl=32&sc=yes

The last logged url is where the download is happening from and it is 3 times slower than the direct url you shared. Hope this helps.

RobbieArmz commented 6 years ago

Is there any way to speed that up somehow, because I can download those at high speeds via browser but not through fetch x.x

EDIT--

I just tried via browser normally and the speed is low too... strange... I tried downloading directly through chrome to check the speed, and it was around 200kb/s, used the same link on a download manager(Free Download Manager) and it hit 6mb/s.

My question is, what these download managers do to reach those speeds? Not only on windows but when I use ADM on android it hit the maximum speed aswell.

EDIT-- I've been reading around and I think I figured it out why they're faster, some download managers split the download into chunks and download them all im parallel, I guess this is why they're faster. Is there any way I can do something like this on Fetch?

tonyofrancis commented 6 years ago

@RobbieArmz right now Fetch does not support chuck downloads. I'll will brainstorm this as a feature over the weekend and will get back to you.

RobbieArmz commented 6 years ago

Alright tonyo, thaks alot.

tonyofrancis commented 6 years ago

@RobbieArmz I started working on the chunk file downloader extension for Fetch. Got to download a completed file lol. This chunk file downloader still needs lots of polishing and proper integration with Fetch. Don't have a hard release date yet. Hoping in less than 2 weeks, but I will keep updating this post with updates. I will be adding chunk downloading as a feature once all the bugs are worked out. You can also follow up with me if you want to know where this stands. Thanks

RobbieArmz commented 6 years ago

No prob tonyo, I've heard that this old free project called "GigaGet" used the chunk download methodology, and could get good speeds but that project got discontinued and thats a big nono for me. They might even be on github somewhere if you wanna take a look for ideas.

tonyofrancis commented 6 years ago

@RobbieArmz I have made lots of progress with the chunk file downloader. Works really well. Just need to retrofit/ polish it up a bit and it should be ready to go in the next few days.

RobbieArmz commented 6 years ago

@tonyofrancis no problem take, take your time with it. I'm pretty sure is gonna be awesome.

tonyofrancis commented 6 years ago

@RobbieArmz Just pushed the new version of Fetch 2.0.0-RC20 with the new Parallel/Chunk Downloader. Parallel/Chunk downloading is not enabled by default. The examples below shows you how you can enable chunk downloading.

Example 1. Passing the new Downloader.FileDownloaderType.PARALLEL file downloader type in the OkHttpDownloader or HttpUrlConnectionDownloader constructor and adding the downloader on the Fetch Builder.

//Using OkHttpDownloader
 final Downloader okHttpDownloader = new OkHttpDownloader(client,
                Downloader.FileDownloaderType.PARALLEL);
or
//Using HttpUrlConnectionDownloader
final Downloader HttpUrlConnectionDownloader = new HttpUrlConnectionDownloader(httpUrlConnectionPrefs, Downloader.FileDownloaderType.PARALLEL);

new Fetch.Builder(this, namespace)
                .setLogger(new FetchTimberLogger())
                .setDownloader(okHttpDownloader) // set the downloader on Fetch
                .setDownloadConcurrentLimit(1)
                .enableLogging(true)
                .enableRetryOnNetworkGain(true)
                .build();

Example 2. If you are extending an existing downloader or implementing your own, you need to override the following methods in the custom downloader.

    public static class CustomDownloader extends OkHttpDownloader {

        //Override this method to return the FileDownloaderType
        @NotNull
        @Override
        public FileDownloaderType getFileDownloaderType(@NotNull Request request) {
            return FileDownloaderType.PARALLEL; //FileDownloaderType.SEQUENTIAL is the default.
        }

        /**
         * Override this method if you want to tell Fetch how many pieces the download
         * should be sliced into for parallel downloading. See the java docs for more information.
         * Fetch automatically slices the download based on the content length if null is returned.
         */
        @Nullable
        @Override
        public Integer getFileSlicingCount(@NotNull Request request, long contentLength) {
            return super.getFileSlicingCount(request, contentLength);
        }
    }

Please let me know if this works well for you. Would like your feedback.

RobbieArmz commented 6 years ago

Hi, I tested with and without the new feature, sure as hell the speed its much better. As a matterfact, it is 3x faster than usual, when I dont use it my speed is 150kb/s, with, it goes up to 600kb/s+

One question, I assume the chunks are split in 4 threads(hence the 4x faster speeds), is there any way to increase the amount of threads?

tonyofrancis commented 6 years ago

@RobbieArmz Answer in my previous response in last example. Override getFileSlicingCount method for the custom downloader. The value returned by this method is the number of threads that will be used for the specified request. Sorry it was not so clear.

RobbieArmz commented 6 years ago

So this is how I should go about if I want it sliced into 10 pieces? Or do I need to enter the size I want it sliced in?

@Override
    public Integer getFileSlicingCount(@NotNull Request request, long contentLength) {
        return super.getFileSlicingCount(request, 10);
    }

I'm still struggling to understand where to put what and how, could you provide me with an example? On how I'd use the custom downloader by using this: final Downloader okHttpDownloader = new OkHttpDownloader(new OkHttpClient.Builder().followRedirects(true).followSslRedirects(true).build(), Downloader.FileDownloaderType.PARALLEL);

tonyofrancis commented 6 years ago

@RobbieArmz Create a customer downloader class that extends the existing okHttpDownloader class

     public class CustomDownloader extends OkHttpDownloader {

        public CustomDownloader() {
            this(null);
        }

        public CustomDownloader(@Nullable OkHttpClient okHttpClient) {
            super(okHttpClient);
        }

        @NotNull
        @Override
        public FileDownloaderType getFileDownloaderType(@NotNull Request request) {
            return FileDownloaderType.PARALLEL; //For chunk downloading
        }

        @Nullable
        @Override
        public Integer getFileSlicingCount(@NotNull Request request, long contentLength) {
            return 10; //Return the number of threads/slices. No need to call super.
        }
    }

Create instance of Fetch that uses the custom downloader

   final OkHttpClient client = new OkHttpClient.Builder().build();
        final Downloader customDownloader = new CustomDownloader(client);
        Fetch fetch = new Fetch.Builder(this, namespace)
                .setLogger(new FetchTimberLogger())
                .setDownloader(customDownloader) // set custom downloader
                .setDownloadConcurrentLimit(4)
                .enableLogging(true)
                .enableRetryOnNetworkGain(true)
                .build();

RobbieArmz commented 6 years ago

Thanks Tonyo, now my app downloads at full speeds even tho the server limits it. One question, since chunk downloading splits the file in several threads, lets assume I split a file into 20 slices, how many should I set on concurrent limit so it doesnt impact on the device performance.

Cause 20 slices = 20 threads, times lets say... 5 files = 100 threads? Or am I wrong?

tonyofrancis commented 6 years ago

@RobbieArmz

Concurrent limit dictates how many requests that can be downloaded at a given time.

Here is Fetch’s thread count break down.

Fetch uses 1 thread to do all the main work. Such as scheduling, queuing requests etc.
Each request that is currently being downloaded spins up 1 initial thread. If Parallel downloading is enabled for the request, the total threads per downloading request is 1 + splits. Each split/chunk that needs to download is a thread.

Example, say you have 5 downloads currently running and each have parallel enabled, and you requested 10 splits per downloading request. Each downloading request will have 11 threads. Times that by 5 and that would be equal to 55 threads. If you also add the main thread Fetch uses to do the scheduling, that would be a total of 56.

Yeah that’s a lot of threads. In retrospect many of the threads will be short-lived. Many of them will only live a few seconds. The only time I foresee an issue is if you are downloading several gigabyte files at the same time with a high split count. Those could take a very long time to complete causing performance issues.

Measure your app and see what works best. Ask these question.

How many concurrent downloads should the user be allowed to download? More is not always best. Find a balance.
Is it critical that the user downloads this file/data instantly? I would consider critical map information, podcasts and on the go information.
What can you do to Prefetch data for the user over a period of time. You may want to look into downloading data when the phone is on the charger and has WiFi. Check out Job Scheduler for Android.

Hope this helps.

RobbieArmz commented 6 years ago

Alright Tonyo, I'll perform some testings, and see what configs goes best for me. Thanks once again.

tonyofrancis commented 6 years ago

@RobbieArmz Just pushed out version 2.0.0-RC21 with bug fixes for the parallel downloader.

tonyofrancis / Fetch

Is there any way to accelerate the downloads? #136