Important changes for Firefox 75

jingyu9575 commented 4 years ago

Please see the wiki page for details.

Old information

Firefox is planning to remove the storage functionality `IDBMutableFile` used by this extension. This will affect the extension severely. From version 3.0, a new storage method is implemented that only uses supported functions. The caveats are:

You will need to pause all tasks before exiting the browser. Otherwise, some progress will be lost depending on the extension parameters.
Existing downloads using the old method cannot be continued with the new method.
Performance drops have not been noticed during tests, but they may vary with different system conditions and parameters.

This new method is not enabled by default yet. The options of choosing the storage method and the parameters have been added, if you want to test it now. --- Right now it is unknown when the `IDBMutableFile` API will be removed. This issue will be left open to track the status.

Mike-Reddick commented 4 years ago

All right, sad change by mozilla, I think even if some day this method can be default, is good keep an setting to use old method in old firefox versions. Thanks for the informations

Mike-Reddick commented 4 years ago

@jingyu9575 you plan keep implementing new features to this extension, or after this mozilla change, just apply bugfixes?

Nokia808 commented 4 years ago

@jingyu9575 I test it as following: Advanced setting | Storage API | select "Segmentation" but leave other parameters on "Default" since I do not know about them. Is this enough or I have to change other things ?

Also, is this workaround need to additional copy & decrease performance by making storage slower? From your change log I conclude that there is no such decrease in performance, isn't it?

Also, what is the maximum file size that can be downloaded by this extension? 4 GB or more?

jingyu9575 commented 4 years ago

@Mike-Reddick Feature requests are still welcome. I'm not sure how many users will remain after Firefox's change, but I still use it myself and won't abandon it.

jingyu9575 commented 4 years ago

@Nokia808

Advanced setting | Storage API | select "Segmentation"

Yes it is enough. If this option is selected, the new method will be used for new download tasks. Existing tasks still use the old one (otherwise they cannot continue).

is this workaround need to additional copy & decrease performance

No. Detailed comparison:

Storage API	Performance
Old method, Firefox 73-	1 copy in temp storage
Old method with Firefox 74+ workaround	2 copies in temp storage
New method	1 copy in temp storage (segmented)

what is the maximum file size that can be downloaded by this extension? 4 GB or more?

The hard limit is 50% of free disk space, imposed by Firefox.

With the old method I've downloaded files around 4 GB successfully and I've heard about other people doing so. I have not heard about failures due to large size, unless they are reaching the hard limit.

With the new method there is another issue. The file is stored as many segments, and some people say Windows can be very slow if there are lots of files in a directory. Some say 1 million files are still acceptable while some say 10000 files are already too many.

To workaround it, the extension will only write the segments every n seconds, with n gradually increasing during the download. With the default parameters, n is 2 seconds at first and grows to 1 minutes gradually. It will take 20 hours of download to reach 10000 files, so the number of files can be kept small for most downloads. However if you have a very slow connection and attempt to download a large file for many days, the performance may drop.

Also this is why you will need to pause all tasks before exiting Firefox.

jingyu9575 commented 4 years ago

About the other advanced options:

Connection API: Streams is the default since v2. It has a little better performance than StreamFilter, which is the v1 method. I think both are stable and there is no reason to revert to StreamFilter.
Cache mode: By default the download is not cached, because I've seen some bugs with caches when developing v2. Many servers do not allow downloads to be cached anyway, and if you get some wrong content you won't want them to remain in cache. The meanings of the values are here.

Mike-Reddick commented 4 years ago

Thanks all explanations, I'm using your extension as my unique download manager and to be honest after this new mozilla change I can't make my Firefox update soon, to continue using the old storage method. But you said to user using new FF versions to pause to avoid progression loss, the FF API allow extensions ask something before FF close, like when you have many tabs open? So, maybe you can add a optional feature to ask to pause downloads when the user close FF? This can help the users with new FF versions, is just some suggestions.

jingyu9575 commented 4 years ago

@Mike-Reddick

add a optional feature to ask to pause downloads when the user close FF

I currently only know one way to ask things when Firefox is closed: create a hidden tab that shows the "your changes may not be saved" dialog. It can also be used to automatically save the progress on exit. However, this requires:

dom.require_user_interaction_for_beforeunload set to false in about:config
Permission to hide tabs, and the initial prompt when hiding the first tab
Creating a hidden tab in every window, forcing the ▼ icon to be shown
Dealing with the case that the user closes all other tabs, since Firefox will then show the hidden tab. It must close the window or create a new tab depending on user preference, which the extension cannot read and must be set separately.

I think it is just too invasive. Probably if there is another extension e.g. "Ask on Browser Exit" that the user can install separately and agree to all the invasive changes, my extension can make use of that.

Mike-Reddick commented 4 years ago

I think it is just too invasive

I think the same, is a lot of workarounds, maybe better don't implement it :)

Nokia808 commented 4 years ago

@jingyu9575 Hi dear. Regarding issue of loss of storage if exit FF before pause ongoing download tasks, I have 2 suggestions:

1) I suggest - if it is possible - to make your add-on automatically pause ongoing download tasks when user exit FF: I mean making your add-on automatically detect exit action & automatically pause ongoing download tasks before exit.

2) if above solution is not possible or highly invasive, then I think the best solution is to add "permanent (fixed)" warning statement to download tasks window of add-on, saying for example: "Warning! You should pause all ongoing download tasks before exit FF, otherwise you will loss download progression". Such warning fixed statement you can put it to be just above the icons of options at lower boarder of window.

jingyu9575 commented 4 years ago

@Nokia808 It is currently not possible to save large amount of data during browser exit.

It is still unclear how soon the API will be removed, and whether there will be new possibility to deal with this gracefully, so I am keeping things as-is now. We can notify the users when the API is about to be removed in Nightly.

Mike-Reddick commented 4 years ago

@jingyu9575 just a question the new method (segmented) can increase HD or SSD wear comparated for instance with the browser build-in download manager, in your opnion? What you think about?

jingyu9575 commented 4 years ago

@Mike-Reddick I don't think there is significant wear increase after switching to segmented. It is the same content but separated into multiple files.

jingyu9575 commented 4 years ago

The new storage API has been set to default in version 3.2. I'd close this issue now.

SeanFrohman commented 2 years ago

So, I am confused, were we able to get it working? API type change did nothing for me, but I am a programmer as well, let me know if I can help in any way.

jingyu9575 commented 2 years ago

@SeanFrohman If you come here from the error message at the bottom of the extension panel, that is an accidental display problem (#180) that does not affect the functions. Unfortunately there is also a bug #179 that corrupts the files. Firefox 105 will fix it. I plan to remove the error message when Firefox 105 releases.

SeanFrohman commented 2 years ago

Right now when I try to download something it just says download failed permission storage something.

So I'm unable to use it. That's why I clicked it and now I'm trying to figure out if I did something wrong.

Tried reinstalling I tried adjusting settings, still nothing.

On Sun, Aug 21, 2022, 10:07 AM jingyu9575 @.***> wrote:

@SeanFrohman https://github.com/SeanFrohman If you come here from the error message at the bottom of the extension panel, that is an accidental display problem (#180 https://github.com/jingyu9575/multithreaded-download-manager/issues/180) that does not affect the functions. Unfortunately there is also a bug #179 https://github.com/jingyu9575/multithreaded-download-manager/issues/179 that corrupts the files. Firefox 105 will fix it. I plan to remove the error message when Firefox 105 releases.

— Reply to this email directly, view it on GitHub https://github.com/jingyu9575/multithreaded-download-manager/issues/128#issuecomment-1221553130, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKWUBU5XD5YI2KCIJBP375DV2IZY7ANCNFSM4KMXJBHQ . You are receiving this because you were mentioned.Message ID: @.*** com>

jingyu9575 / multithreaded-download-manager

Important changes for Firefox 75 #128