Vault: need preloading animation when opening vault with many ads

[dhowe]

Probably some kind of progress meter, something like: http://creative-punch.net/2014/03/make-animated-image-gallery-percentage-preloader-css-jquery/

[mushon]

how bout this one (pushed to img):

[mushon]

corrected frames (pushed)

[dhowe]

how do I see this? or are you just showing the image?

[mushon]

huh?

[dhowe]

did you commit the working feature, or just the image?

[mushon]

Just the image, I think it needs some JS to take the image (or background image) disappear when the Packery composition loads

[dhowe]

So, no progress bar or percentage?

[mushon]

No, the percentage based progress bar died a long time ago with Shockwave Flash ;)

[dhowe]

If the user has to wait several seconds for a large set of ads to load, a repeating gif animation doesn't tell them anything -- the page will still appear to be hung

[mushon]

The role of a loader animation is to say: Yes, we know you’re not seeing what you’re expecting to see just yet, but it would load shortly

Loading bars are frustrating as they build expectations (for loading speed) they never meet. That’s why you don’t see them anymore.

[dhowe]

added some basic js for the preloader... please adjust css for the size/location you want. selector is: $('#loading-img')

[mushon]

done ada10bedef04e3e5e63bffe262b9ac42340ce345

[dhowe]

does this work for you? in my view it seems to show only one frame of the gif, then waits for all the other images to load (no animation)...

(note that I removed the redundant inline styles for the div)

[mushon]

Maybe add a slight delay for the loading so it doesn’t appear with every repack. I think a 2000ms delay would do the trick 

[CyrusSUEN]

Even if I comment out

setTimeout(function() { if (!done) $('#loading-img').show(); }, 2000); 

$('#loading-img').hide();

from repack() in vault.js, the first iteration of the animation gif is still laggy when I open the vault page.

Not until I commented out

createSlider(true);

from line 40 in vault.js will the animation be smooth again.

Of course this will render the vault page useless but the animation gif is playing smoothly in this situation.

Do you think we should solve this performance issue before working on this? Or is there any workaround for this? Defer the execution of createSlider(true); until the gif has been played at least once? But I'm afraid during that execution time the animation will again be laggy.

[dhowe]

so the order we want is: 1) start gif animation 2) create-slider 3) show images

is this correct?

[CyrusSUEN]

On a second thought I don't think deferring createSlider() can solve the problem. It will be just like above situation commenting out the 2 lines from repack().

Preferably there is a worker or a thread to avoid overloading just one CPU core. https://blog.mozilla.org/addons/2011/01/18/threads-n-workers-in-firefox-4/ http://stackoverflow.com/questions/9502281/concurrency-and-multi-threading-in-firefox-extensions

[mushon]

If nothing works, try keeping the gif img element always there and hiding it by giving it a alpha: 0 (not sure it would help, but it's worth trying)

[dhowe]

thats how it works now

[CyrusSUEN]

I think the current implementation of the animation is correct and the animation should have been playing and looping.

But the problem is when we open the vault page some functions such as createSlider() use too much CPU resource that other elements and the page are not responsive. Right after the createSlider() finishes and the CPU become free, the gif animation has been called to go away.

I think it's the lagginess that 'takes away' the animation.

The flow suggested by Daniel can ensure the animation will be played smoothly at least once (or some given time) but the animation will become laggy once reach Step 2.

so the order we want is: 1) start gif animation 2) create-slider 3) show images

[mushon]

4) alpha:0 gif animation

And on repack:

1) alpha:1 gif animation 2) repack

Mushon Zer-Aviv Mushon.com | Shual.com | @mushon

On May 26, 2015, at 22:27, dhowe notifications@github.com wrote:

so the order we want is: 1) start gif animation 2) create-slider 3) show images is this correct?

— Reply to this email directly or view it on GitHub.

[CyrusSUEN]

In order not to freeze UI while running the time consuming function createSlider() in vault.js I tried to make something like;; var worker = new Worker("chrome://AdNauseam/data/vault-slider.js"); and replace all createSlider(true); with worker.createSlider(true); in vault.js.

But I have difficulty implementing the worker in the addon. Here is the reference I studied: http://stackoverflow.com/questions/9216984/concurrency-with-firefox-add-on-script-and-content-script?rq=1 https://developer.mozilla.org/en-US/docs/Using_workers_in_extensions

Can you guys have a look at this?

[dhowe]

except for the occasional update messages from the addon, this page is just a web page...

so before we look for addon specific solutions, we should make sure that the problem doesn't also happen outside of the addon (test this by setting TEST_MODE = 1 in lib/config.js and trying various json files via the TEST_ADS property)

[CyrusSUEN]

Cannot run that in test mode. I guess that is because of line 24 of vault.js var locale = self.options && self.options.locale; // for localization and the relevant calls to properties of locale. But this is another issue.

[dhowe]

Does the problem exist when running in test-mode?? If so, then it has nothing to do with addon workers...

[CyrusSUEN]

Does the problem exist when running in test-mode??

When running in test mode, the vault page finishes so quickly that the gif animation doesn't have time to finish.

BTW when not running in test mode and the number of ads is low, I can see the animation works correctly. Test it with: Clear Ads > Import Ads > lib/test/test-ad-data.json

But when the number of ads is high, the animation freezes and the problem https://github.com/dhowe/AdNauseam/issues/327 appears:

[dhowe]

do you mean that the images load so quickly...? @CyrusSUEN -- have you tried with a huge json file?

[dhowe]

@CyrusSUEN please respond to my question above

Then please try the last solution on this page:

I found a hack to get WebWorkers in the extension's background page:

if(typeof(Worker) == 'undefined')
{
    var chromewin   =   win_util.getMostRecentBrowserWindow();
    var Worker      =   chromewin.Worker;
}
var worker      =   new Worker(data.url('path/to/script.js'));

By accessing the main window's window object, you can pull the Worker class into the current scope. This gets around all the obnoxious Page.Worker workaround junk and seems to work fairly well.

[CyrusSUEN]

have you tried with a huge json file?

Tried in test mode, same laggy animation.

try the last solution on this page:

The code can't be ran from a content script due to the following:

const Data = require("sdk/self").data;
const utils = require('sdk/window/utils');

[dhowe]

I don't think they are talking about a content-script...

But, as I say above, if the problem exists in TEST-mode, then it has nothing to do with browser threading, right?

Please reply with your (detailed) plan for addressing this issue (and don't waste your time on multi-process FF, which may never happen).

[CyrusSUEN]

if the problem exists in TEST-mode, then it has nothing to do with browser threading, right?

I've used a physics library box2djs on a HTML school project by following a tutorial Box2D and Web workers for JavaScript developers.

The article states that Web worker can separate UI thread and physics update loop thread. Back then I didn't know what was the purpose of this but now I realize that is because the Javascript-related thread eats up too much CPU resources within the same thread as the UI thread affecting the smoothness of the HTML page.

reply with your (detailed) plan for addressing this issue

My plan would be creating a web Worker object and then setting up all the communications interface.

don't waste your time on multi-process FF

Another approach I think will be solving https://github.com/dhowe/AdNauseam/issues/327. If we can lower the computation time (of vault-slider.js and related code) by a order of magnitude. We may not need to rely on any Worker method.

These two are the only plans I can think of.

[dhowe]

First, please try the same TEST_MODE tests with this branch, to see if there is any improvement: https://github.com/dhowe/AdNauseam/tree/crossfilter-sepans

[CyrusSUEN]

try the same TEST_MODE tests with this branch

Same problem. The loading animation is frozen or the page only shows a loading mouse cursor until all the content has been loaded.

[dhowe]

OK, so start with a super simple Web page that uses Web workers, then we will slowly add our features, one by one

On June 26, 2015 5:33:19 PM GMT+08:00, Cyrus S notifications@github.com wrote:

try the same TEST_MODE tests with this branch

Same problem. The loading animation is freeze until all the content has been loaded.

---

Reply to this email directly or view it on GitHub: https://github.com/dhowe/AdNauseam/issues/335#issuecomment-115610320

[mobile]

[CyrusSUEN]

so start with a super simple Web page that uses Web workers, then we will slowly add our features, one by one

OK. Made a simple demo on web worker but I just realized that it cannot access DOM.

So I was digging around to see which function used the most computation time and make a worker object on that function alone. Starting with vault-slider.js with lib/test/lots-of-ads.json:

console.log: adnauseam: computeMinDateFor took:  0.7858959003131076  millisecond
s.
console.log: adnauseam: runFilter took:  0.1578996346694339  milliseconds.
console.log: adnauseam: dateFilter took:  0.3635293870431724  milliseconds.
console.log: adnauseam: doLayout took:  17061.880446561783  milliseconds.

then inspected doLayout() from vault.js on a second run:

console.log: adnauseam: createDivs took:  21600.039625003377  milliseconds.
console.log: adnauseam: computeStats took:  5.981874562847224  milliseconds.
console.log: adnauseam: enableLightbox took:  193.67620968831397  milliseconds.
console.log: adnauseam: repack took:  3.322496495286032  milliseconds.

createDivs() needs to be run in a separate worker (thread). But since it involves lots of DOM operations, can you suggest a way so that we can separate the computation part from the DOM part?

[dhowe]

Almost 22 seconds for createDivs()!? That is insane. How many divs are being created? Can you profile the various parts of that function and see where the time is being spent? (then also try the same tests with @sepans branch, called 'crossfilter-sepans'

[CyrusSUEN]

createDivs running time 17904.148317588395 milliseconds / 93 adsets.length = 192 ms Each iteration of the loop takes around 0.2 second on average. But adding up the number tens of times it is huge.

The time spent on layoutAd() (in each iteration) varies from 50ms to 1250ms. Sampled from the latest entries:

console.log: adnauseam: layoutAd took:  1260.4395373480666  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.6528429438976673  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  569.9228213002416  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.905002056297235  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  575.4370007474718  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.6513419967996015  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  886.6148539728565  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.6987719250846567  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  874.049225061015  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.6636497630006488  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  258.6447048087339  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.7311923823945108  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  156.3710701452619  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.663349573582309  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  156.44491674246456  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.6813609387518227  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  327.4595269615129  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.6057132050318614  milliseconds.
console.log: adnauseam: Currently running:  layoutAd
console.log: adnauseam: layoutAd took:  325.14356558988584  milliseconds.
console.log: adnauseam: Currently running:  hoverOnDiv
console.log: adnauseam: hoverOnDiv took:  1.6672520360352792  milliseconds.

From within layoutAd(), sampled from the latest entries:

console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  12.88713178015496  millisecon
ds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  16.04542466296334  milliseconds.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  94.69625333585464  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.2980880935865571  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.674966904118264  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.1624024759603344  milliseconds.

console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  34.54429745169182  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.2977879041663982  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.149935609371823  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.16420361247946857  milliseconds
.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  34.690189509581614  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.2731723717661225  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.473539803617314  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.16390342305930972  milliseconds
.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  34.899421534990324  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.29658714648940077  milliseconds.

Looks like appendMetaTo() and appendTextDisplayTo() take the large proportion of the computation time.

[dhowe]

and in sepans branch?

[CyrusSUEN]

Using the same set of test data lib/test/lots-of-ads.json with the methods from https://github.com/CyrusSUEN/AdNauseam/commit/1db90f3fca9b4fe9355118034a74a80e620bdd8d

createDivs took: 15610.485616423965 milliseconds

From within layoutAd(), sampled from the latest entries:

console.log: adnauseam: appendMetaTo took:  34.60373495675594  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.29418563113358687  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.597818223299328  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.16900664319291536  milliseconds
.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  34.57671790900167  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.2803769178353832  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.453126923090167  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.16840626435077866  milliseconds
.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  34.20208151343468  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.28818184274132363  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.167946974541337  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.16450380189780844  milliseconds
.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  33.90069133623183  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.2875814639046439  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.220780312378338  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.17050759029007168  milliseconds
.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  35.87293582250277  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.28397919087183254  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.145732957493237  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.16060133944483823  milliseconds
.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  34.29183814987118  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.28007672841522435  milliseconds.
console.log: adnauseam: Currently running:  appendTextDisplayTo
console.log: adnauseam: appendTextDisplayTo took:  11.213575766309077  milliseco
nds.
console.log: adnauseam: Currently running:  appendBulletsTo
console.log: adnauseam: appendBulletsTo took:  0.16840626435077866  milliseconds
.
console.log: adnauseam: Currently running:  appendMetaTo
console.log: adnauseam: appendMetaTo took:  34.16305688889588  milliseconds.
console.log: adnauseam: Currently running:  setItemClass
console.log: adnauseam: setItemClass took:  0.2845795697085123  milliseconds.

[CyrusSUEN]

Don't think we can use Worker for doing this http://stackoverflow.com/questions/18056922/is-there-a-way-to-create-out-of-dom-elements-in-web-worker

Found another workaround: looks like we can use setTimeout to make little pause in each iteration to prevent the main UI thread from freezing http://stackoverflow.com/questions/714942/how-to-stop-intense-javascript-loop-from-freezing-the-browser

[CyrusSUEN]

Trying the workaround: https://github.com/CyrusSUEN/AdNauseam/commit/592bbeb1142b8b6d994283fb0df23aa832512297

The loading animation is a bit better but not smooth enough. (and doesn't display ads after loading)

Another approach I can think of is to create and store a HTML vault page inside memory or hard disk instead of generating the page right after clicking 'view ad vault'. That vault page should be updated every time when detecting new ads.

[dhowe]

How are you doing the performance tests above?

[CyrusSUEN]

How are you doing the performance tests above?

The method I'm using https://github.com/CyrusSUEN/AdNauseam/commit/1db90f3fca9b4fe9355118034a74a80e620bdd8d

var t0 = performance.now();
var t1 = performance.now();
console.log("createDivs", "took: ",(t1 - t0), " milliseconds.");

[dhowe]

you need to run the functions many times, and take an average (and only print results at the end) -- just the printing alone will make these tests very inaccurate -- see https://developer.mozilla.org/en-US/docs/Tools/Performance/Scenarios/Intensive_JavaScript

[CyrusSUEN]

see https://developer.mozilla.org/en-US/docs/Tools/Performance/Scenarios/Intensive_JavaScript

Tried using Firefox Developer Version to run with XPI file and using data lib/test/lots-of-ads.json because the tools seem to be only available in the Firefox Developer Version.

Uploaded the saved performance profiling data with instuctions: https://github.com/CyrusSUEN/AdNauseam/commit/91d3c26f1555471fa61172b927e6a4cc5b8fcdcd

---

run the functions many times, and take an average (and only print results at the end)

Using the performance.now() method for profiling: https://github.com/CyrusSUEN/AdNauseam/commit/145670a7e9dab1ded3d9a63f5def4d4e14a7b0d1 inside layoutAd():

append*To:
console.log: adnauseam: Total time:  1090.2339376146297
console.log: adnauseam: Avg time:  11.722945565748706

appendBulletsTo:
console.log: adnauseam: Total time:  1404.3347352179217
console.log: adnauseam: Avg time:  15.100373496966899

appendMetaTo:
console.log: adnauseam: Total time:  12262.118796960878
console.log: adnauseam: Avg time:  131.85073975226751

setItemClass:
console.log: adnauseam: Total time:  25.820792920322106
console.log: adnauseam: Avg time:  0.2776429346271194

appendMetaTo() uses the most CPU resources.

[dhowe]

this commit ^ adds lazy-creation of inspector meta-data (only run when user enables spotlight for an ad)

[CyrusSUEN]

this commit ^ adds lazy-creation of inspector meta-data (only run when user enables spotlight for an ad)

With this commit, createDivs took: 2,880.5141043998765 milliseconds down from 15,610.485616423965 ms using the data lib/test/lots-of-ads.json.

I will see if there are other areas that can be optimized.

[dhowe]

What is the status of the setTimeout calls? What is the delay time? Are they in a PR?