search

danielvdende / jqmTestSuite

Public repository for jqm team for DelftSWA course
0 stars 1 forks source link

Further work on test suite? #1

Open danielvdende opened 11 years ago

danielvdende commented 11 years ago

Hi guys,

It's been a while since we last did any work on this project, maybe we can continue our work? I'm up for doing some more work on it anyway. I think it would be great experience and I think it's great to stay in touch with people like Jasper. I drew up an initial outline of what I think still needs to be done:

code

work that still needs to be done:

blog post:

vhogendoorn commented 11 years ago

@danielvdende, I still want to work on this, have you had any contact with Jasper? Is he still interested? Because it's now three months later and maybe they're not interested anymore.

A month ago they announced the 1.4 alpha version, the one (I think) we already had built for ourselves. http://jquerymobile.com/blog/2013/07/25/announcing-jquery-mobile-1-4-0-alpha/

maybe if we're going to refactor everything, we might directly take the published alpha version to do all of our tests.

I think we also need a document-file in the parentfolder, in which we describe what we're testing, which analysis we do and the statistics.

Also in the blog post, presenting results with some charts.

vhogendoorn commented 11 years ago

So I think we can still use the pages we created, but start from scratch with the tests. Because we now haven't any time pressure we can first think about a good structure of testing.

lycheung commented 11 years ago

I've a busy schedule but I would like to work on this with you guys. Contacting Jasper seems to be a first thing that should be done, in order to see if he is still interested in our work.

danielvdende commented 11 years ago

I sent Jasper an e-mail ;). Now we can only wait for his response :)

vhogendoorn commented 11 years ago

Let's hope he's positive about it. After that we can start making a plan about the way of working and start working on it again. About the way of automatically save it into a file, I prefer to CSV, because it's easy to use in javascript again to use an external library (like google charts) to create fancy shit haha

danielvdende commented 11 years ago

Yeah definitely. I guess your experience with Google Charts will come in handy :). CSV is fine by me but I know Google Charts works well with JSON too. Doesn't really matter that much. It's a minor detail.

vhogendoorn commented 11 years ago

You don't need to have any experience for Google Charts haha. Json is also useful indeed. Both filetypes, csv and json, are easy to read with javascript to obtain the data. And google charts will do the rest. I'm just a lazy bastard haha.

danielvdende commented 11 years ago

Jasper responded!

Hi all,

Thanks for your email. I think it’s best if we, the jQuery Mobile team, first discuss how we want to continuously test performance and then talk with you guys to see if/how you can help with that. Currently we are very busy with the upcoming 1.4 release, so I think we don’t have time to look into this before the end of September.

For now it would be great if we can use your test suite for a 1.4 versus 1.3 performance test report. I would like to include the results in my 1.4 final release blog post. If I am not mistaken you guys only tested each widget on its own. Would it be possible to run the same tests on a page where we combine several widgets? What I have in mind is creating two or three test pages that give a good picture of the overall performance improvements and show those results. I can deliver the markup for that.

Thanks again for all your hard work on this!

Jasper

PS. Did you publish something on the web about your project?

vhogendoorn commented 11 years ago

Nice reaction.

They're Busy with the 1.4 version, but they have already a version of 1.4 online. So I think we should test everything with that online version.

combining widgets on a single page is also a good idea, gives a better overview of the overall performance.

I really look forward to work on this project again!

danielvdende commented 11 years ago

Yeah, agreed. I think we basically have 4 main tasks (correct me if I'm wrong, or if I've missed anything!)

danielvdende commented 11 years ago

@NathanMol Are you still interested in continuing our work on this project, Nathan?

NathanMol commented 11 years ago

Hi guys, im very busy currently at the moment, but im interested to continue with this next week!

Verzonden vanaf Samsung Mobile

-------- Oorspronkelijk bericht -------- Van: danielvdende notifications@github.com Datum: 03-09-2013 13:07 (GMT+01:00) Aan: danielvdende/jqmArchitecture jqmArchitecture@noreply.github.com Cc: NathanMol nathanmol@talpaedesign.nl Onderwerp: Re: [jqmArchitecture] Further work on test suite? (#1)

@NathanMol Are you still interested in continuing our work on this project, Nathan?

— Reply to this email directly or view it on GitHub.

vhogendoorn commented 11 years ago

@NathanMol Can you make it next friday (13-09), on 10.45 (after Random Signal Processing)? If you @NathanMol and @danielvdende can also bring your laptop, we have three 'fast' laptop so we can all test different parts of our test suite.

@danielvdende is going to contact Jasper again about the HTML-page. Perhaps they could deliver us the markup before next friday. He's also going to make a beginning on the blog post.

danielvdende commented 11 years ago

Ok, we divided the work as follows.

Please all, DON'T FORGET TO CLEAR YOU CACHE! We need to get data without any cache interference, as this may contaminate our results. Per browser you can clear you cache, but you also need to ensure that the cache is disabled between the test runs.

danielvdende commented 11 years ago

Hi guys,

Some bad news regarding our test runs. I'm afraid we'll have to redo them :(. Jasper mentioned in the e-mail that jqm 1.3.2 does not officially support jquery 2.0. So that's a first issue. Moreover, we have been testing with jqm 1.3.1 vs jqm git build. Not only is 1.3.1 not the most recent version of jqm, our git build is also a bit old. I ran some tests on Jasper's pages and it does seem to make a difference (not a huge difference, but noticeable). I'm going to reconfigure the test suite and will let you know when you can start testing again. I can run the Chrome Mobile tests again if necessary, I can't do the Android Browser or Safari, could you guys do those once I have changed the test suite set up?

danielvdende commented 11 years ago

Ok, here is my suggestion for how we should proceed. I have created a folder secondDump in our dropbox folder, please dump your results for your browser in there.

For now, I will simply e-mail you the new suite set up (not great version control, I know! :P, but this is the easiest way to ensure everyone is running the correct code. I have upped the load time somewhat, as it appeared that the load times for Jasper's pages was significantly higher than for our simple examples (which makes sense). If we can get the test runs and analysis done soon, we should be able to complete the blogpost by next weekend. I will try to contact Arie van Deursen as well, perhaps he can post the blogpost on the TU Delft domain as well. Furthermore, the company I work for has shown great interest, and they probably will want to post our results as well. So we'll probably be linked to from 3 places (JQM blog, TU Delft, Mobile Professionals) :). I think that would be a great result for this project :)

lycheung commented 11 years ago

I can rerun the tests now, and put the result in the second dump folder :)

danielvdende commented 11 years ago

That would be great :). I'm currently running the chrome tests :) On 15 Sep 2013 14:59, "lycheung" notifications@github.com wrote:

I can rerun the tests now, and put the result in the second dump folder :)

— Reply to this email directly or view it on GitHubhttps://github.com/danielvdende/jqmArchitecture/issues/1#issuecomment-24470994 .

lycheung commented 11 years ago

Just do the same as last time right?

vhogendoorn commented 11 years ago

I'll run the tests for the android browser tonight =)

danielvdende commented 11 years ago

Yeah, there are three more to run, the pages supplied by Jasper. You'll see them in the file where you change the name

lycheung commented 11 years ago

ok

danielvdende commented 11 years ago

The Chrome test runs are complete. The results are in the secondDump folder in our dropbox folder :)

lycheung commented 11 years ago

The safari IOS test runs are also completed and can be found in the secondDump folder :)

lycheung commented 11 years ago

oops sorry clicked in wrong button XD

vhogendoorn commented 11 years ago

I'm running the last test now... For some reason I've the same problem as @NathanMol with navBar... this one doesn't give results... miscPage and flipswitch didn't gave results first either, for this one I changed the times: Loadtime: 1800, nexttime: 2500. flipswitch I'm running right now, hopefully it will give results this time.

danielvdende commented 11 years ago

Basically android browser sucks :P

NathanMol commented 11 years ago

Some results are still blank, i guess some problems occur during testing...?

Browser: Chrome Android (@danielvdende) Tests: test_listPage_chrome_android, test_listview_chrome_android, navbar (but i guess that this test is no longer required), test_select_chrome_android, test_slider_chrome_android, test_textinput_chrome_android

Browser: Chrome Android (@lycheung) Tests: test_slider_chrome_android

danielvdende commented 11 years ago

Yeah I didn't check my results, I did have some issues... Sometimes when testing one or more iframes would just not load. The load times would end up being over 10 seconds after which the iframes were no longer removed. I wasn't sure whether this behaviour would affect the results, but apparently it does. Is it a huge problem?

NathanMol commented 11 years ago

@lycheung I guess you accidentally copied some results to other tests or something like that..., check your results:

listview 348,10 303,06 45,04 miscPage 348,10 303,06 45,04 radiobuttons 348,10 303,06 45,04 slider 348,10 303,06 45,04 textinput 348,10 303,06 45,04

The last 5 results are all the same...

NathanMol commented 11 years ago

@danielvdende yeah, actually there are no results at all, its just blank...

vhogendoorn commented 11 years ago

If during a test a second or third Iframe loads, it sometimes happen it doesn't save the results (so, a blank result list). I had the same problems, only with miscpage and flipswitch. for those I raised the timers.. and then I got results.. (only not with navbar, I don't know what's wrong with navbar xD )

NathanMol commented 11 years ago

I filtered the outliers as far as possible for all (3) tests, except those tests delivering blank results. The results can be found in the overview_{browser}.xlsx files. The above mentioned tests have to be executed again. After that the t-test procedural needs to be performed...

lycheung commented 11 years ago

@NathanMol huh thats weird.. I did all the test and copied it on the right way..

danielvdende commented 11 years ago

I'll rerun the missing Chrome tests tomorrow guys, I have done quite a lot of work on the blog :). Not too much left :)

vhogendoorn commented 11 years ago

For information: Tested on:

HTC One X, Android version 4.2.2 HTC Sense version 5.0 Browser Version: WebKit/534.30

NathanMol commented 11 years ago

Can i execute the t-test for every browser?

NathanMol commented 11 years ago

Safari iOS is still missing: test_select_safari_ios

NathanMol commented 11 years ago

The results (except Safari iOS test_select_safari_ios:

results_overview

The android browser has 7 tests performing significant faster using the new version. The one's that are slower, aren't significant slower...

With respect to the Chrome Android browser only one tests scores significantly faster; listPage.

Safari iOS has two tests performing significantly faster, but also two significantly slower...

I don't know what we can conclude with these results.... It mainly doesn't perform much better

danielvdende commented 11 years ago

Hmm those results aren't very good. The only thing we can investigate is whether jQuery 2.0 makes any difference and perhaps run the 1.9 tests on a desktop too. I'll run the tests on Chrome at some point tomorrow. Perhaps we can then see whether the improvements are mainly caused by jQuery 2.0. I do notice that in most cases (apart from Safari) the performance has improved, though not significantly. Perhaps this could be our conclusion?

NathanMol commented 11 years ago

Statistically you cant say much using the results... Maybe the usage of the mobile browsers/jQuery versions makes a difference related to the outcome of our first tests. If we decide to run another set of tests, i think we should go for the jQuery versions, since we also used that version for the first tests ?

danielvdende commented 11 years ago

Yeah, but if the results of those runs then turn out to be positive, basically what we're saying is that jqm 1.4 doesn't do much to improve performance, but jquery 2.0 does. Maybe we should try some independent tests. Perhaps run a test on desktop chrome with jquery 1.9 and a run on mobile chrome with jquery 2.0. Then we have some values that we can compare

vhogendoorn commented 11 years ago

why desktop chrome jquery 1.9, and mobile chrome 2.0? Maybe I'm missing something? Because what kind of values you can compare then? This will also have a conclusion (if positive) jquery 2.0 improves performance instead of jquery 1.9?

danielvdende commented 11 years ago

Because our first set of results seemed to indicate a huge performance increase, but those tests were run on a desktop. By doing those runs I suggested you can find out what might have caused this decrease in performance. It might be caused by jqm itself, but I think it would be good to validate the desktop/jquery 2.0 parameter

lycheung commented 11 years ago

I rerunt the test: test_select_safari_ios several times yesterday and today, but somehow it doesn't write the results in the excel file, while it is showing the frames with value in safari? XD

vhogendoorn commented 11 years ago

does it opens a second or third frame? because if that happens you may have to change the iframe load time en next frame timesettings..

and @danielvdende, now I understand what you mean. but on desktop we didn't run all the tests for as far as I can remember. So maybe we have to do those test also again (or at least those we didn't have)

NathanMol commented 11 years ago

first we need to check whether the performance gain is caused by jquery 2.0 instead of 1.9. after that we could check the desktop vs mobile test. but that seems to be rather unnecessary, because jquery mobile is mainly used WITH a mobile phone using a mobile browser instead of a desktop browser...

vhogendoorn commented 11 years ago

Yes, but I still think it will be difficult to conclude something... Because if JQM 1.4 perfoms better with jquery 2.0 instead of jquery 1.9, you can say it´s because Jquery 2.0, but it´s also possible it performs better with jquery 2.0 because JQM 1.4 is optimized to work well with jquery 2.0, and give less performance when using an older version of Jquery..

danielvdende commented 11 years ago

I suggest the following. I will re-run the tests on Chrome for desktop (with jQuery 1.9). @vhogendoorn or @NathanMol could you re-run the tests on Crome for mobile (with jQuery 2.0) ?

vhogendoorn commented 11 years ago

I'll re-run the test for Chrome mobile, but I can't start today (yes, maybe 1 or 2 pages) because I'm not home before 11.. Tomorrow I can do the rest.