search

jonsamwell / flutter_gherkin

A Gherkin parsers and runner for Dart and Flutter which is very similar to cucumber
MIT License
205 stars 109 forks source link

Flutter App performance with help of BDD #73

Open GarimaVats opened 4 years ago

GarimaVats commented 4 years ago

I am trying to test performance of my app. Went through with Performance profiling page https://flutter.dev/docs/cookbook/testing/integration/profiling in flutter driver. Is any way to test performance within flutter_gherkin framework.

GarimaVats commented 4 years ago

@jonsamwell Please suggest, if you have any opinion.

jonsamwell commented 4 years ago

@GarimaVats Sorry for the late reply. I haven't really done much performance stuff (I know I should get to grips with it).

So the basic scenario here is you want to perform a step within a driver.traceAction block but not necessary wrap the action in that function yourself? Then at the end save the results?

GarimaVats commented 4 years ago

Yes you are right.

jonsamwell commented 4 years ago

OK I wonder how best to approach this as Gherkin doesn't support tags on the step level so 

@profile
Given I tap the "save" button

would be going against the spec.

Maybe a comment?

## profile
Given I tap the "save" button

Any ideas on how you would like this to be expressed?

GarimaVats commented 4 years ago

I think ## profile would be a better approach as it will show altogehter a separate approach and avoid confusion.

jonsamwell commented 4 years ago

Yeah I think that is probably best. Would we also have a corresponding ## end profile comment as well? Perhaps it would profile the step it is above if a closing block is not found.

We would also need a profile reporter as well which would save the json files of the traced timelines.

Could you give me a real world scenario that you want to achieve so I can be sure I get everything in we need. It seems there are a couple of tracing methods on the driver and they have parameters so we would probably need to support those too somehow.

GarimaVats commented 4 years ago

Closing with ## end profile is a good idea. I am using your framework in a big application. It has almost all feature, but I would suggest login page, Content screen loading, Content screen scrolling, list scrolling, tapping, redirecting to a website in a in-app browser, navigation from one page to another , user registration etc.

GarimaVats commented 4 years ago

Hello @jonsamwell , Did you get chance to looking this.

lsuhov commented 4 years ago

Hello @jonsamwell,

  1. If you are thinking on supporting performance testing, please also consider profiling entire scenarios. It's not clear to me how would the above proposal be working. Is it something like this?:
Scenario: my scenario
    Given I have "Sign in" button in "Login Page"
## profile
    When I tap on "Sign in" button in "Login Page"
    Then I have "dummy" button in "Main Page"
## end profile 
    And I  have "Back" button in "Main Page"

  1. We are exploring https://pub.dev/packages/ozzie as the candidate for doing the performance testing. Maybe you can take some hints from there

  2. In case you don't have the bandwidth for doing this, maybe we can help you by raising a Pull request. The problem is on first look it's not trivial to find the best place in the library code for doing the change. Can you give a suggestion?

jonsamwell commented 4 years ago

@GarimaVats I have given it some thought and looked initially at what would need to be done but not had sufficient time yet to actually do it. I think I can have it done towards the mid to the end of August.

@Isuhov Thank you for you points, yes, I agree when implementing performance I would offer a complete solution i.e. profiling a step or one or more sets of steps in addition to whole scenarios. Yes I have seen ozzie before and quite like it so I'll dig into their code to see how they achieve it. Help * PR's would be very much appreciated!

I think the problem we have here is that profiling isn't really the concern of BDD / Gherkin so any declarative thing we do will be non-standard which, for me reduces the likelihood of it actually being used. That said if we can make it simple there should be less barriers to use it. I would very much like to have a scenario where you could define warning limits of things taking too long etc and the reporters would warn or even error if the required perf is not met. Also, I think for it to be of use, it would need to produce an HTML report.

For the declarative part of this performance profiling i.e.

Scenario: my scenario
    Given I have "Sign in" button in "Login Page"
#! flutter_profile --name "login"  --time_limit_ms 1500
    When I tap on "Sign in" button in "Login Page"
    Then I have "dummy" button in "Main Page"
 #! end flutter_profile
    And I  have "Back" button in "Main Page"

This would have to be done in the base Gherkin library (https://github.com/jonsamwell/dart_gherkin). We would need a new type of matcher similar to (https://github.com/jonsamwell/dart_gherkin/blob/master/lib/src/gherkin/syntax/comment_syntax.dart) that could macth this type of declaration line. My idea would be to make the #! {extension_point_name} {optional parameters} a configurable extension point that anyone could setup via config. As we would be doing exactly this in the Flutter Gherkin library. So the parser would parse the line and when it is executed it would call a function defined in the configuration file.

I think this will be a big change as there is not really any way of having execution blocks with the core test runner so I need to give this some thought.

To profile complete scenarios we could either do this via a tag @flutter_profile and a custom hook or via the #! declaration. That way we can start the profiling code before the start of the scenario execution and stop it at the end. We would also have access to the test run configuration to compare to the perf limits that may have been set.

Once all that is done we can create a custom reporter to collate and display the results.

Lots to think about, any comments would be great!

BTW I am not sold on the #! yet so idea are welcome!

lsuhov commented 3 years ago

Hello @jonsamwell , I finally have good findings. After an initial round of deep dive into your code from flutter_gherkin and gherkin, where I also did the proof of concept with traceAction function, I stumbled upon functions provided by test_driver which are not requiring us to add the action parameter.

My full focus was given to Future<Timeline> traceAction ( Future action(), {List<TimelineStream> streams: const [TimelineStream.all], bool retainPriorEvents: false} ) because that is the function documented in https://flutter.dev/docs/cookbook/testing/integration/profiling plus the function used by https://pub.dev/packages/ozzie.

But while looking at the source code of traceAction, I found: Future<void> startTracing ( {List<TimelineStream> streams: const [TimelineStream.all], Duration timeout: kUnusuallyLongTimeout} ) and Future<Timeline> stopTracingAndDownloadTimeline ( {Duration timeout: kUnusuallyLongTimeout} ) which should be able to run outside of your libraries in a hook. For example if we add in some scenarios the @profiling tag, we can run the tracing for that specific scenario(s) and at the end of the BDD run, we can try generating the HTML report with the Reporter from https://github.com/jorgecoca/ozzie.flutter/blob/master/lib/reporter.dart

Raised https://github.com/jonsamwell/flutter_gherkin/pull/83 as one step for enabling the above. Any opinions are welcome.