search

apache / cordova

Apache Cordova
https://cordova.apache.org/
622 stars 64 forks source link

Better testing of platform releases regarding plugin compatibility #54

Open janpio opened 5 years ago

janpio commented 5 years ago

The 7.1.3 release of cordova-android had a breaking bug with multiple plugins that included app in their name, including our own cordova-plugin-inappbrowser.

The cordova-android tests did not catch that.

We should figure out a way to better test our platforms with plugins.

janpio commented 5 years ago

The plugin tests (run on CI via paramedic) themselves would probably have caught that, as they create a project and then see if functionality of the plugins work. But those tests are done with the released Cordova CLI and platforms.

We have the mobilespec project, but I am not exactly sure what that is and how it works.

brodycj commented 5 years ago

I just raised https://github.com/apache/cordova-mobile-spec/pull/147 to document a few known quirks & issues:

brodycj commented 5 years ago

and not all of the automatic tests are passing for me. It seems to help to select 1 plugin at a time to run the tests on.

janpio commented 5 years ago

So mobilespec is one thing.

For Android I just discovered this: https://github.com/apache/cordova-android/tree/master/test (Was linked at https://github.com/apache/cordova-coho/blob/master/docs/platforms-release-process.md#android-extras, under https://github.com/apache/cordova-coho/blob/master/docs/platforms-release-process.md#ios-extras mentions something similar for iOS)

brodycj commented 5 years ago

So mobilespec is one thing.

For Android I just discovered this: [...] ([...] mentions something similar for iOS)

Here are the updated Android & iOS unit test links for the sake of clarity, in response to https://github.com/apache/cordova/issues/54#issuecomment-441290852 above:

NOTE that these unit tests are included by npm test and are run by CI, according to https://github.com/apache/cordova/issues/54#issuecomment-442457744 below.

janpio commented 5 years ago

How is this related to plugins @brodybits?

(All those tests are include in npm test and thus run by CI, the Android ones for example are run via npm run java-unit-tests)

brodycj commented 5 years ago

How is this related to plugins @brodybits?

Here is my response, along with a motion below.

I provided updated information in response to some information with an outdated link in https://github.com/apache/cordova/issues/54#issuecomment-441290852. I edited my response to clarify.

(All those tests are include in npm test and thus run by CI, the Android ones for example are run via npm run java-unit-tests)

Thanks for the info, which triggered https://github.com/apache/cordova-coho/pull/213. I added the info to my response above.

I would like to motion the following:

My motion is amended as follows:

And I am doing one more thing as a collaborator: mark the comments hidden so far as "resolved". My rationale is that the first hidden comment I wrote was in response to a comment by someone else (fair discussion) before we decided to apply the strikeout.

brodycj commented 5 years ago

So mobilespec is one thing.

This and https://github.com/apache/cordova-mobile-spec/issues/184#issuecomment-442827057 imply that there are some other ways to test platform releases and maybe some other package releases that I don't know about. Can you enlighten me and others?

janpio commented 5 years ago

So mobilespec is one thing.

This

That was the conclusion of us finding and understanding mobilespec as a way to test plugins compatibility on platforms in the comments before (which this topic is about).

apache/cordova-mobile-spec#184 (comment)

The comment you link to is about plugin testing, not platform releases. Generally plugin testing is done via all the plugin tests, that are run via cordova-paramedic in CI. But that doesn't offer any (simple) options to control the platform being used.

But if you have such questions, please ask them in a new issue, Slack or dev mailing list thread and don't abuse one of my issues that has a clear topic or question - which is not about enlightening you.

janpio commented 5 years ago

Step back:

What would help to "test platform releases regarding plugin compatibility"? How can we find out if a new platform release breaks anything with plugins?

  1. Check if plugins install on the platform
  2. Check if normal app (without plugin usage) still works on platform with plugin installed
  3. Check if plugin tests app (cdvtests/index.html via cordova-plugin-test-framework) still works on platform with plugin installed
  4. Check if plugin Appium tests (currently only runnable on CI via cordova-paramedic) still work on platform with plugin installed

Other ideas?

janpio commented 5 years ago

Could we maybe somehow compare what a plugin installation does between two platform versions? That could catch if changes in our "copy files from a to b" logic (that caused problems in the past) will break a plugin without executing its code.

brodycj commented 5 years ago

I will need some time to evaluate these ideas. I think more comprehensive testing would be good on major releases and occasional minor releases. I would really favor a more straightforward testing procedure for hotfixes and other patch releases.

janpio commented 5 years ago

This exists:

https://github.com/alsorokin/cordova-periodic-build-android https://github.com/alsorokin/cordova-periodic-build-ios

(Take a look at the Travis build history)

Broken right now, but could be a blueprint on how to create a repo that can test platforms with all plugins.