Complete information from Web Scraper

[diesalbla]

The GooglePlay subsystem obtains the data for the cards from two sources. One of these is the Web Scraper, which fetches web pages from the Play Store, such as this one, and extracts from the html document the information for the card. To analyse the html documents, it uses the standard scala XML library.

The goal of this ticket is to extend the implementation, contained in the object GooglePlayPageParser, to provide all the information needed for the answers of the endpoint created in ticket #507:

• The field free, a boolean, to indicate if the app is free to install.
• The field icon, a string, which should be an URL to the Icon of the application.
• The field stars, a number, with the average star rating score of the app.
• The field downloads, a string, which indicates (in ranges) the number of downloads of this app.

Note that this ticket is only concerned with the AppCard objects created in ticket #507.

[noelmarkham]

Just be aware this uses more than the standard Scala XML implementation - it uses TagSoup for making sense of HTML - https://github.com/47deg/nine-cards-backend-google-play/blob/ac94a315cefcf810c4a4b50a2609cbb776ce6220/src/main/scala/com/fortysevendeg/ninecards/googleplay/service/free/interpreter/Converters.scala#L4

[diesalbla]

Here is the PR for this ticket.

[diesalbla]

These are the instructions for performing the QA. Done on a Ubuntu machine

• Setup the computer to run the application. This means installing the protocol buffer parsers and redis. These are detailed here.
• Before executing the application, clear the redis cache.
• Compile and run the application.

For the tests, we use the endpoint to get one app card, GET http://localhost:8081/googleplay/cards/{packageName}. Using the client curl, a full request for the Youtube App would be like the following one:

 curl -H "X-Google-Play-Token: token"\
               -H "X-Android-ID: androidId"\
               -H "X-Android-Market-Localization: en-US"\
              "http://localhost:8081/googleplay/cards/com.google.android.youtube"

The QA should compare this with the information shown in a web browser by the web page of Google Play, such as [https://play.google.com/store/apps/details?id=com.google.android.youtube&hl=en-US].

The QA should perform this request on several packages, and for each of them it should check the following:

• That the name is as the one shown in the web
• The the value of starRating, rounded to the first decimal, would be what the page shows.
• That the value of free is true only if the app has no price.
• That the link shown in icon can be directly opened in a browser (or passed to a curl command, and it shows the image of the icon.

In order to make this exhaustive enough, the QA should try at least a non-free app, such as com.mojang.minecraftpe.

[franciscodr]

QA passed successfully. Good job!!

[franciscodr]

You should close the PR https://github.com/47deg/nine-cards-backend-google-play/pull/12 because it was created against a branch that has been already merged into master