everypolitician / scraper_test

Data-driven scraper tests for Scraped
0 stars 0 forks source link

Handle paths in instructions file #23

Closed ondenman closed 7 years ago

ondenman commented 7 years ago

A future PR (WIP: https://github.com/everypolitician/scraper_test/pull/22) will allow the testing of a subset of the response data. In anticipation of that PR, this PR provides a way to specify that subset.

The Instructions class now returns a path specified in the instructions file.

For example, the response data might contain multiple members within an array. A path can now be specified in the instructions file to target the data of a specific member.

{ members: [{id: 1, name: 'Jim' }, { id: 2, name: 'Joe' }] }

Although Instructions doesn't care about the value in the :path field, the path is anticipated to follow the format: key/key=value.

To target Joe, the path would be: members/id=2 (key/key=value).

The yaml file would be:

:class: MembersPage
:url: www.example.com
:path: members/id=2
:to_h:
    :id: 2
    :name: Joe

The path can be used to drill down any number of levels.

:houses:
  :lower:
    :members:
      - :id: 1
        :name: David Bowman
      - :id: 2
        :name: Frank Poole
  :upper:
    :members:
      - :id: 1
        :name: Victor Kaminsky
        :notes:
          - Survey Team Leader
          - Deceased
      - :id: 2
        :name: Jack Kimball
        :notes:
          - Geophysicist
          - Deceased
      - :id: 3
        :name: Charles Hunter
        :notes:
          - Astrophysicist
          - Deceased

To target Victor Kaminsky's notes, the path would be: house/upper/members/id=2/notes.

ondenman commented 7 years ago

Closing in favour of: https://github.com/everypolitician/scraper_test/pull/25