citybikes.com scraper - Githubissues

jkwening / bike-price-predictor

Machine learning project that given bike information will predict price and evaluate value comparatively to a listed price.

GNU General Public License v3.0

1 stars 0 forks source link

citybikes.com scraper #40

Closed jkwening closed 5 years ago

jkwening commented 5 years ago

base = 'https://www.citybikes.com/' products endpoint = 'product-list/bikes-1000 bike category endpoints = 'rb_ct=1001' (1001 = road)

1001 = road
1006 = mountain
1017 = commuter/urban
1020 = comfort
1021 = cruiser
1249 = fitness
1022 = hybrid
1023 = children
1037 = other number items endpoint = '?&maxItems=60' alt = 30 bike spec endpoint = 'product/' product page endpoint = '&startrow=61' (page 2 of 3)

Products category heading

div -> class='secategoriesFacetPanel'
- div -> id='Facets-categories
- li -> class='seFacet
  - a -> href; data-filterparameter; data-filtervalue; title
  - span -> class='badge'.contents (example "(28))

Products - html is fetched

div -> id='seProductsTab'
- div -> id='SearchProducts'
- div -> class='seSearchProductWrapper'
- div -> class='seProduct'

Product Specs

div -> class='seProductTitle'
- a -> href
- span -> class='seBrandName'
- a -> title
div -> class='seProductPrice'
- span -> class='seRegularPrice'
- regular price but could be noted as a range
- span -> class='seSpecialPrice'
- span -> class='seOriginalPrice'
- sale prices used in combination

Page number text (under 'seProductsTab')

div -> class-'sePaginationContainer'
- div -> class='sePaginationWrapper'
- span -> id='PaginationText' -> "Page 2 of 3" - example

Bike specs - html is fetched

div -> id='ProductDetails'
- div -> class='seProductBuyContainer'
- div -> id='Prices'
  - div -> id='RegularPrice'
  - div -> id='SpecialPrice'
  - div -> id='OriginalPrice'
- div -> class='seProductDataContainer'
- div -> class='seProductPaneContainer'
  - div -> id='ProductSpecs'
  - table -> class='seProductSpecTable'
    - tbody -> tr: (1) (2)

jkwening commented 5 years ago

Enhancement: For those bike products that don't have price, fetch product specs and get price from there.