search

mwpenny / kijiji-scraper

A lightweight node.js module for retrieving and scraping ads from Kijiji
MIT License
92 stars 42 forks source link

Kijiji Mingle API Issue #73

Open jas124 opened 2 months ago

jas124 commented 2 months ago

Is anyone else having this issue? It appears the mingle api is no longer being used by the app?

mymtw commented 2 months ago

+1

mwpenny commented 2 months ago

Looks like they are using a new API. I'll look into this.

For now, you can use the HTML scraper. See https://github.com/mwpenny/kijiji-scraper?tab=readme-ov-file#scraper-options. I just tested it and it works fine other than a small bug I fixed. Searching would use the API scraper in some cases even if you tried to use the HTML scraper. I also updated it to return higher-resolution image URLs. That's all in the latest version now on NPM (6.3.5).

jas124 commented 2 months ago

I've attempted to capture the web requests using Fiddler, but I've had no luck. For some reason, the API isn't appearing. Have you encountered this problem too?

mwpenny commented 2 months ago

The app uses HTTPS. So in addition to capturing app network traffic, you also need to either disable the certificate verification or get it to trust your own certificates.

The typical way of doing this is:

  1. Modify the APK to enable trust for user-added certificate authorities
  2. Install the modified APK and your own CA cert on a device
  3. Change the device's proxy settings to point to your own machine listening for requests

It's a bit more involved here since the Kijiji app (even the oldest still-working version) is using Flutter, which does not use the device's certificates or proxy settings. So the app's certificate checks need to be patched out and traffic can be rerouted using something like a VPN or rooted device + iptables rules.

jas124 commented 2 months ago

On iOS you can use a proxy or capture over usb. But now nothing related to the api was showing up on Fiddler. Using wireshark I was able to see some requests to api.ca-kijiji-production.classifiedscloud.io but it appears they are using TLS v1.2 which is beyond my capabilities.

mwpenny commented 2 months ago

Yeah I can see the requests but am having some issues disabling the certificate checks on my Android device. I'll try with an Android emulator. For now, I've made the HTML scraper the default.

paulwainwright commented 6 days ago

Did anybody get this to work? Is Kijiji not scrapeable? Should I move my code to Facebook market place. Any suggestions on scraping Facebook market place. Do I need a paid services to hide my IP? some are advertising $75 per month. not worth it to me for what I am scraping for.

mymtw commented 6 days ago

30$ per month for api of service that can scrape kijiji