search

BelgianNoise / colruyt-products-scraper

An application written in Go that scrapes Colruyt's API to retrieve all product listings.
https://colruyt-prijzen.nasaj.be/
6 stars 1 forks source link

Puppeteer for X-CG-APIKey #12

Open Merilairon opened 3 days ago

Merilairon commented 3 days ago

Hey,

I have been running this scraper for a while now on my own server for Zottegem. I was just wondering why you are using Puppeteer to get the X-CG-APIKey as this is also available on the following endpoint:

https://www.colruyt.be/content/clp/nl.model.json

Feels like a lot of overhead of puppeteer could be avoided by using this endpoint to get the api key unless I am missing something else ofc...

BelgianNoise commented 3 days ago

I honestly just did whatever came to mind at that time. Didn't know it was in that json file :) Might change it later cause i don't think puppeteer was running correctly in gh actions. I just set that key via the env var, which was a quick patch lol.

BelgianNoise commented 3 days ago

@Merilairon do you run the scraper with proxies on gh actions? If so, which ones?

Merilairon commented 3 days ago

I've moved the whole setup to docker as it was running strangely on github actions. I'm using packet-stream right now. No issues so far.