search

llermaly / damagedecoder

4 stars 2 forks source link

Get parts database #6

Open llermaly opened 9 months ago

llermaly commented 9 months ago

This databases tend to be paid, so probably we will end up with a reduced version which is fine, we can adapt to it.

I will report and close when ready.

llermaly commented 9 months ago

I decided to just grab a couple of specific cars, crawl every single spare part and build the examples based on them for many reasons:

  1. There is no reason to support so many cars
  2. Trying to ingest everything will break any free tier (size is 10GB)
  3. No people will be testing with real data anyways.
  4. Crawling will be painful (throttling/bans/etc)

Having a list of examples to test with and giving the ability to the users to change the images will be enough to make the point.

llermaly commented 9 months ago

I created a crawler with crawlee, so eventually we can get cars from anywhere on: https://ford.oempartsonline.com , the complement with the agent that googles will be nice so we can use any car we want.