search

benjaminjonard / koillection

Koillection is a self-hosted service allowing users to manage any kind of collections.
https://github.com/koillection/koillection/wiki
MIT License
581 stars 27 forks source link

[Feature Request] Api scrapper field #873

Open tedr56 opened 1 year ago

tedr56 commented 1 year ago

First, thanks for this niece piece of software. I just installed it and I have a few idea for my usecase that others may need.

As mentionned in #721, a data scrapper is really important for big collections fields with public data. I was thinking a Api field with sub Url, Header, Query subfields, that could use previous data fields. For example, I'm working on pokemon cards, but could be anything:

CardNumber: Number
Set: String
ApiData:
  Url: https://api.pokemontcg.io/v2/cards
  Method: Get
  Headers:
    x-api-key: 0123456789
  Query: q=number:{{CardNumber}} set:{{Set}}
Illustration: ApiData.images.large

I know this a big piece of code. I tried to look into it but I'm not that good of a developper. What do you think?

Thanks.

benjaminjonard commented 1 year ago

Hello,

That's something I'd like to implements, but I have yet to find a good way to do that.

As Koillection aims to handle any kind of collections, I don't want to couple it too much with one particular use case. Your example works for that API, that returns json. But I know people wants to scrap HTML too. That's why I added a REST API as a first solution, people can make their own custom scrapping scripts to import items into Koillection.

I'll see if adding a scrapper builder directly into Koillection is doable

tedr56 commented 1 year ago

Ho. I didn't know that. I've seen api commits but did't know the actual use.

I was thinking this kind of big setting could go near template and choice list menu. But then, it would be harder to link it with dynamic fields.

Also, we could imagine a rest scraper field, and an html scraper fields?