30350n / inventree_part_import

CLI to import parts from suppliers like DigiKey, LCSC, Mouser, etc. to InvenTree
MIT License
24 stars 8 forks source link

Note: Don't do scheduled updates. Stress the APIs as little as possible. #27

Open sle118 opened 3 months ago

sle118 commented 3 months ago

wanting to possibly schedule updating parts automatically, I enabled using "/" to process recursively all categories. Not sure if this could be useful for others, but I will park this here:

inventree_helpers.py

def get_category(inventree_api: InvenTreeAPI, category_path):
    name = category_path.split("/")[-1]
    categories = PartCategory.list(inventree_api, search=name)
    if category_path == "/":
        return [c for c in categories if c.parent == None]
    for category in categories:
        if category.pathstring == category_path:
            return [category]

    return None

def get_category_parts(part_categories: list[PartCategory], cascade):
        parts=[] 
        for part_category in part_categories:
            if len(part_categories) > 0:
               info(f'Getting parts from category {part_category.name}')
            parts.extend(Part.list(
                part_category._api,
                category=part_category.pk,
                cascade=cascade,
                purchaseable=True,
            ))
        return parts

cli.py (optional

        if not (categories := get_category(inventree_api, category_path)):
            error(f"no such category '{category_path}'")
            return
        parts = [part for part in get_category_parts(categories, bool(update_recursive))]
30350n commented 3 months ago

FYI this tool in it's current form (and all other similar ones) break all the TOSs of all the APIs in use here. So scheduled updates for what it's worth are a really terrible idea.

30350n commented 3 months ago

Also funny: I just got an E-Mail from TME a few hours ago, stating that they'll limit API requests to 1 request every 2 seconds for some actions.

So yeah, in the interest of everyone using these APIs, it's best to use them as little as possible.

sle118 commented 3 months ago

this tool in it's current form (and all other similar ones

I guess I did not read the fine prints, then. It is a shame, though, as getting current pricing for parts would be nice. But I can certainly live with a shortcut to update prices when needed.

30350n commented 3 months ago

Yeah definitely. In the long run, ideally this functionality would be integrated directly into InvenTree, which might make things easier (as the main problem with the TOS is that they don't allow storing stuff like pricing information, so getting them only on demand would be fine afaik).

Also from what I noticed atleast, they don't change that much and if they do, with the current system it's nice and easy to see how much they did.

sle118 commented 3 months ago

I understand better now. As for integration with InvenTree, I think that it would ease the creation of new parts from these vendors. The workflow would be simple: identify the part using the part selector that is the most responsive (sometimes Digikey is realllly slow here in Canada) and then from InvenTree search from all supported suppliers. The way you built this would make it not only to integrate, but also to support new suppliers (Aliexpress, Alibaba, etc).

You saved me a lot of manual entry (which I quickly compensated with debugging and hacking around)