30350n / inventree_part_import

CLI to import parts from suppliers like DigiKey, LCSC, Mouser, etc. to InvenTree
MIT License
24 stars 8 forks source link

Feature: Custom prompt format for multiple supplier part match #23

Closed sle118 closed 3 months ago

sle118 commented 3 months ago

When deciding on which supplier part to use for a given supplier, availability, packaging and pricing are the three things I look at before clicking on a link to check for compatibility and specs, but I suspect that other users could have different requirements. So I wanted to give you my modest contribution for consideration. The idea is to provide users with a way to decide on the choice format themselves, falling back on a default format if not provided.


class ApiPartProcessor:
    members = get_config().get("choice_format","MPN,quantity_available,price_breaks,manufacturer,parameters.Package / Case,SKU,supplier_link")
    @staticmethod
    def normalize(sl:list[str]):
        max_len = max(len(itm) for itm in sl)
        return [itm.ljust(max_len) for itm in sl]
    @staticmethod
    def extract_nested_attribute(obj, path):
        """Recursively extract nested attributes based on a dynamic path."""
        if not isinstance(obj, dict):
            if hasattr(obj, '__dict__'):
                obj = vars(obj)  
            else:
                return ''  

        parts = path.split('.', 1)  # Split at the first dot only

        # Handle dictionary key in brackets or direct key
        if '[' in parts[0] and ']' in parts[0]:
            key = parts[0][parts[0].find('[')+1:parts[0].find(']')]
            current_value = obj.get(key, '')
        else:
            current_value = obj.get(parts[0], '')

        if len(parts) == 1:  # Base case: no more parts
            return current_value
        else:  # Recursive case: more parts to process
            if isinstance(current_value, dict) or hasattr(current_value, '__dict__'):
                return ApiPartProcessor.extract_nested_attribute(current_value, parts[1])
            else:
                return ''  

    @staticmethod
    def format_attribute(attribute, member_name):
        """Format a single attribute based on its type and member name."""
        if member_name.endswith("supplier_link") and attribute:
            return f"({attribute})"
        elif isinstance(attribute, dict):
            key, value = next(iter(attribute.items()), ('', ''))
            return f"{value} [{key}]" if key else ''
        elif isinstance(attribute, list):
            return ", ".join(map(str, attribute))
        else:
            return str(attribute)

    @staticmethod
    def get_formatted_attributes(api_parts: list, member_name: str):
        """Get formatted attributes for a given member across all api_parts."""
        formatted_attributes = []
        for api_part in api_parts:
            attribute = ApiPartProcessor.extract_nested_attribute(api_part, member_name)
            formatted_attribute = ApiPartProcessor.format_attribute(attribute, member_name)
            formatted_attributes.append(formatted_attribute)
        return ApiPartProcessor.normalize(formatted_attributes)

    @staticmethod
    def select_api_part(api_parts: list):
        """Get formatted choice list."""
        formatted_data = {member: ApiPartProcessor.get_formatted_attributes(api_parts, member) for member in ApiPartProcessor.members.split(",")}

        links = formatted_data.pop("supplier_link", [])
        non_link_parts = zip(*(formatted_data[member] for member in formatted_data))
        choices = ["{} {}".format(" | ".join(non_link_part), link) for non_link_part, link in zip(non_link_parts, links)]
        return choices

I am more fluent in C than in Python, so there's a chance this isn't the ideal implementation, but at least it's food for thoughts.

Below is an output example:

searching for BSS138 ...

found multiple parts at DigiKey, select which one to import:
> BSS138 | 150000 | 0.0596 [3000]  | onsemi                              | TO-236-3, SC-59, SOT-23-3 | BSS138TR-ND      (https://www.digikey.ca/en/products/detail/onsemi/BSS138/244210)                                                                          
  BSS138 | 164320 | 0.02786 [3000] | ANBON SEMICONDUCTOR (INT'L) LIMITED | TO-236-3, SC-59, SOT-23-3 | 4530-BSS138TR-ND (https://www.digikey.ca/en/products/detail/anbon-semiconductor-int-l-limited/BSS138/16708474)                                             
  BSS138 | 57000  | 0.03091 [3000] | Good-Ark Semiconductor              | TO-236-3, SC-59, SOT-23-3 | 4786-BSS138TR-ND (https://www.digikey.ca/en/products/detail/good-ark-semiconductor/BSS138/18667774)                                                        
  Skip ...                                                                                                                                                                                                                                                        

It's not ideal as the pricing tear for cut tape doesn't come by default from the digikey wrapper, but inventory and the lowest pricing tier should be a good enough proxy.

Also, I realize that parameters would have to be handled using aliases.

sle118 commented 3 months ago

Adding on to this. Knowing that I probably bastardized bit your work, I did make a change to blindly go after aliases to handle the parameters:

class ApiPartProcessor:
    members = get_config().get("choice_format","MPN,quantity_available,price_breaks,manufacturer,Package Type,SKU,supplier_link")
    def __init__(self,parameter_map) -> None:
        self._parameter_map = parameter_map

    def normalize(self,sl:list[str]):
        max_len = max(len(itm) for itm in sl)
        return [itm.ljust(max_len) for itm in sl]
    def extract_nested_attribute(self,obj, path):
        """Recursively extract nested attributes based on a dynamic path."""
        if not isinstance(obj, dict):
            if hasattr(obj, '__dict__'):
                obj = vars(obj)  
            else:
                return ''  

        parts = path.split('.', 1)  # Split at the first dot only

        # Handle dictionary key in brackets or direct key
        if '[' in parts[0] and ']' in parts[0]:
            key = parts[0][parts[0].find('[')+1:parts[0].find(']')]
            current_value = obj.get(key, '')
        else:
            current_value = obj.get(parts[0], '')

        if len(parts) == 1:  # Base case: no more parts
            if current_value == '':
                return self.get_parameter(obj,path)
            return current_value
        else:  # Recursive case: more parts to process
            if isinstance(current_value, dict) or hasattr(current_value, '__dict__'):
                return self.extract_nested_attribute(current_value, parts[1])
            else:
                return  ''
    def get_parameter(self, api_part: ApiPart, parm_name: str):
        # Use a generator expression instead of a list comprehension
        result = api_part['parameters'].get(parm_name,None)
        if not result:
            gen_expr = (api_part['parameters'].get(alias) for parm in self._parameter_map.get(parm_name, []) for alias in parm.aliases if api_part['parameters'].get(alias) is not None)
            # Use next() to get the first item from the generator, or '' if no items are found
            result = next(gen_expr, '')

        return result

    @staticmethod
    def format_attribute(attribute, member_name):
        """Format a single attribute based on its type and member name."""
        if member_name.endswith("supplier_link") and attribute:
            return f"({attribute})"
        elif isinstance(attribute, dict):
            key, value = next(iter(attribute.items()), ('', ''))
            return f"{value} [{key}]" if key else ''
        elif isinstance(attribute, list):
            return ", ".join(map(str, attribute))
        else:
            return str(attribute)

    def get_formatted_attributes(self,api_parts: list, member_name: str):
        """Get formatted attributes for a given member across all api_parts."""
        formatted_attributes = []
        for api_part in api_parts:
            attribute = self.extract_nested_attribute(api_part, member_name)
            formatted_attribute = self.format_attribute(attribute, member_name)
            formatted_attributes.append(formatted_attribute)
        return self.normalize(formatted_attributes)

    def select_api_part(self,api_parts: list):
        """Get formatted choice list."""
        formatted_data = {member: self.get_formatted_attributes(api_parts, member) for member in self.members.split(",")}

        links = formatted_data.pop("supplier_link", [])
        non_link_parts = zip(*(formatted_data[member] for member in formatted_data))
        choices = ["{} {}".format(" | ".join(non_link_part), link) for non_link_part, link in zip(non_link_parts, links)]
        return choices

and using it:

    def select_api_part(self,api_parts: list[ApiPart]):
        part_selector = ApiPartProcessor(self.parameter_map)
        choices = part_selector.select_api_part(api_parts)

I am aware, though, this is just jumping ahead of the mapping process for a given category, but I can live with the consequences locally 😆

30350n commented 3 months ago

Not sure if this does anything else now, but I feel like the custom ApiPart formatting thing should only take like 10 LOC or so, at max. Will take a look at this later and try a more minimal implementation ^^

sle118 commented 3 months ago

Not sure if this does anything else now, but I feel like the custom ApiPart formatting thing should only take like 10 LOC or so, at max. Will take a look at this later and try a more minimal implementation ^^

This is me, brute forcing my way in and also getting a proof of concept. I am certain that a more elegant solution would involve a new method against ApiPart perhaps. So I will not feel offended if you don't reuse my code; consider it as a proof of concept for some enhancements.

In the meantime, it did serve my purpose of loading all my parts into the system, so thanks for that. I also had a command line prompt opened in parallel to my BOM import so I could search and add the parts for which no match was found in my existing list. This works so well that in the end, I think the plugin could very well be integrated in a workflow where one would want to search for new parts from the UI, pulling from known vendors.

but this is my 2 cents.

30350n commented 3 months ago

This is me, brute forcing my way in and also getting a proof of concept. I am certain that a more elegant solution would involve a new method against ApiPart perhaps. So I will not feel offended if you don't reuse my code; consider it as a proof of concept for some enhancements.

Gotcha. Thanks for the idea though! If you update from master there should be a part_selection_format config option now (usage is documented here).

In the meantime, it did serve my purpose of loading all my parts into the system, so thanks for that.

No problem, that's exactly why I made this, I also collected a whole bunch of parts over the years that I had to stocktake into InvenTree 😅

This works so well that in the end, I think the plugin could very well be integrated in a workflow where one would want to search for new parts from the UI, pulling from known vendors.

I pretty much only use DigiKeys parametric search to look for fitting parts and implementing something similar here would be out of scope for this tool. But if what's already here works for you, that's great ^^

sle118 commented 3 months ago

It does indeed work. I checked your implementation and I guess the icing on the cake would be to also expose the object parameters (after normalization) to reach down to the footprint info. Although searching for a part first before using the tool is probably also going to work for single parts at a time... now that I've loaded my purchase history from Digikey 😃

sle118 commented 3 months ago

Oh one thing about tier pricing. It does look like the tool only gets the full reels. It would be great to also pull pricing tiers for tapes, as this is how I purchase these...

30350n commented 3 months ago

It does indeed work. I checked your implementation and I guess the icing on the cake would be to also expose the object parameters (after normalization) to reach down to the footprint info.

Not quite the icing I guess, but this now works pre normalization. So you can for example use a format string like {parameters[Package / Case] to get the "Package / Case" parameter for digikey parts. The formatter will also no longer fail on unknown keys, so you won't get an error if you use it with another supplier, you will just get an empty value (which in theory allows you to chain multiple of those expressions to do your own quasi-normalization).

30350n commented 3 months ago

Oh one thing about tier pricing. It does look like the tool only gets the full reels. It would be great to also pull pricing tiers for tapes, as this is how I purchase these...

As you already seemed to notice, this is mainly due to some recent changes in the DigiKey API (#14).

sle118 commented 3 months ago

Yup. I did stumble on the other issue for the upgrade to the 4.0 API