search

hildogjr / KiCost

Build cost spreadsheet for a KiCad project.
MIT License
496 stars 97 forks source link

Components with `manf#` and `digikey#` may create empty groups (crashing spreadsheet creation) #304

Closed MrSurly closed 5 years ago

MrSurly commented 5 years ago

I pulled the fix for #302.

I instrumented the code in spreadsheet.py, added a debug breakpoint:

    # Then, order the part references with priority ref prefix, ref num, and subpart num.
    def get_ref_key(part):
        match = re.match(PART_REF_REGEX, part.collapsed_refs)
        try:
            return [match.group('prefix'), match.group('ref_num'), match.group('subpart_num')]
        except Exception as e:
            import pdb; pdb.set_trace()
            raise e

In this case, part.collapsed_refs is an empty string (i.e. repr(part.collapsed_refs) yields '').

$ kicost -i main.xml --include digikey  -var bat_pwr --overwrite -d1 --throttling_delay 0 --currency USD
Exchange rate: 1 EUR = 1.18 USD
# Getting from XML 'main.xml' KiCad BoM...
Getting authorship data...
Getting parts library...
Getting components...
# Removing do not populate parts...
Splitting subparts in the manufacture / distributors codes...
# Grouping parts...
Getting groups of identical components...
Checking the seemingly identical parts group...
Merging field asked in the identical components groups...
Propagating field values to identical components...
Create HTML page for parts with custom pricing...
Initialising digikey                                                                                                                                                                                               
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
# Configuring the distributors locale and currency...                                                                                                                                                              
https://www.digikey.com:443 "GET /en/resources/international HTTP/1.1" 200 None                                                                                                                                    
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
# Scraping part data for each component group...                                                                                                                                                                   
Scraping digikey                                                                                                                                                                                                   
https://www.digikey.com:443 "GET /products/en?keywords=TLV8544PWR HTTP/1.1" 200 None                                                                                                                               
https://www.digikey.com:443 "GET /product-detail/en/texas-instruments/TLV8544PWR/296-47855-2-ND/6910133 HTTP/1.1" 200 None                                                                                         
https://www.digikey.com:443 "GET /product-detail/en/texas-instruments/TLV8544PWR/296-47855-1-ND/8133204 HTTP/1.1" 200 None                                                                                         
https://www.digikey.com:443 "GET /product-detail/en/texas-instruments/TLV8544PWR/296-47855-6-ND/8133377 HTTP/1.1" 200 None                                                                                         
https://www.digikey.com:443 "GET /products/en?keywords=1276-1946-2-ND HTTP/1.1" 403 None                                                                                                                           
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=1276-1946-2-ND HTTP/1.1" 200 None                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL10B105KP8NNNC/1276-1946-1-ND/3890032 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL10B105KP8NNNC/1276-1946-6-ND/3892460 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /products/en?keywords=C3216X5R0J336M130AC HTTP/1.1" 200 None                                                                                                                      
https://www.digikey.com:443 "GET /product-detail/en/tdk-corporation/C3216X5R0J336M130AC/445-4060-2-ND/1965648 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /product-detail/en/tdk-corporation/C3216X5R0J336M130AC/445-4060-1-ND/1965707 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /product-detail/en/tdk-corporation/C3216X5R0J336M130AC/445-4060-6-ND/1965766 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603FT4K70TR-ND HTTP/1.1" 403 None                                                                                                                      
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603FT4K70TR-ND HTTP/1.1" 200 None                                                                                                                      
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603FT4K70/RMCF0603FT4K70CT-ND/1943033 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603FT4K70/RMCF0603FT4K70DKR-ND/1943939 HTTP/1.1" 200 None                                                                       
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603FG4K70/RMCF0603FG4K70TR-ND/1714016 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /products/en?keywords=AK9753AE HTTP/1.1" 200 None                                                                                                                                 
https://www.digikey.com:443 "GET /product-detail/en/akm-semiconductor-inc/AK9753AE/974-1157-2-ND/7388309 HTTP/1.1" 200 None                                                                                        
https://www.digikey.com:443 "GET /product-detail/en/akm-semiconductor-inc/AK9753AE/974-1157-1-ND/7388315 HTTP/1.1" 200 None                                                                                        
https://www.digikey.com:443 "GET /product-detail/en/akm-semiconductor-inc/AK9753AE/974-1157-6-ND/7388321 HTTP/1.1" 200 None                                                                                        
https://www.digikey.com:443 "GET /products/en?keywords=CL10B103KB8NNNC HTTP/1.1" 403 None                                                                                                                          
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=CL10B103KB8NNNC HTTP/1.1" 200 None                                                                                                                          
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL10B103KB8NNNC/1276-1009-2-ND/3886667 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL10B103KB8NNNC/1276-1009-1-ND/3889095 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL10B103KB8NNNC/1276-1009-6-ND/3891523 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /products/en?keywords=IRS-B210ST01-R1 HTTP/1.1" 200 None                                                                                                                          
https://www.digikey.com:443 "GET /product-detail/en/murata-electronics-north-america/IRS-B210ST01-R1/490-12776-2-ND/4420883 HTTP/1.1" 200 None                                                                     
https://www.digikey.com:443 "GET /product-detail/en/murata-electronics-north-america/IRS-B210ST01-R1/490-12776-1-ND/5797745 HTTP/1.1" 200 None                                                                     
https://www.digikey.com:443 "GET /product-detail/en/murata-electronics-north-america/IRS-B210ST01-R1/490-12776-6-ND/5798010 HTTP/1.1" 200 None                                                                     
No 'digikey#' or 'manf#' field: cannot lookup part [] at digikey.                                                                                                                                                  
No 'digikey#' or 'manf#' field: cannot lookup part [] at digikey.                                                                                                                                                  
https://www.digikey.com:443 "GET /products/en?keywords=GRM31CD80J107ME39L HTTP/1.1" 403 None                                                                                                                       
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=GRM31CD80J107ME39L HTTP/1.1" 200 None                                                                                                                       
https://www.digikey.com:443 "GET /product-detail/en/murata-electronics-north-america/GRM31CD80J107ME39L/490-10525-2-ND/5027609 HTTP/1.1" 200 None                                                                  
https://www.digikey.com:443 "GET /product-detail/en/murata-electronics-north-america/GRM31CD80J107ME39L/490-10525-1-ND/5026461 HTTP/1.1" 200 None                                                                  
https://www.digikey.com:443 "GET /product-detail/en/murata-electronics-north-america/GRM31CD80J107ME39L/490-10525-6-ND/5027069 HTTP/1.1" 200 None                                                                  
https://www.digikey.com:443 "GET /products/en?keywords=939-1066-1-ND HTTP/1.1" 200 None                                                                                                                            
https://www.digikey.com:443 "GET /product-detail/en/ethertronics-avx/M620720/939-1066-2-ND/6072490 HTTP/1.1" 200 None                                                                                              
https://www.digikey.com:443 "GET /product-detail/en/ethertronics-avx/M620720/939-1066-6-ND/6072498 HTTP/1.1" 200 None                                                                                              
https://www.digikey.com:443 "GET /products/en?keywords=CL10B102KB8SFNC HTTP/1.1" 403 None                                                                                                                          
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=CL10B102KB8SFNC HTTP/1.1" 200 None                                                                                                                          
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL10B102KB8SFNC/1276-1916-2-ND/3887574 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL10B102KB8SFNC/1276-1916-1-ND/3890002 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL10B102KB8SFNC/1276-1916-6-ND/3892430 HTTP/1.1" 200 None                                                                            
Problem merging price tiers for Digikey part CL10B102KB8SFNC with alternate packaging!                                                                                                                             
Problem merging price tiers for Digikey part CL10B102KB8SFNC with alternate packaging!                                                                                                                             
https://www.digikey.com:443 "GET /products/en?keywords=587-1210-2-ND HTTP/1.1" 200 None                                                                                                                            
https://www.digikey.com:443 "GET /product-detail/en/taiyo-yuden/UMK105CG820JV-F/587-1210-1-ND/930987 HTTP/1.1" 200 None                                                                                            
https://www.digikey.com:443 "GET /product-detail/en/taiyo-yuden/UMK105CG820JV-F/587-1210-6-ND/1817994 HTTP/1.1" 200 None                                                                                           
Problem merging price tiers for Digikey part 587-1210-2-ND with alternate packaging!                                                                                                                               
Problem merging price tiers for Digikey part 587-1210-2-ND with alternate packaging!                                                                                                                               
https://www.digikey.com:443 "GET /products/en?keywords=C2012X7R1C335K125AB HTTP/1.1" 403 None                                                                                                                      
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=C2012X7R1C335K125AB HTTP/1.1" 200 None                                                                                                                      
https://www.digikey.com:443 "GET /product-detail/en/tdk-corporation/C2012X7R1C335K125AB/445-7582-2-ND/2733017 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /product-detail/en/tdk-corporation/C2012X7R1C335K125AB/445-7582-1-ND/2733654 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /product-detail/en/tdk-corporation/C2012X7R1C335K125AB/445-7582-6-ND/2734292 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /products/en?keywords=CRCW0603560KJNEA HTTP/1.1" 200 None                                                                                                                         
https://www.digikey.com:443 "GET /product-detail/en/vishay-dale/CRCW0603560KJNEA/541-560KGTR-ND/1174295 HTTP/1.1" 200 None                                                                                         
https://www.digikey.com:443 "GET /product-detail/en/vishay-dale/CRCW0603560KJNEA/541-560KGCT-ND/1179435 HTTP/1.1" 200 None                                                                                         
https://www.digikey.com:443 "GET /product-detail/en/vishay-dale/CRCW0603560KJNEA/541-560KGDKR-ND/1187401 HTTP/1.1" 200 None                                                                                        
https://www.digikey.com:443 "GET /product-detail/en/vishay-dale/CRCW0603560KJNEB/CRCW0603560KJNEB-ND/5072850 HTTP/1.1" 200 None                                                                                    
https://www.digikey.com:443 "GET /products/en?keywords=BAS16GWJ HTTP/1.1" 403 None                                                                                                                                 
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=BAS16GWJ HTTP/1.1" 200 None                                                                                                                                 
https://www.digikey.com:443 "GET /product-detail/en/nexperia-usa-inc/BAS16GWJ/1727-7320-2-ND/7390545 HTTP/1.1" 200 None                                                                                            
https://www.digikey.com:443 "GET /product-detail/en/nexperia-usa-inc/BAS16GWJ/1727-7320-1-ND/7390571 HTTP/1.1" 200 None                                                                                            
https://www.digikey.com:443 "GET /product-detail/en/nexperia-usa-inc/BAS16GWJ/1727-7320-6-ND/7390597 HTTP/1.1" 200 None                                                                                            
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603JT15M0 HTTP/1.1" 200 None                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT15M0/RMCF0603JT15M0TR-ND/1757981 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT15M0/RMCF0603JT15M0CT-ND/6053182 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT15M0/RMCF0603JT15M0DKR-ND/6053510 HTTP/1.1" 200 None                                                                       
https://www.digikey.com:443 "GET /products/en?keywords=CR0603-JW-682GLF HTTP/1.1" 403 None                                                                                                                         
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=CR0603-JW-682GLF HTTP/1.1" 403 None                                                                                                                         
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=CR0603-JW-682GLF+Bourns+Inc. HTTP/1.1" 200 None                                                                                                             
https://www.digikey.com:443 "GET /product-detail/en/bourns-inc/CR0603-JW-682GLF/CR0603-JW-682GLFTR-ND/3784418 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /product-detail/en/bourns-inc/CR0603-JW-682ELF/CR0603-JW-682ELFTR-ND/3784417 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /product-detail/en/bourns-inc/CR0603-JW-682ELF/CR0603-JW-682ELFCT-ND/4247892 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /product-detail/en/bourns-inc/CR0603-JW-682ELF/CR0603-JW-682ELFDKR-ND/4248296 HTTP/1.1" 200 None                                                                                  
https://www.digikey.com:443 "GET /product-detail/en/bourns-inc/CR0603-JW-682GLF/CR0603-JW-682GLFCT-ND/3925376 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /products/en?keywords=DMP3125L-7DITR-ND HTTP/1.1" 403 None                                                                                                                        
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=DMP3125L-7DITR-ND HTTP/1.1" 200 None                                                                                                                        
https://www.digikey.com:443 "GET /product-detail/en/diodes-incorporated/DMP3125L-7/DMP3125L-7DICT-ND/8275361 HTTP/1.1" 200 None                                                                                    
https://www.digikey.com:443 "GET /product-detail/en/diodes-incorporated/DMP3125L-7/DMP3125L-7DIDKR-ND/8275364 HTTP/1.1" 200 None                                                                                   
https://www.digikey.com:443 "GET /products/en?keywords=1276-1122-2-ND HTTP/1.1" 200 None                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL21A106KQFNNNE/1276-1122-1-ND/3889208 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL21A106KQFNNNE/1276-1122-6-ND/3891636 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL21A106KQFNNNG/1276-6457-2-ND/3894422 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL21A106KQFNNNG/1276-6457-1-ND/5958085 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL21A106KQFNNNG/1276-6457-6-ND/5958148 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL21A106KQFNNNF/CL21A106KQFNNNF-ND/3894421 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /products/en?keywords=160-1830-2-ND HTTP/1.1" 403 None                                                                                                                            
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=160-1830-2-ND HTTP/1.1" 200 None                                                                                                                            
https://www.digikey.com:443 "GET /product-detail/en/lite-on-inc/LTST-C193KRKT-5A/160-1830-1-ND/2356251 HTTP/1.1" 200 None                                                                                          
https://www.digikey.com:443 "GET /product-detail/en/lite-on-inc/LTST-C193KRKT-5A/160-1830-6-ND/2356252 HTTP/1.1" 200 None                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=RC0603JR-071M5L+Yageo HTTP/1.1" 200 None                                                                                                                    
https://www.digikey.com:443 "GET /product-detail/en/yageo/RC0603JR-071M5L/311-1.5MGRTR-ND/726690 HTTP/1.1" 200 None                                                                                                
https://www.digikey.com:443 "GET /product-detail/en/yageo/RC0603JR-071M5L/311-1.5MGRCT-ND/729637 HTTP/1.1" 200 None                                                                                                
https://www.digikey.com:443 "GET /product-detail/en/yageo/RC0603JR-071M5L/311-1.5MGRDKR-ND/732584 HTTP/1.1" 200 None                                                                                               
https://www.digikey.com:443 "GET /products/en?keywords=68021-406HLF HTTP/1.1" 403 None                                                                                                                             
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=68021-406HLF HTTP/1.1" 200 None                                                                                                                             
Problem parsing URLs from product page for 68021-406HLF from digikey                                                                                                                                               
https://www.digikey.com:443 "GET /products/en?keywords=490-16143-2-ND HTTP/1.1" 200 None                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/murata-electronics-north-america/CMWX1ZZABZ-078/490-16143-1-ND/6834151 HTTP/1.1" 200 None                                                                      
https://www.digikey.com:443 "GET /product-detail/en/murata-electronics-north-america/CMWX1ZZABZ-078/490-16143-6-ND/6834152 HTTP/1.1" 200 None                                                                      
https://www.digikey.com:443 "GET /products/en?keywords=CRG0603ZR HTTP/1.1" 200 None                                                                                                                                
https://www.digikey.com:443 "GET /product-detail/en/te-connectivity-passive-product/CRG0603ZR/A106051TR-ND/2380828 HTTP/1.1" 200 None                                                                              
https://www.digikey.com:443 "GET /product-detail/en/te-connectivity-passive-product/CRG0603ZR/A106051CT-ND/3477691 HTTP/1.1" 200 None                                                                              
https://www.digikey.com:443 "GET /product-detail/en/te-connectivity-passive-product/CRG0603ZR/A106051DKR-ND/3477830 HTTP/1.1" 200 None                                                                             
https://www.digikey.com:443 "GET /products/en?keywords=609-5154-2-ND HTTP/1.1" 403 None                                                                                                                            
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=609-5154-2-ND HTTP/1.1" 200 None                                                                                                                            
https://www.digikey.com:443 "GET /product-detail/en/amphenol-fci/89898-303ALF/609-5154-1-ND/5967709 HTTP/1.1" 200 None                                                                                             
https://www.digikey.com:443 "GET /product-detail/en/amphenol-fci/89898-303ALF/609-5154-6-ND/5967710 HTTP/1.1" 200 None                                                                                             
https://www.digikey.com:443 "GET /products/en?keywords=1276-1601-2-ND HTTP/1.1" 200 None                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL05C040CB5NNNC/1276-1601-1-ND/3889687 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /product-detail/en/samsung-electro-mechanics/CL05C040CB5NNNC/1276-1601-6-ND/3892115 HTTP/1.1" 200 None                                                                            
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603JT68K0 HTTP/1.1" 403 None                                                                                                                           
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603JT68K0 HTTP/1.1" 200 None                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT68K0/RMCF0603JT68K0TR-ND/1758034 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT68K0/RMCF0603JT68K0CT-ND/1943201 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT68K0/RMCF0603JT68K0DKR-ND/1944107 HTTP/1.1" 200 None                                                                       
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JG68K0/RMCF0603JG68K0-ND/1712327 HTTP/1.1" 200 None                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603JT10K0TR-ND HTTP/1.1" 403 None                                                                                                                      
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603JT10K0TR-ND HTTP/1.1" 200 None                                                                                                                      
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JG10K0/RMCF0603JG10K0TR-ND/1712133 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JG10K0/RMCF0603JG10K0CT-ND/4425128 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT10K0/RMCF0603JT10K0CT-ND/1943191 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT10K0/RMCF0603JT10K0DKR-ND/1944097 HTTP/1.1" 200 None                                                                       
https://www.digikey.com:443 "GET /products/en?keywords=2N7002-TP HTTP/1.1" 403 None                                                                                                                                
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=2N7002-TP HTTP/1.1" 200 None                                                                                                                                
https://www.digikey.com:443 "GET /product-detail/en/micro-commercial-co/2N7002-TP/2N7002-TPMSTR-ND/1959296 HTTP/1.1" 200 None                                                                                      
https://www.digikey.com:443 "GET /product-detail/en/micro-commercial-co/2N7002-TP/2N7002-TPMSCT-ND/1960099 HTTP/1.1" 200 None                                                                                      
https://www.digikey.com:443 "GET /product-detail/en/micro-commercial-co/2N7002-TP/2N7002-TPMSDKR-ND/1960525 HTTP/1.1" 200 None                                                                                     
No 'digikey#' or 'manf#' field: cannot lookup part ['FID101', 'FID102', 'FID103'] at digikey.                                                                                                                      
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603JT1M30 HTTP/1.1" 200 None                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT1M30/RMCF0603JT1M30TR-ND/1757939 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT1M30/RMCF0603JT1M30CT-ND/2418183 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT1M30/RMCF0603JT1M30DKR-ND/2427077 HTTP/1.1" 200 None                                                                       
https://www.digikey.com:443 "GET /products/en?keywords=SIP32431DNP3-T1GE4TR-ND HTTP/1.1" 403 None                                                                                                                  
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 43610                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=SIP32431DNP3-T1GE4TR-ND HTTP/1.1" 200 22640                                                                                                                 
https://www.digikey.com:443 "GET /product-detail/en/vishay-siliconix/SIP32431DNP3-T1GE4/SIP32431DNP3-T1GE4CT-ND/2673920 HTTP/1.1" 200 22616                                                                        
https://www.digikey.com:443 "GET /product-detail/en/vishay-siliconix/SIP32431DNP3-T1GE4/SIP32431DNP3-T1GE4DKR-ND/2673928 HTTP/1.1" 200 22932                                                                       
https://www.digikey.com:443 "GET /products/en?keywords=160-1828-2-ND HTTP/1.1" 200 22300                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/lite-on-inc/LTST-C193KGKT-5A/160-1828-1-ND/2356247 HTTP/1.1" 200 22288                                                                                         
https://www.digikey.com:443 "GET /product-detail/en/lite-on-inc/LTST-C193KGKT-5A/160-1828-6-ND/2356248 HTTP/1.1" 200 22588                                                                                         
https://www.digikey.com:443 "GET /products/en?keywords=455-1734-2-ND+JST HTTP/1.1" 403 None                                                                                                                        
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=455-1734-2-ND+JST HTTP/1.1" 200 None                                                                                                                        
https://www.digikey.com:443 "GET /product-detail/en/jst-sales-america-inc/B2B-PH-SM4-TB-LF-SN/455-1734-1-ND/926831 HTTP/1.1" 200 None                                                                              
https://www.digikey.com:443 "GET /product-detail/en/jst-sales-america-inc/B2B-PH-SM4-TB-LF-SN/455-1734-6-ND/1059117 HTTP/1.1" 200 None                                                                             
https://www.digikey.com:443 "GET /products/en?keywords=1904-1010-2-ND HTTP/1.1" 200 None                                                                                                                           
https://www.digikey.com:443 "GET /product-detail/en/espressif-systems/ESP32-WROOM-32/1904-1010-1-ND/8544305 HTTP/1.1" 200 None                                                                                     
https://www.digikey.com:443 "GET /product-detail/en/espressif-systems/ESP32-WROOM-32/1904-1010-6-ND/8544309 HTTP/1.1" 200 None                                                                                     
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603JT330RTR-ND HTTP/1.1" 403 None                                                                                                                      
Starting new HTTPS connection (1): www.digikey.com:443                                                                                                                                                             
https://www.digikey.com:443 "GET / HTTP/1.1" 200 45736                                                                                                                                                             
Received 403, scraper possibly detected: Starting new session for https://www.digikey.com                                                                                                                          
https://www.digikey.com:443 "GET /products/en?keywords=RMCF0603JT330RTR-ND HTTP/1.1" 200 None                                                                                                                      
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT330R/RMCF0603JT330RCT-ND/1943164 HTTP/1.1" 200 None                                                                        
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JT330R/RMCF0603JT330RDKR-ND/1944070 HTTP/1.1" 200 None                                                                       
https://www.digikey.com:443 "GET /product-detail/en/stackpole-electronics-inc/RMCF0603JG330R/RMCF0603JG330RTR-ND/1712243 HTTP/1.1" 200 None                                                                        
Progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 37/37 [01:00<00:00,  1.53s/part]Creating the 'main.xlsx' spreadsheet...
Writing the global parts informations...
Traceback (most recent call last):
  File "/home/epoulsen/.local/bin/kicost", line 11, in <module>
    load_entry_point('kicost==0.1.47', 'console_scripts', 'kicost')()
  File "/home/epoulsen/.local/lib/python2.7/site-packages/kicost-0.1.47-py2.7.egg/kicost/__main__.py", line 286, in main
    local_currency=args.currency)
  File "/home/epoulsen/.local/lib/python2.7/site-packages/kicost-0.1.47-py2.7.egg/kicost/kicost.py", line 348, in kicost
    user_fields, '-'.join(variant) if len(variant)>1 else variant[0])
  File "/home/epoulsen/.local/lib/python2.7/site-packages/kicost-0.1.47-py2.7.egg/kicost/spreadsheet.py", line 182, in create_spreadsheet
    parts, user_fields, collapse_refs)
  File "/home/epoulsen/.local/lib/python2.7/site-packages/kicost-0.1.47-py2.7.egg/kicost/spreadsheet.py", line 468, in add_globals_to_worksheet
    parts.sort(key=get_ref_key)
  File "/home/epoulsen/.local/lib/python2.7/site-packages/kicost-0.1.47-py2.7.egg/kicost/spreadsheet.py", line 467, in get_ref_key
    return [match.group('prefix'), match.group('ref_num'), match.group('subpart_num')]
AttributeError: 'NoneType' object has no attribute 'group'

main.zip

hildogjr commented 5 years ago

Your issue is related with some behavior of your XML file. Debugging, after KiCost group the part, it return:

... ['D301', 'D302'] D301 {'libpart': 'device:D', 'manf#_qty': '1', 'value': 'BAS16GWJ', 'manf#': 'BAS16GWJ', 'spn': '1727-7320-2-ND', 'manf': 'Nexperia USA Inc.', 'footprint': 'Diodes_SMD:D_SOD-123'} D302 {'libpart': 'device:D', 'manf#_qty': '1', 'value': 'BAS16GWJ', 'manf#': 'BAS16GWJ', 'spn': '1727-7320-2-ND', 'manf': 'Nexperia USA Inc.', 'footprint': 'Diodes_SMD:D_SOD-123'} [] [] ['J9'] J9 {'digikey#_qty': '1', 'libpart': 'conn:Conn_01x02', 'manf#_qty': '1', 'value': 'B2B-PH-SM4-TB(LF)(SN)', 'digikey#': '455-1734-2-ND', 'manf': 'JST', 'manf#': 'B2B-PH-SM4-TB(LF)(SN)', 'spr': 'digikey', 'footprint': 'Connectors_JST:JST_PH_B2B-PH-SM4-TB_02x2.00mm_Straight'} ...

The [] means two empty groups of components. I never saw this behavior before, could you provide more information of your schematic? (If used some not conventional filed in the parts, KiCad version and the "Edit table field" list).

I am trying to track this behavior.

hildogjr commented 5 years ago

This issue appear to be created because the group algorithm and component without digikey# field. The error just appear when Digikey distributor is active generating the empty groups.

MrSurly commented 5 years ago

I do a fair amount of Python -- Is this something I could dig into, or is it pretty involved? Maybe give me a few hints around where these hooks are?

hildogjr commented 5 years ago

The problem is in the group_parts() function in the file eda_tools.py, need some correcting / re-manufacture. And I need to merge here the #266. It is a function to group the the parts and propagate the the manf# / digikey# / ... to the semi-identical part (by default KiCost propagate the manf# to parts of same kind, footprint and value, that are empty). All help and new developers are welcome.

hildogjr commented 5 years ago

KiCost works in this sequence:

  1. Read the file (files);
  2. Merge the BOMs (in the case of multiple files);
  3. Group the parts and propagate the manf# (the function here, where is the issue);
  4. Scrape the distributors (as multi-thread); a. Configure the currency and language of the distributor (in beta / just working for Digikey); b. Look for the specific part number; c. Look for tree of part that match if 3b doesn't return;
  5. Assembly the spreadsheet.
mobilinkd commented 5 years ago

I am hitting this bug. Are there any work-arounds for this?

hildogjr commented 5 years ago

This bug are not present on new implementation #314 (will be released in few days). If you would like to have a try, replace the current code installation by https://github.com/xesscorp/KiCost/tree/octopart

Re-open the discussion if the bug persists.