30350n / inventree_part_import

CLI to import parts from suppliers like DigiKey, LCSC, Mouser, etc. to InvenTree
MIT License
23 stars 8 forks source link

Mouser and LCSC crawling is broken #49

Open hillsandales opened 2 weeks ago

hillsandales commented 2 weeks ago

I've tried Digikey, Mouser, and LCSC, but none of them work.

Digikey brings me to what looks like a Lynx interactive for their login page. I can type in username and password, but I'm not sure why I need to if it's using an API private key. Also Lynx won't let me select Login (assuming it's a button) anyway. So there's no way to continue.

Mouser does return a list of parts with my API key, but when I select the part I want, it says warning: failed to get page, retrying in 3.0s via 'www2.mouser.com' with new session and user agent. It tries another eu. url too, but no luck. Then it returns warning: failed to finalize part specifications from 'https://www.mouser.com/ProductDetail/Microchip-Technology/2N4001?qs=7D1LtPJG0i0bXRytgh1wrg%3D%3D' (blocked) and exits saying it failed to import.

LCSC doesn't return anything and says failed to get webpage.

This tool looks like it's a lot simpler than others, which is nice, but I can't get it to work for some reason.

Also, I tried it on WIndows with Git Shell, and the tool doesn't work because the shell doesn't recognize selection with Spacebar and Enter. Tried it on linux, and the script in the bash shell works.

30350n commented 2 weeks ago

Digikey brings me to what looks like a Lynx interactive for their login page. I can type in username and password, but I'm not sure why I need to if it's using an API private key. Also Lynx won't let me select Login (assuming it's a button) anyway. So there's no way to continue.

Not sure what you mean with "Lynx interactive" exactly (google also didn't come up with any useful results). Are you just talking about the OAuth browser popup? If so, I'm honestly also not sure why that's required in addition to the API key, but I think that's just how it is. (See the digikey-api python library and the DigiKey API docs).

Mouser does return a list of parts with my API key, but when I select the part I want, it says warning: failed to get page, retrying in 3.0s via 'www2.mouser.com' with new session and user agent. It tries another eu. url too, but no luck. Then it returns warning: failed to finalize part specifications from 'https://www.mouser.com/ProductDetail/Microchip-Technology/2N4001?qs=7D1LtPJG0i0bXRytgh1wrg%3D%3D' (blocked) and exits saying it failed to import.

Can confirm, this is happening for me too. The problem here is that Mousers API sucks ass, which I've been circumventing by just crawling the page directly. Seems like they recently "hardened" their crawling protection again. That's great.

LCSC doesn't return anything and says failed to get webpage.

Similar thing, LCSC doesn't really offer an API directly, but there's API endpoints which URLs got recently changed (see #47 and #48).

Also, I tried it on WIndows with Git Shell, and the tool doesn't work because the shell doesn't recognize selection with Spacebar and Enter. Tried it on linux, and the script in the bash shell works.

I'm using/developing this from Windows, mainly using the VSCode terminal and Cmder. Standard cmd should also work I'm pretty sure.

hillsandales commented 2 weeks ago

I see. Good to know it's not just me then. Thanks!