[Feature] update existing parts

[si-hb]

First off, hats off to the brilliant effort here - thank you!

I am new to Inventree, needed to inventory years of old stock which has now been added (by way of some python btw, I couldn't get the Inventree csv import tools to work for the life of me). Alas, it is all void of datasheet links, parameters, images et al.

Maybe this already exists and I'm not aware of it, but it sure would be handy to have a tool I could run against my existing Inventree DB of bare naked parts, and reach out to the clouds for all the dressings...

for example: $ inventree-part-import update Electronics/Passives would update that subcategory.

I tried to export my parts to a csv file, which I then tried $ inventree-part-import parts.csv. But it only got past a coupla parts before it would trigger #4 and/or #3.

Thanks again.

[30350n]

This seems useful. Quickly implemented it, you can update categories via --update CATEGORY_PATH and --update-recursive CATEGORY_PATH now!

Edit:

First off, hats off to the brilliant effort here - thank you!

Thanks btw!

[si-hb]

Brilliant!!!!!!!!!!!!!!!!!! Works like a charm.

It does kick out on a certain part however. Is this an issue with my DB? If so if you can catch/skip the error and let it continue, would be wicked, then I can deal with that specific part on my own.

requests.exceptions.HTTPError: 

{
  "detail": "Error occurred during API request",
  "url": "<my api...>/company/part/1004/",
  "method": "PATCH",
  "status_code": 400,
  "body": "{\"manufacturer_part\":[\"Linked manufacturer part must reference the same base part\"]}",
  "headers": {
    "AUTHORIZATION": "Token <token>"
  },
  "params": {
    "format": "json"
  },
  "data": {
    "manufacturer_part": 27,
    "description": "TRIAC 600V 4A TO220F",
    "link": "https://www.digikey.ca/en/products/detail/ween-semiconductors/BT136X-600-127/1155132",
    "packaging": "Tube",
    "available": 1126
  }
}

_For anyone else reading this, $ pipx run --spec git+https://github.com/30350n/inventree_part_import.git inventree-part-import --update-recursive <category> allowed me to run this patch._

[30350n]

It does kick out on a certain part however. Is this an issue with my DB? If so if you can catch/skip the error and let it continue, would be wicked, then I can deal with that specific part on my own.

Do you have a full traceback?

[si-hb]

Yes...

Traceback (most recent call last):
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/bin/inventree-part-import", line 8, in <module>
    sys.exit(inventree_part_import())
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/cli.py", line 38, in wrapper
    raise e
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/cli.py", line 22, in wrapper
    func(*args, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/cli.py", line 199, in inventree_part_import
    last_import_result = importer.import_part(part, supplier, only_supplier)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 77, in import_part
    import_result |= self.import_supplier_part(supplier, api_part)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 172, in import_supplier_part
    update_object_data(
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/inventree_helpers.py", line 77, in update_object_data
    obj.save(data)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree/base.py", line 242, in save
    response = self._api.patch(self._url, data, files=files)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree/api.py", line 457, in patch
    response = self.request(
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/retries.py", line 68, in request
    return super().request(api_url, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree/api.py", line 371, in request
    raise requests.exceptions.HTTPError(detail)

[30350n]

_For anyone else reading this, $ pipx run --spec git+https://github.com/30350n/inventree_part_import.git inventree-part-import --update-recursive <category> allowed me to run this patch._

You can also just install via pipx install git+https://github.com/30350n/inventree_part_import.git, although you might need to uninstall the pypi version first.

[30350n]

Try again (with the master branch). I'm handling all HTTPErrors now, so this doesn't fix the actual error, but it shouldn't crash anymore.

[30350n]

Actually I think I also know what the actual problem is. The updating I implemented is somewhat stupid. It just dumps all the part names from the category and imports them. This doesn't ensure the parts actually get updated though. In some cases it might keep the old parts and create new duplicate ones. So maybe wait a bit more until updating everything, I should be able to fix that fairly easily.

[si-hb]

OK, some more insight, an error from my DB:

2024-01-09 20:19:51.192 UTC [84594] ERROR: insert or update on table "part_supplierpart" violates foreign key constraint "part_supplierpart_part_id_d20a9b93_fk_part_part_id"

ChatGPT's take on the error:

Non-Existent Reference: You are trying to insert or update a row in part_supplierpart with a part_id that does not exist in the part table. The foreign key constraint ensures that every part_id in part_supplierpart must correspond to an existing part_id in the part table.

Deleting or Altering a Referenced Row: If you're performing an update, it's possible that you're trying to alter or delete a part_id in the part table that is still being referenced by a row in part_supplierpart.

To resolve this error, you will need to:

Ensure that any part_id you're trying to insert into or update in the part_supplierpart table exists in the part table. If you're updating the part table, make sure you're not altering or deleting any part_id values that are still being referenced in the part_supplierpart table.

[30350n]

ChatGPT's take on the error: \<bunch of gibberish>

Sorry, but that's really not helpful.

[si-hb]

ChatGPT's take on the error:

Sorry, but that's really not helpful.

No worries, fortunately it handles critique rather unflinchingly lol.

[30350n]

No worries, fortunately it handles critique rather unflinchingly lol.

I mean 'ChatGPT'/"""AI""" in general 🙂

Okay, with 5e89920 now, it should actually update the existing parts and that specific error you got should also not happen anymore 🤞

[si-hb]

Closer! It cut out at this part with:

searching for 1803277 ...
successfully added Mouser part 651-1803277 (https://<link>/supplier-part/1028/)
found existing DigiKey part 277-1206-ND ...
error: failed to import part with: 
    link: ['Ensure this field has no more than 200 characters.']

the following parts failed to import:
100-44-316-41-001000
100-93-308-41-001000
1803277

the following parts are incomplete:
103
1803280
1803578
etc...

[30350n]

Can you run that part again with -v to get the full traceback? (I'm mainly curious why that digikey link is >200 chars long).

[30350n]

Btw, it's generally a good idea to use DigiKey as the first supplier to check. You can reorder the suppliers in the suppliers.yaml in your config directory.

[si-hb]

Can you run that part again with -v to get the full traceback? (I'm mainly curious why that digikey link is >200 chars long).

209 characters long to be precise...

https://www.phoenixcontact.com/product/pdf/MTgwMzI3Nw?_realm=us&_locale=en-US&blocks=commercial-data%2Ctechnical-data%2Cdrawings%2Capprovals%2Cclassifications%2Cenvironmental-compliance-data%2Call-accessories

[si-hb]

Btw, it's generally a good idea to use DigiKey as the first supplier to check. You can reorder the suppliers in the suppliers.yaml in your config directory.

Good to know! Thnx

[30350n]

209 characters long to be precise...

and dead LMAO.

[si-hb]

209 characters long to be precise...

and dead LMAO.

LOL!, hmm... possible to skip adding link if it's dead?

[si-hb]

Can you run that part again with -v to get the full traceback? (I'm mainly curious why that digikey link is >200 chars long).

searching for 1803277 ...
searching at Mouser ...
found existing Mouser part 651-1803277 ...
successfully updated Mouser part 651-1803277 (https://<url>/supplier-part/1028/)
searching at DigiKey ...
found existing DigiKey part 277-1206-ND ...
error: failed to import part with: 
    link: ['Ensure this field has no more than 200 characters.']

FULL TRACEBACK:
Traceback (most recent call last):
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 87, in import_part
    import_result |= self.import_supplier_part(supplier, api_part, existing_part)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 183, in import_supplier_part
    part.addLinkAttachment(datasheet_url_safe, comment="datasheet")
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/base.py", line 501, in addLinkAttachment
    return AttachmentSubClass.add_link(
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/base.py", line 401, in add_link
    if response := api.post(cls.URL, data):
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/api.py", line 419, in post
    response = self.request(
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/retries.py", line 68, in request
    return super().request(api_url, **kwargs)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/api.py", line 371, in request
    raise requests.exceptions.HTTPError(detail)
requests.exceptions.HTTPError: {'detail': 'Error occurred during API request', 'url': 'https://<url>/api/part/attachment/', 'method': 'POST', 'status_code': 400, 'body': '{"link":["Ensure this field has no more than 200 characters."]}', 'headers': {'AUTHORIZATION': '<token>'}, 'params': {'format': 'json'}, 'data': {'part': 1123, 'comment': 'datasheet', 'link': 'https://www.phoenixcontact.com/product/pdf/MTgwMzI3Nw%3F_realm%3Dus%26_locale%3Den-US%26blocks%3Dcommercial-data%252Ctechnical-data%252Cdrawings%252Capprovals%252Cclassifications%252Cenvironmental-compliance-data%252Call-accessories'}}

the following parts failed to import:
1803277

[30350n]

It's not dead dead, the page loads. Just limited the link to 200 chars now, should be fixed.

[si-hb]

It's not dead dead, the page loads. Just limited the link to 200 chars now, should be fixed.

still getting same error for part# 1803277

$ inventree-part-import -v 1803277

[30350n]

Ahhhh, it's a datasheet link. Gimme a sec.

[30350n]

Okay, now it should be fixed.

[si-hb]

Okay, now it should be fixed.

Running --update, now past 1803277 and onwards... stay tuned.

side note, when it says warning: found 3 parts at Mouser, skipping import, what's the reasoning behind say not selecting just the first part? Perhaps could be a config setting..?

[30350n]

side note, when it says warning: found 3 parts at Mouser, skipping import, what's the reasoning behind say not selecting just the first part? Perhaps could be a config setting..?

If you are running with --interactive twice (which is default) it'll ask you what part you want to import after it's done going through all parts once.

If you run with --interactive true it'll ask you right away.

Just picking the first part is not really a good idea in many cases.

[si-hb]

made it through about 100 parts, and stopped on this part:

$ inventree-part-import -v PBC36DFAN

searching for PBC36DFAN ...
searching at Mouser ...
(no results at Mouser)
searching at DigiKey ...
found existing DigiKey part S2211E-36-ND ...
updating DigiKey part ...
error: failed to import part with: 
    manufacturer_part: ['Linked manufacturer part must reference the same base part']

FULL TRACEBACK:
Traceback (most recent call last):
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 87, in import_part
    import_result |= self.import_supplier_part(supplier, api_part, existing_part)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 198, in import_supplier_part
    update_object_data(
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/inventree_helpers.py", line 77, in update_object_data
    obj.save(data)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/base.py", line 242, in save
    response = self._api.patch(self._url, data, files=files)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/api.py", line 457, in patch
    response = self.request(
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/retries.py", line 68, in request
    return super().request(api_url, **kwargs)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/api.py", line 371, in request
    raise requests.exceptions.HTTPError(detail)
requests.exceptions.HTTPError: {'detail': 'Error occurred during API request', 'url': 'https://<url>/api/company/part/902/', 'method': 'PATCH', 'status_code': 400, 'body': '{"manufacturer_part":["Linked manufacturer part must reference the same base part"]}', 'headers': {'AUTHORIZATION': '<token>'}, 'params': {'format': 'json'}, 'data': {'manufacturer_part': 110, 'description': 'CONN HEADER VERT 72POS 2.54MM', 'link': 'https://www.digikey.ca/en/products/detail/sullins-connector-solutions/PBC36DFAN/861346', 'packaging': 'Bulk', 'available': 1510.0}}

the following parts failed to import:
PBC36DFAN

[30350n]

Thought I'd have already fixed that one. Guess I missed another edge case, should actually be fixed now 🤞

[si-hb]

Thought I'd have already fixed that one. Guess I missed another edge case, should actually be fixed now 🤞

Same error.

[30350n]

Okay, going to need some more information here probably:

1. What's the PK of the PBC36DFAN Part? (should be part of the URL)
2. Is that exact same part set as the base part on the S2211E-36-ND SupplierPart?
3. There should be a PBC36DFAN ManufacturerPart aswell. Is the same part also set as it's base part?

[30350n]

Actually nvm, I might have got it, gimme a sec again ^^

[si-hb]

Actually nvm, I might have got it, gimme a sec again ^^

ok, there's actually three, 1095,1096,1097. Same manufacturer part, different part variations though and different Digikey parts because it's a pin header.

[si-hb]

I can give you a token and url if you want.

[30350n]

Ah I see, do you actually need all 3? 🤔

[30350n]

I can give you a token and url if you want.

Thanks for the offer, let's try it one more time now first, if it doesn't work now, something very weird is going on 😅

[si-hb]

Ah I see, do you actually need all 3? 🤔

Hmmm. I suppose not, now that you mention it. However, the different variations are ideal.

[si-hb]

I can give you a token and url if you want.

Thanks for the offer, let's try it one more time now first, if it doesn't work now, something very weird is going on 😅

Worked!! I'll run it against the whole DB again...

[30350n]

Perfect! Thanks for helping to work through all of this btw, your database seems to be a great "integration testsuite" ^^

[si-hb]

Perfect! Thanks for helping to work through all of this btw, your database seems to be a great "integration testsuite" ^^

You are most welcome. I can't tell you how much I appreciate your work and effort and responsiveness! Happy to test anything else you might need.

Yes my great "integration testsuite" is what you get when you give a noob a youtube video and access to a computer...

[si-hb]

Successfully made it through all parts in my DB :1st_place_medal:

[30350n]

Wohoo! Great, I guess I can close this then ^^

Thanks again and if you run into any more issues, be sure to let me know!