Closed maholli closed 2 years ago
Hi @maholli thanks for starting this discussion - and your contributions elsewhere to the project!
I would be very excited to see pretty much everything you discussed here be implemented. Let's go through the points is some further detail:
The usefulness of an inventory management tool like InvenTree is directly correlated to its ease of use. Too tedious and folks wont use it.
100%
I'm aware of the TOS limitations of the various distributors (as discussed in #541 and #566), but there has got to be a better way. Take the digikey API for example, if the individual registers for a free API account, it appears they could use QR/barcode and Product retrieval API while adhering to the TOS.
I would like to see native integration (e.g. with Digikey) where all the user has to do is supply their API key (could be per-user or per-install).
What I would like to do is use my phone with the InvenTree app to scan bag barcodes.
I have long considered an integrated app to be a key feature for InvenTree - I have started working on this - https://github.com/inventree/inventree-app - but I have little (to no) experience working with Dart and also limited time.
I would love to see the app advanced. One of the key motivating factors would be to use a barcode scanner, with generated QR codes stuck on each inventory item (e.g. tubs / bags / etc).
These activities would take place in a "Parts Cache" tab in InvenTree
I really like this idea, I hadn't thought of having a "temporary working area". It could be quarantined to the current user, so e.g. the person in charge of receiving goods has a list of parts they still have to "deal with".
Part Actions
Everything here sounds really nice. It would probably require some fairly complex "business logic" on the back-end to make the user experience pleasant and simple. But, that's why we're here, right?
I have a fairly streamlined digikey API workflow in python for scanning barcodes, retrieving part/invoice/pricing info, and populating a spreadsheet (android).
Interesting - I think it sounds like there is a lot of functional overlap there!
This would allow laptops, desktops with webcams, and phones to all act as input sources.
Allowing the use of webcams would also assist with - https://github.com/inventree/InvenTree/issues/623 . I currently have no idea how to do this, it would be a super handy feature though!
Pulling together the thoughts above, there are a few key features I think would be amazing to have:
A framework for accepting images from a webcam / connected camera would be great. Useful for uploading part images, or scanning barcodes.
The integrated app would bring Inventree usability to everyone's pocket. Even just the ability to quickly check-out parts as you use them.
This is a huge area where usability can be improved. I really like the idea of "scanning in" parts (either via a manufacturer barcode, or some other method) and placing them into a "staging area" - a virtual storage location where the user has to decide what to do with the parts. This "staging area" can be synced between the phone app and desktop interface - the "staged" parts are in a special database table waiting to be sorted and assigned.
The manufacturers barcode (Digikey / Mouser / etc) can be used to automagically work out how each incoming part maps to the database. A generic "manufacturer" framework can be implemented, so a new manufacturer style can be accommodated as needed.
The ideas you enumerated above are really nice, and I'd love to see them integrated!
@SchrodingersGat I'm glad to see your enthusiasm! Thank you for taking the time to discuss the ideas.
I spent a moment today streamlining my barcode scanning method:
I will defer the SQL handling to those that are familiar with managing the flow of data into the db. An example of the resulting Digikey JSON I'm generating looks like this (some info redacted):
SEE UPDATE BELOW
Let me know if you're interested in playing with the webcam scanner / API query and I'll be more motivated to sanitize it and put it up on github. 🙂
It occurred to me that while we're going to all the trouble, we may as well query the product info for the scanned bag as well. Below is the full product info query for the scanned bag (some info redacted).
... "you agree that you will not, and will not assist or enable others to: a. cache, record, pre-fetch, or otherwise store any portion of the Mouser Electronics Content"
This is pretty darn limiting. ONE potential workaround is populating our InvenTree data from purely the contents encoded in the barcode. There appears to always be quantity, MFPN, and invoice number...
I went ahead and put my quickie webcam script up and then wrote a step-by-step for folks to get the example working with the new Digikey v3 API.
See https://github.com/maholli/barcode-scanner
@SchrodingersGat any progress making an InvenTree interface to accept these JSON responses?
@maholli that looks fantastic! Does the zxing javascript tool you mentioned above support this? It would be amazing to integrate this directly into the InvenTree web interface.
@SchrodingersGat any progress making an InvenTree interface to accept these JSON responses?
The JSON API is currently undergoing some heavy development mostly to support integration with an Android / iOS app
the app currently supports scanning "InvenTree" style QR codes which allow the user to jump directly to a particular stock-location or stock-item. For example, say you have a box which contains a whole bunch of capacitors, you print a barcode that represents that box as a StockLocation. Scanning that barcode in the app then displays the information for that StockLocation.
This means that you do not have to lug a computer around etc, you have stock lookup and management right in your hand.
There is obviously a lot of information available in the DigiKey / Mouser / LSCS / etc barcode. Where possible, these existing codes should be used to track StockItem objects rather than have to print out a separate barcode.
The information available could be mapped to the InvenTree data so that the digikey (or other manufacturer) barcode data is enough to uniquely represent a physical stock item. It also looks like there is other information available too, like Order ID. This could be used to mark items as "received" against a purchase order! The quantity is there too so partial-receiving could be supported.
The first step would be to work out a framework for generically handling manufacturer barcodes. I don't want to hard-code it for DigiKey and it doesn't work for Mouser, for example.
However we could have a generic "ManufacturerBarcode" class which then can be sub-classed for each manufacturer as the codes will need some custom management. Also some barcodes will have more data than others.
I would also like the data decoding to be handled on the server. This way the web interface and the app do not have to be "smart" and introspect the data, rather the server which has access to a lot more information works out what to do with the data.
The workflow would (basically) consist of the user-interface capturing the barcode data, and sending it to the server as a JSON object. Then the server works out if it can process it, and what to do with it.
Let's just focus on a DigiKey barcode.
Stock Scanning
Say you have a reel of capacitors and want to know how many parts are left in the reel. Just scan the DigiKey barcode, which will look up the StockItem in the database, and you'll see the current estimated quantity.
Incoming Goods
The barcode has order-id information, we could add a new field called "SupplierOrderID" or something to the PurchaseOrder, then an incoming item could be scanned and marked as "received" automagically
@SchrodingersGat holy cow, amazing progress! See my comments below for a few of the items you mention.
@maholli that looks fantastic! Does the zxing javascript tool you mentioned above support this? It would be amazing to integrate this directly into the InvenTree web interface.
Unfortunately it doesn't. In fact, there isn't a single open source barcode scanner with native browser support that reads data matrix (digikey & mouser) barcodes! How crazy.
I don't want to hard-code it for DigiKey and it doesn't work for Mouser, for example.
I touch on this briefly in my comment above, but at the moment Mouser doesn't even have a public barcode API as far as I can tell. Nevertheless, I think linking all the data to the raw barcode value is the way to go, that way it's unique (this is in line with your third party integration comment).
I would also like the data decoding to be handled on the server.
This is excellent! In that case, is it that troublesome to use the python barcode decoder already employed in my digikey example?
Stock Scanning
OMG yes
@SchrodingersGat what else can I do to help? If decoding on the server with Python is too cumbersome, I can try and port data matrix handling to zxing (for example).
I signed up for GitHub, finally, so that I could come here and offer the little bits of work I’d done to extend the InvenTree Python API, and to ask about when some other features might become available… only to find Maholli laying down my own thoughts but far more eloquently, and with more progress on his own solutions. And that SG has reworked - is reworking - the JSON API, rendering obsolete my potential contributions but all in order to add a phone app to do all the things I dreamed about doing with InvenTree… Wow, you guys. Wow.
I’d really like to pitch in here if I can. I love the direction this is taking. I haven’t worked with Django before and my Python is rudimentary, so I have some coming up to speed on the coding side to do. However, I wanted to also contribute thoughts about the business logic and some use cases, if I may be so presumptuous:
It seems that there are really three types or aspects of Part in the database: Part, Supplier Part, and Stock Item. I understand (#680) that the Supplier Part is being transitioned to a Child of a Part, although I’m not sure what the progress thereof is - I love this idea, by the way. Does that mean that Stock Items are now going to be Children of Supplier Parts, which are themselves Children of Parts? In any case, the impetus to add a Part to the database usually comes from one end or the other of this chain, and only when needed. In other words, we add a Part because we have a design that requires that Part’s function, and then we add Supplier Parts to that Part, buy some, and those become a Stock Item. The other way is when we HAVE parts, and we need to keep track of them - how many, where, etc. In that case, we need to add a Stock Item, which requires a Supplier Part, which requires a Part. As Maholli points out, for both the Supplier Part and the Part we only create a new Part if we can’t find one already in existence. Could we have Stock Items associated with a Part without a Supplier Part?
I mention the two ends of the chain where we want to inject Parts; my current focus is a pair of part inventories, one at my day job and one at home. The former has a lot of recent parts, most with barcodes and most existing in a CAD system's database somewhere. The latter is by far the larger, far more varied, and many of the parts are bulk parts sorted into bags with handwritten labels. I need to be able to add these parts - both inventories! - in a batch-wise manner to my InvenTree database(s) as quickly and effortlessly as possible, or better yet get my 14-year-old to do it. It seems that the Part Template construct has the potential to support this, but that it either is incomplete or works differently than I understand. Operating under the assumption that it is simply incomplete, I’m currently envisioning a workflow that leverages the Part Template and Part Parameter constructs to simplify this immensely; see what you think. Key to this scheme are the ideas (perhaps pending implementation) that Part Templates have Children that are also Part Templates, and that Part Templates inherit Part Parameters and Keywords from their Parents and pass them on to their Instances.
I believe inheritance of Keywords and Parameters to multiple generations of Parent/Child Template relationships is key. Further, I think the ability to assign default values or lists to Parameters and select or confirm those values at Part Instance creation is highly useful, especially for commodity-type parts like resistors, machine screws, 74-series logic, Zener diodes, etc.
Example: ‘Resistor’ part has Parameters ‘Resistance’’Ohms’ and ‘Power’’Watts’. New Parts can be created from this Template with any additional Keywords or Parameters desired, or a Child Template can be created with additional Keywords, Parameters, and / or Parameter Defaults. e.g. ‘Metal Film 1% 1/4 W’ as a Child Template of ‘Resistor’ would add those terms as Keywords, would default ‘Power’ to 0.25 and would assign the list of standard E92 values to ‘Resistance’ as defaults. Another value of ‘Resistance’ could be specified at the time of Stock Item Instantiation if not available in the list. Of course, the Resistance value selected would also be added to the Keyword list, formatted in engineering notation (1k02 Ω, 33R2 Ω, etc.) for improved search ability. Standard checks for existing Parts apply, of course. Categories would be assigned to each Part Template - not inherited from Parents - and would be assigned to Parts created from that Template. Is there room to marry the Part Template and Category concepts more closely?
Part Template ‘Screw’ has only one Parameter, ‘Length’’in’, and one Keyword, ‘Screw’. However, Child Part Template ‘Machine, #10-32, SS316L Philips Panhead’ adds those terms as Keywords. When we create an Instance (‘Variant’?) of this Child Part Template, it has all of those keyword and the length we chose.
I don't think InvenTree needs to add those categories, of course - I expect each house to implement their own part taxonomy.
Of course, the IPN is another can of worms. If I create a set of House Part#s in my CAD system, I want to be able to use those same numbers as my IPNs. In this case, I’m likely to be adding a large number of parts in a batch via spreadsheet or similar. If I’m adding a tub of donated, bagged, hand-labeled components, I want to (optionally) generate a series of IPNs for my new Parts that follow a system or pattern. I might, for example, assign an ‘IPN Prefix’ of ‘RMFWJ’ to indicate that it is a ‘Resistor Metal Film 1% 1/4 W’, and then append a field containing the Engineering-Notation-Encoded resistance value followed by a very short serial number, to have something like ‘RMFWJ-1.02k-00’ automagically suggested as the new IPN. If I later added a new but slightly different part with a tighter TempCo or something, I might use the same Part Template but increment the serialized suffix and add a ‘±10ppm’ Keyword. Or I might create a new Child Part Template ‘Metal Film 1% 10 ppm 1/4 W’ that adds the Keyword by default.
I have a 2D barcode scanner that I’ve been using to parse DK’s DataMatrix packages; again, I’m clearly way behind here. My ideas about workflow dictate that the addition of a new Stock Item to the database should have the optional ability to print a QR code and some human-readable text; I use a Dymo LabelWriter 450, and would very much like to add a QR-code label to each of these new Stock Items when I create them. Stock Items added programmatically from scans of DK barcodes, on the other hand, shouldn't need a QR code printed as they have sufficient uniqueness as is.
Which reminds me - that webcam DataMatrix scanner is flat-out amazing. Seems like it wouldn't be that enormous an extension of that code to grab all of the barcodes on a bag label at the same time, if you knew where the various fields were located. I've got bags from Mouser, Arrow, NAC Semi, and Newark that all have multiple 1D codes with the info we typically need. We could basically treat that set of barcodes as one large unique Stock Item ID.
OK, I've probably gone on long enough. I sincerely hope I haven't stepped in where I'm not able to be truly helpful. Please let me know where you would like assistance directed, up to and including “leave us alone you idiot”. And thank you, Maholli and especially Schrodinger's Gat, for all the work you've done on this.
@rco3 I agree entirely with your comments, especially the part templates and IPN structures. I've been wondering how to best implement a custom IPN scheme myself and have yet to land on anything I feel is sufficient.
Glad you like it! I actually spent some more time on it last night and have it working for all Mouser DigiKey DataMatrix and 1D barcodes! 🎉 Here's a quick screenshot to illustrate your point about grabbing all the barcodes at once. Working well 👍. I'll update the repo shortly.
Chugging along. Here's the current status of data extraction from the following suppliers:
[)>
>[)>
) and searched using Mouser's APICust PO
, Line Items
, Mouser P/N
, MFG P/N
, QTY
, COO
, and Invoice No
.productCode
, orderNo
, pickerNo
,pickTime
, and checkCode
. So far all the tool can do is search LCSC for the PN but the user needs to navigate the page and extract the info. Need to write a javascript web scraper. This barcode scanning tangent project can be found at maholli/barcode-scanner
@rco3 there's a lot of good ideas there but probably too many varying ideas to tackle in this thread. Are you able to break some of the key points out into separate issues and we can discuss there? I won't have the bandwidth to deal with them all immediately, but at least we can make a start on them :)
@SchrodingersGat what else can I do to help? If decoding on the server with Python is too cumbersome, I can try and port data matrix handling to zxing (for example).
@maholli my preferred approach would be for the "client" device (web browser / barcode scanner / phone app / etc) to scan the barcode data (i.e. do the image processing component) and then send the decoded JSON data to the server, which can introspect the data and work out what to do with it.
This would probably mean using zxing for the browser interface. I believe that the barcode library I'm using for the app is based on zxing too.
Does zxing not currently support the data matrix codes? If you are familiar with the format it would be fantastic to add that feature and then we can work on supporting Digikey (and other suppliers) natively!
I have been considering how to integrate these external supplier codes natively. I think that https://github.com/inventree/InvenTree/issues/680 should be implemented first, so that all "Parts" can be associated with a supplier / manufacturer.
A question for you at this stage - does the digikey barcode data provide a unique identifier for an individual package (e.g. a single reel). If so, then we could add a "UUID" field to the StockItem class which we simply populate with the digikey ID. That way, can do a UUID lookup when we scan the digikey barcode and then locate the associated StockItem.
If there's no purpose-built UUID field we could do a hash of some other barcode data (order code / part code / date) to make a reasonably unique identifier.
A question for you at this stage - does the digikey barcode data provide a unique identifier for an individual package (e.g. a single reel).
@SchrodingersGat as far as I can tell, the barcode itself will always be unique. The downside is for digikey a barcode can be upwards of 356 characters (utf-8 encoded). This will likely get unwieldy, so we could hash it or something.
Does zxing not currently support the data matrix codes?
I originally dismissed it since folks were having trouble (like this), but I tested another online demo and got it to successfully decode a digikey datamatrix! Wahoo! (note I had to enable try-hard mode).
Decoding the barcode is only a single line in my example python script. Everything else is taking the barcode data and then doing various checks to find the supplier and then making the appropriate API call. So this should still be very helpful for the server to run once it has the received the decoded barcode from the client.
SchrodingersGat,
I agree that it's a bit chaotic.
The Django webpage is... really cool. Adding one part is really easy. Adding a few parts is only moderately tedious. But I have large numbers of existing parts to add with: DigiKey Barcodes, Mouser/Arrow/etc barcodes, and NO barcodes., and I need to automate a lot of it. Hence the API.
I got to the point of adding Objects to the Python API for PartParameter and PartParameterTemplate, and a method for getting the Template for a given Parameter. But this recent work and discussion with you and Maholli may have had the effect of moving a lot of my process back to within the Django interface. Which I love.
So much of the point of the previous post was to suggest some ways to use Parameters, and ways to extend the current Template system which, like the Parameters system, appears to be a bit of a work in progress.
I shall enumerate more succinctly:
--Parameters are cool. I want to use them.
-- I like the idea of making SupplierParts Children of Parts. I wonder if that means that Stock Items are Children of Supplier Parts, or Children of Parts, or either?
--Part Templates should (my suggestion) carry Categories, Keywords, and Parameters which any new Parts created from that Template inherit.
-- Part Templates should be able to spawn Child Templates which inherit all of the Parent's Keywords and Parameters, add additional ones, and assign a new Category.
These are structural suggestions. I also would like to offer some process/interface suggestions:
-- Add Parts from Part Template: Pre-populate fields from the template wherever possible. After adding a part, offer to add another part with some fields (MFG, Vendor, maybe others) pre-populated from the previous part.
-- Offer various ways of pre-populating Parameter values: -- Value assigned by Template -- Value suggested by Template -- Selected from a list of allowable values -- Selected from a list of suggested values -- entered directly
--Lists of values could be added in a table.
--Some tables could be included:
E92 resistor values,
74-Series logic numbers,
E24 Capacitance values,
SAE and Metric screw threads, and so on.
-- Otherwise, lists could be uploaded via API or accumulated via web interface.
Maholli and I are both interested in IPN generation. It's a topic upon which I have ruminated long, pondering the philosophy of the Part Number. Smart? Or Dumb? Random/serial? Or encoded and meaningful?
At the end of the day, I think that any scheme encoding enough information into part codes small enough to be usable requires significant memorization by the user to truly have any advantage over a random number looked up from a table. Plus, the effort of developing and maintaining such a scheme could be significant overhead for a small organization.
With that said, I believe that there could be some low-hanging fruit to be gathered by developing a basic IPN generator with provisions for: -- a Prefix -- Fields populated from Parameters --- the List concept from above could be extended to have a Field Code associated with each Value. E.g., we could assign tolerance codes for resistors so that a Resistor whose Parameter is 1% (from a list) would have a Code of 'J' inserted into the automagical IPN, but a 0.5% would get a 'K', and so on.
I would be happy to separate some of these out into separate issues and post them, if you think there is merit. I fully recognize that there is a limit to your bandwidth.
I'm loving the direction InvenTree is going, and have been running on a local server, and updating occasionally to see if it's in a state I'm able to use it more efficiently than a spread sheet.
The usefulness of an inventory management tool like InvenTree is directly correlated to its ease of use. Too tedious and folks wont use it.
This is the reason I haven't been able to use InvenTree yet for recording my electronic components at home. There's a lot of steps involved in entering in the basic values of a component like a capacitor with adding its paramaters, vs keeping a spreadsheet. Each parameter for each part requires 2 clicks. Compared to a tab press on the keyboard in a spread sheet.
Once the data is in there it seems easy enough to make use of.
For me personally I think the ability to import parts with their parameters in a CSV import be a good start to get the existing data in. Kinda like /admin/part/part/import/
does, but allow for extra columns that match parameter names for that category.
Might also be nice to have tabular view like /part/category/X/parametric
but being able to edit the values all at once in that view. Can make for quick data entry and corrections. Even better if you can add a row in that view.
There's a lot of steps involved in entering in the basic values of a component like a capacitor with adding its paramaters
I shared a lot of your opinion on this topic and Ki-nTree, a tool I'm working on, is automating the entry of parameters based on those fetched from Digi-Key. It relies on the InvenTree API to populate them.
You can also do the same with a quick python script that would take your spreadsheet as input and set your parts and parameters the way you'd like.
@awulf I agree that a tabular-style data entry system would really speed up the process. Probably at this stage the real blocker for implementation is bandwidth.
A tabular-style entry would require a lot of front-end code, although the back-end API is pretty fully featured already! It would require some investigation on how best to implement an editable table in the browser.
I think a table-style data entry system would be great for:
(And probably other use cases too!)
@SchrodingersGat there's a couple of options I think I would consider maybe.
If you went with google spreadsheets, google would be doing most of the heavy lifting for you. The workflow would look something like this:
The second option is through an API.
@SchrodingersGat @rcludwick There is add-in with functionality like that in bootstrap-table: X-Editable. There are some issues with the implementation but it is a good starting point.
Ooooh that does look very fancy. Might make a lot of existing form-based editing redundant if we can use in-line editables
Just FYI this project is quite dead...
@eeintech you are quite right about that, I did not see that. Quick search found something quite similar that could be implemented with the formatters from bootstrap-table. It seems to be active (enough) and is written by a fullstack-dev that seems to keep everything simple and small. I will do a fast check on the weekend.
There is however no License, I opened a issue about that. And it is MIT!🙌
@maholli the function you asked for should be possible to implement easily with the new plugin functionality.
Closing out this issue, it grew into quite the long-winded discussion and there's little in the way of specific actionable items here.
I wanted to get this discussion started and hopefully avoid myself or anyone else having to reinvent the wheel here.
The usefulness of an inventory management tool like InvenTree is directly correlated to its ease of use. Too tedious and folks wont use it.
Hobbyist and Academic Makerspace Use-cases
I use InvenTree for my home inventory and am starting to integrate it into an academic makerspace. My most common activities with the tool are as follows:
Problem
These two activities (especially 1) are extremely tedious. As far as I can tell, the expected workflow is outlined below.
For adding a part:
I'm aware of the TOS limitations of the various distributors (as discussed in #541 and #566), but there has got to be a better way. Take the digikey API for example, if the individual registers for a free API account, it appears they could use QR/barcode and Product retrieval API while adhering to the TOS.
Envisioned Improvements
What I would like to do is use my phone with the InvenTree app to scan bag barcodes. These activities would take place in a "Parts Cache" tab in InvenTree. The "Parts Cache" would be concurrent between phone/desktop and therefore would allow the user to batch-scan items that can be curated on a desktop, or just scan a single item and act upon it.
Part Actions
Once scanned, the barcode data would be used to retrieve part information from the relevant distributor. The part info is then used to generate a line-item in the "Parts Cache." Next to this line-item would be quick actions that are dependent on whether or not this particular barcode has been entered into InvenTree or not:
Path Forward
I have a fairly streamlined digikey API workflow in python for scanning barcodes, retrieving part/invoice/pricing info, and populating a spreadsheet (android).
I could use help with the barcode scanning into the SQlite db ⭐ Since phone browsers can now easily access hardware like cameras...
Is the best approach is to implement barcode scanning from the browser?
This would allow laptops, desktops with webcams, and phones to all act as input sources.
Interested in hearing everyone's thoughts.