snipe / snipe-it

A free open source IT asset/license management system
https://snipeitapp.com
GNU Affero General Public License v3.0
10.74k stars 3.12k forks source link

Custom Model Attributes #98

Closed peelman closed 8 years ago

peelman commented 10 years ago

@snipe has mentioned this in another ticket, but I was hoping to get some discussion going on it and define the idea and storage model some.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

snipe commented 10 years ago

Every model (I think this would be at the model level, makes the most sense for my use-cases, but open to discussion - could maybe work on categories instead?) could have custom fields which would then be associated with all assets within that model. Almost like a form-builder. So the ASUS 23" Monitor model might not have any custom fields, but the MacBook Pro 13" Retina Late 2012 might have custom fields like:

Every time a new asset is created that belongs to that model, the add asset form would dynamically update (or there would be a page 2 for asset creation) that would pull those fields.

Fields can be marked as required per model, so an asset can't be created until those values are filled in.

The obvious, inelegant solution would be to dynamically add column names. This is how Tracmor does it, and it works well, despite making a little part of my soul die every time. The advantage to this method is that searching on these fields is pretty straightforward, whereas with more complicated join tables, searching can get to be a bit of a bitch.

snipe commented 10 years ago

Ack - hit submit too soon. Meant to add that I'm not sure which way I'd want to handle this, the normalized, join-heavy way or the de-normalized, inelegant way. Searching is definitely a concern - I might want to do a search to see all of the RTD pieces I have that have 16GB of RAM, for example. Obviously, those types of searches aren't impossible using properly normalized tables, but they definitely get more complicated.

peelman commented 10 years ago

From a growth perspective, the join method is the way to go, whatever it does to make searching difficult. Redmine's system has worked well for me in the past (database schema here), but I don't know what it would be like layering searching atop that.

Being able to add custom attributes at the model level works well enough for me, but if we can generalize that attribute system enough, there's no reason you couldn't make it available to individual assets/licenses/certificates as well. Allowing one-offs like that definitely opens up some headaches, but it also adds another layer of flexibility too. Not saying I am a particular advocate of that, but it should definitely be something to keep in mind when designing this, because re-applying custom attributes beyond hardware assets and licenses will be beneficial down the road I'm sure.

I'll add that when I am looking at this, I don't care about the machine specs as much as I care about the unique attributes:

snipe commented 10 years ago

Yes, as stated, there are advantages to both ways. The schema would be abstracted the same way the asset log is, so licenses, hardware, etc could use it. But searching is a very real (and I'd argue pretty common) use case, so it can't be ignored.

snipe commented 10 years ago

In my use cases, accessories wouldn't make sense for a use case here, as they would be covered by #16

peelman commented 10 years ago

We generally don't tag anything that doesn't have an (easily) visible serial number, so a lot of that would be happening for us.

At some point the question becomes when would we be better served by a true inheritance model (faux pax though gen-spec relationships are in the database world) and flesh out true classes for asset types, leaving a general purpose one for things that don't fit into the provided buckets. At this point I'm spitballing, but its an idea I've circled repeatedly since I first rebuilt SimpleAssets to suit my then-needs in 2008 (lots of working strategies in that project, but it was long dead even back then). A more abstract, expandable model is certainly a better goal on the face of it, but as you have pointed out, it makes searching (and many other facets) a hell of a lot harder to deal with down the road.

snipe commented 10 years ago

We don't typically tag things like keyboards, etc, but I'm considering starting, just so I have some way of knowing how many of x we have, who has x, etc. It's obviously less critical (Hmm... $30 keyboard versus $3k MBP...) but it's something of an annoyance for me. I never know if we have spare peripherals, etc.

One of those use cases is repair work as well. The purpose of #16 is partly to also be able to track total expense of a machine, repair record etc. One laptop has had 4 harddrive swap, etc. The hard-drive is currently in this laptop, but it may end up in another laptop at some point. I want to be able to track cost for those new component parts, and what's been swapped out when.

snipe commented 10 years ago

I should mention, we don't typically tag hard drives, even though they have a serial number.

peelman commented 10 years ago

All of that would fall to the notes/comments section for me (or at least it has in the past). It serves as a status log, work log, etc. for each asset. New hard drives, motherboards, etc., all would get documented inside the ticket. If a new drive was installed it's serial would get logged, as well as the outgoing part (and what happened to it: was it destroyed?, how?on what date? etc).

At one point I based a system entirely off of redmine (which I started building a true asset management plugin for) and it worked rather well; treating each classification as a project, and each asset as a ticket, using custom fields to define particular datasets. It definitely wasn't designed for it, and it showed, but when all you have is a hammer...

This is all theoretical at our current place, but I'm going off of how we handed it in my past positions, trying to roll the best of each clusterfuck we had and leave behind as many of the nasty parts as a I can.

To achieve the TCO you're going for in #16, I wonder if you don't need to do something that is more dedicated to expense tracking, starting with the base price of a machine and adding entries for every upgrade/repair. Something akin to:

Expense

I guess you could add the requisite fields to the notes table; but this just feels cleaner and easier to manage going forward...it doesn't negate the value of having the ability to define parent/child relationships ala #16, just creates more work :(

snipe commented 10 years ago

I'd prefer to use quantifiable data for repair parts, versus just making a note. (Bear in mind, this is all v2, not even v1, IMHO.) I don't typically need to track hours and I may or may not track cost on repair items, so I see it less as an expense tracking thing than component tracking. I know myself (and my techs) will consistently forget to use any kind of time tracking, so if it was added, it should be a setting you could disable in the settings page and would be added for the benefit of other people using this system, not for me. (And I'm totally open to adding things I don't personally need, as long as there are clear uses cases for other folks - but they may not be my first priority to add.)

What the custom fields are doesn't really matter, IMHO, but I feel strongly that they're needed - and they would be extensible enough that they could tie into #79 as well. For all of this (very useful) back and forth, we still haven't come up with any decisions on how to handle custom fields tho. :P

snipe commented 10 years ago

The base requirement in my head is that models can have custom fields. You can define what kind of custom field (checkbox, text, etc - very basic form builder type of functionality) and whether a value is required. Tracmor does this and it's been very helpful.

Then when you go to create a new asset of Model X, as soon as you pick Model X from the drop down list, the new fields are created in that form. As I mentioned, Tracmor does this by creating columns on the fly, but that makes me feel oogie.

Knowing that search is a priority, we could also build out some kind of search table that basically has some kind of key-value store. It's janky, but it could simplify queries quite a bit. Basically a search index table that lets us keep the data normalization we so desperately crave, without making those search queries turn into something truly unholy (and slow).

peelman commented 10 years ago
custom_attribute
custom_attribute_type
custom_attribute_value

If you wanted to allow Radio buttons it could be done through simple JSON parsing via a "values" column as well. The separate type lookup table could be eliminated (probably), and there are varying ways to handle constraints and validation.

There will probably need to be a weight or order value associated with it, to determine where in the Asset form it shows up.

At this point I'm leaning towards the Asset form being two parts, and having all custom entry done on the second page; from a lazy developer's perspective, that seems easiest.

peelman commented 10 years ago

One thing that does occur to me, do we / how do we want to handle adding custom attributes to licenses?

snipe commented 10 years ago

Re: your last note - we certainly could add it for licenses, I'm just not sure that it's needed. More importantly, I'm going to want this when the services/vendor section of the system is built out, so it makes sense to think of it beyond just assets. Basically, everything in the system is some sort of property - sometimes they live in different tables (assets, licenses) because there are business rules around all of it, but all of the goodies should be designed to extend to them. (One thing I've been pondering, as this thing progresses, is whether to pull licenses back into assets, and make all of they details (warranty, etc) as custom fields instead.)

Re: the pref note - I don't thin weighting is necessary. The custom fields appear in the order they were created. We could certainly add custom ordering over time if it were requested, but have never had a need for it and until I hear a compelling use case from someone who does, I don't see a reason to add it.

Your proposed schema makes sense to me, and was similar to what I had in mind. (I wasn't going to give validation beyond required/not required, but with the built-in validators in Laravel, it seems silly not to, as long as we can make the UX work.

Speaking of UX, I think I want to keep the asset creation to one page. We could do a wizard, but its just as easy with query to dynamically populate new fields based on what's selected from the models menu. Better (faster) UX overall, IMHO.

uberbrady commented 10 years ago

Hi guys - I was pondering this a little myself. I'm going to stay away from the nitty-gritty of database schemas and whatnot and instead talk through a use-case I imagined:

So I imagined that things like RAM, num CPU's, CPU speed, etc - would all be 'custom attributes'. And - just as @snipe mentioned - I could easily see wanting to search across different models to find "any computer with greater than 8GB RAM" - and have that spit out MacBook Pro Retina 15, Retina 13, Macbook Pro 13 - and whatever else I have on-hand with those specs.

So I was thinking about how to attach those extra attributes to the model and it struck me as repetitive and mistake-prone to have you have to go through and add each attribute to each model each time. Neither should you put it at the Manufacturer level - as Apple makes all kinds of things other than Macs and it might not be appropriate.

I was thinking you would have something like "asset types" or "custom attribute sets" or something like that - and when you're editing/creating your Model (e.g. Macbook Pro Retina 13), you can pick your "custom attribute set." (Or maybe just Asset Type, that could be simpler).

That way I don't have to manually add: RAM, integer, required, CPU, float, required, HD, float, required... et al to each model I end up specifying.

I can instead just pick from a drop-down (or something like that) and say "Asset Type: Computer" (or "Custom Attribute Set: Computer").

And so I guess I lied about database schema - I would modify custom_attribute_value instead to be like:

asset_id attribute_id value

with a unique index over (asset_id,attribute_id) (unless you guys can imagine a use case where you repeat an field a few times in an asset? BLECH) - and perhaps a covering index (might as well make it unique) over (asset_id,attribute_id, value) <- you can query from that and get a list of assets without ever even touching the custom_attribute_value table. I would lose the 'id' column since I don't think it adds anything. But @snipe will just shoot that idea down anyways so I don't know why I even propose it :) (she likes id's)

snipe commented 10 years ago

In my head, it would work tied to the category. So, category is laptops - every laptop has X, Y and Z attributes. Some laptops may have A and B attributes too. When you create the MODEL within the laptop category, the laptop attributes are provided for you, and you can then choose to include them with that model or not via checkboxes next to the custom fields you configured in the category. The default will be to include them, and we could hide all of that under an expandy "advanced options" section for simplified UI, but the category serves no real purpose otherwise. We can think of categories a "custom attribute collections" - but we won't call them that in the UI because no one will know WTF that means.

So models inherit custom fields from their category, and assets inherit custom fields from their models. This leaves it up to the user to organize their categories in a way that works for them, and prevents me from imposing my business rules on someone else's use cases.

snipe commented 10 years ago

To further illustrate (and I am unfortunately too lazy to mock something up in HTML):

CATEGORIES:

In CATEGORY: computers, I might create the following custom fields:

Then when I create a new MODEL - let's say a MBP 13" Retina, Late 2012, in that interface for creating that model, I have the same fields that currently exist (asset tag, asset name, etc) and then a collapsed section called "Advanced Options" or "Custom Fields". All custom fields for that category will be enabled by default, but you can easily disable them if you'd like the interface for each asset to be a little more compact (this is something I would do, for sure.)

snipe commented 10 years ago

The thought here is that if people create categories in a sane, structured way (which we can help with by seeding the db with some sane defaults), they can extend it out as needed - and we don't create models nearly as often as we create assets, so defining the assets at the category level, opting out of unnecessary ones at the model level, and then implementing them at the asset level seems like it corresponds well with the frequency of use for each of these levels of hierarchy.

sigwo commented 10 years ago

In my workplace, I need the ability to track multiple CATEGORIES with MAKES and several MODELS, LOCATIONS, and SERIAL NUMBERS/ASSET TAGS. As long as I can add/subtract fields for what I need, I would be happy. Example:

Network (Category) Switch (Sub Category) - User created Cisco (Make) 48 port (Custom Attribute) - User created 3750X (Model) Serial Number Location

I would like to see is a QR code/barcode generator to scan equipment for check out, scheduled inventories, accountability showdowns, etc. Enter information then click "Generate" to create/save/print code.

snipe commented 10 years ago

Hi @sigwo - barcodes are already on the planned feature list in #29.

I wonder if you couldn't accomplish the same thing with the structure we're talking about:

CATEGORY: Network: Switch

MODELS:

peelman commented 10 years ago

So what bothers me about a lot of the discussion so far is that you are going to have a lot of categories and/or models that are essentially identical save for one or two distinguishing attributes. And unless we do a many-to-many and create a complicated custom attributes manager, you are going to have a lot of repeated custom attributes to handle all these common cases.

snipe commented 10 years ago

@peelman not sure I agree. Categories' basic purpose in this scenario is to be a collection of custom attributes. Can you give me use cases that support this notion?

snipe commented 10 years ago

In @sigwo's example:

CATEGORY: Network: SUB-CATEGORY: Switch

MODELS:

Cisco 3750X Cisco 3751X Cisco 3752X etc

So basically all switch models would have the same set of attributes available to them, since Switch is the (sub) category.

sigwo commented 10 years ago

@snipe Apologies for missing #29.

@peelman If I select Network -> Switch -> 3750X, I've already selected > 80% of my needed fields for completion. Basically, I would only need to fill in the Asset Tag/Serial Number and location. After the initial CATEGORY/SUB-CATEGORY setup, adding additional like-assets would speed up.

I'm not sure if you will be able to create the fields we want without creating a lot of fields we won't use.

snipe commented 10 years ago

@sigwo HOW DARE YOU NOT KNOW EVERY OPEN GH ISSUE ON THIS PROJECT BY HEART! :tongue:

It's totally fine - just wanted to let you know that that that's in planning and is definitely part of what I consider the v1 MVP. In fact, I may ping you in that thread to clarify your use cases there. I don't need them, so your input would be helpful.

tobint commented 10 years ago

I'm extremely exhausted at the moment, so if I'm thinking too general here

Every time I develop a system that is meant to be deployed by multiple unknown parties, the idea of customization/extensibility comes up. Inevitably, this same discussion comes up every time. The conclusion I always come to is: you can't be all things to all people and still have the best product for those customers. You really do have to make some choices that will limit your potential adopters or you will end up with a system so abstract that is either too complex or too slow for anyone to use.

Who's hero do you want to be? So before I can weigh in, I have to ask about your target audience. For instance, what size businesses? How many assets? How much customization do you want them to be able to make through administrative interface?

Understanding your target audience helps strike a balance between administrative configuration and module-based extensibility. Obviously, administrative configuration will lead you down the paths you've discussed above (and I'm happy to rehash the plusses minuses you've already pointed out), Module-based extensibility allows for ultimate optimization around data-type specifics. You can also create various hybrids of this. For instance, you can create a module-based system for extensibility and create some basic types up front that people can choose from and even extend through a certain number of fields -- leaving larger implementations and highly varying data types to plug in through modularity.

@snipe, you mentioned categories, and I think that's great. I wouldn't hard-code those decisions around those categories though. I'd use those as template starting points. In other words, if someone picks "laptop", they can then extend that with more fields. The problem is that no matter what you do, you're going to end up with too much abstraction to make the system cost-effective and fast. Think about this at even a field level and it can get extremely complex. Suppose you define data types such as a MAC address. You could then let the system know what kind of input mask to accept, client and server-side data validation, localized error message handling, data storage type, etc. You then put a bunch of these together and tell the system to both write to and query from this system on a regular basis for an asset type. It can get very slow if assets are swapping around a ton. More abstraction == more CPU/storage cost.

Sorry, I'm rambling, but my point is that if you define who your v1 customer is and the type of assets they want, then you can stay focused for now on those specifics.

For v2, you can create a more module-driven system with some built in module types (laptop, phone, switch, etc) and some "generic" modules that allow for configuration-based asset types. These types will be slower, but for smaller businesses without the dev resources to create a module, this shouldn't be a problem as they'll have less assets to track anyway. For those with more specific needs, building a module that allows them to search custom fields, validate data, provide custom error messages, etc -- they can build their own module for that.

Just my two cents for now. Sorry if this wasn't helpful. Will go to sleep and review how much damage I've done later. :)

snipe commented 10 years ago

Thanks for the lecture on lean development, @tobint ;)

I don't disagree with anything you've said here - and I'm absolutely NOT trying to be all things to all people. I will not compromise my vision of what I want this to be to be everything to everyone, or even if it means the UX becomes so confusing and shitty that it's no better than every other crappy FOSS (and paid) tool out there.

That said, that's not a reason to think through extensibility. There will be no modules or anything like that, and there's not going to be a lot that's truly customizable through admin settings - this is simply a data architecture discussion.

Folks like Tracmor managed to pull off what I'm talking about here, and their code is shit, so there's no reason why we can't pull it off here either. The decision isn't whether to include custom fields - they ARE being included - the question is how to implement them in a way that makes the most sense based on the relationships of the fields to the data. While I'm a big fan of data normalization, I also have few qualms about de-norming for performance purposes, searchability, etc. This is more of a question of which property the fields get associated with. At this point, I think category still makes the most sense to me - and by changing the category, the user can extend it out as needed, based on the notion that a "category" is simply a collection of custom fields.

So I totally get it - and am just telling you not to worry so much :) While ambitious, I'm also exceptionally lazy and have no intention of over-engineering anything. Just giving some consideration right now. And we may get it all wrong and have to change it later. That's okay too :)

tobint commented 10 years ago

Cool. Wasn't trying to lecture on lean dev. I know you get it. I felt actually silly at one point and deleted part of my post because I remembered who I was talking to. I'm so used to talking to those who have no idea how to break down the problem, let alone go from zero to MVP in a very short time.

snipe commented 10 years ago

Nothing but love for you, @tobint - I know you meant it absolutely the right way - and I can get caught up in the details and get distracted from MVP, so it's a sound reminder. The difference in this project for me is that this MVP isn't for a company, it's for ME personally to make my life easier, so I'm hyper-focused :)

peelman commented 10 years ago

I had family to entertain last night for a belated thanksgiving, so I missed out on a lively discussion it seems :smile:

My concern about duplicate custom attributes was under the assumption that custom attributes would be applied at the model level. As it stands, Categories seems like a second class citizen, a lookup table rather than an inheritance source. So if things can be applied at that level, that changes the ballgame somewhat. And @sigwo's example of subcategories is a completely new thing that hasn't been mentioned before (in anything I've read, at least).

So basically, based on what I've read, and how I interpret it, I want to say that while I see the value in a custom attributes system, building the entire foundation upon a collection of custom attributes and simply populating a default set seems like a terribly complicated way to go about things.

snipe commented 10 years ago

Subcategories already exist (or they did. and then I removed them because they had no purpose at the time). They're easily added back in.

"I want to say that while I see the value in a custom attributes system, building the entire foundation upon a collection of custom attributes and simply populating a default set seems like a terribly complicated way to go about things."

I don't actually understand what you mean here. We'd be populating a default set that people could remove or change, just to make it easier on folks.

peelman commented 10 years ago

I think it parallels what @tobint said, about picking "the right" set of core attributes to track and leaving the custom stuff to the edge cases. otherwise your search algorithm is going to immediately be complicated.

snipe commented 10 years ago

The custom fields are a core attribute for me, though. And as I said, Tracmor does it and their code is awful. For v1, maybe you can't search on them, but they need to exist - and there are ways around the search/join challenge, like a search index table.

mattparkes commented 9 years ago

+1. We'd love to see Custom Fields, even if they aren't initially searchable or the searching is quite naive.

pratikpul commented 9 years ago

The requirement was added under consideration atleast 6 months ago, we would love to get this feature. I am new to Snipe IT and is preety amazing but the first thing I noticed is missing Model attributes. Can we have any updates on when we will get this?

Thanks!

cobrien165 commented 9 years ago

I would certainly like to see this as well in a future release. Even a simple notes field under each asset model would work for my scenario. Along with most people I'd imagine, we purchase most models in bulk that will all have the same specs as far as CPU, RAM, Hard Drive, etc. Then we would detail any upgrades under the notes field of the specific asset.

Some other fields to maybe think about would be FRU part numbers for spare laptop parts ordering such as replacement keyboards, bezels, screens; but again, could be added into a notes field for now.

ari commented 9 years ago

Is this feature still on the cards? It is the only reason we can't adopt your very nice system for our own use. I'd be very pleased with a very simple implementation:

custom_attribute

id
asset_category_id (FK)
type (enum: string, number, password)
allow_multiple
name

custom_attribute_value

id
custom_attribute_id
asset_id
value (string)

No searching. No validation (other than the type).

For us at least, that would be great.

snipe commented 9 years ago

Still on the table. Again, the tricky part is searchability, and even if you don't need it, other folks definitely will. :-/ Hoping to get it handled for the next big release.

ari commented 9 years ago

Cool. I'm not clear why searching is hard. Without looking at your schema for the details, it is something like:

 select id from asset join asset.id = custom_attribute_value.asset_id where custom_attribute_value.value = "something" and custom_attribute_id = 47

where 47 is the attribute id determined from the UI. There would be a drop down menu for choosing which attribute to search on and a field next to that to type in a value. Obviously there is more work needed to construct multi-line searches and allow AND/OR combinations, but a single attribute is trivial.

snipe commented 9 years ago

I think you just answered your own question. It's the and/or combos that are problematic if we want to include an "all things" search (which we do). As I said, hoping to get it handled in the next big release. It's mostly only me developing this, and I run a startup in addition. It's not really about the technical challenge, just about the time.

ari commented 9 years ago

No worries. I know that open source isn't something you can always devote a lot of time to. I had a quick dig into your code, but PHP isn't my strength. I'm guessing you need to edit the search in the controller by adding

->join(‘attributes’, ‘custom_attribute_value.asset_id’, ‘=’, ‘asset.id’)

Anyhow, perhaps I'm telling you how to suck eggs, so I'll leave you to it. Thanks again for a neat app.

snipe commented 9 years ago

With all due respect, I've stated several times that this is not a technological challenge but a time challenge. I don't actually need you to explain my code to me, especially when you've admitted that this language isn't your strength, and you haven't looked at the schema.

While I understand that this feature is important to you, so are many other features to many other users, so unless you have a pull request ready, stop telling me how to manage my software. I've told you I'm hoping to have it in the next big release. No matter how many times or ways you speculate on how this problem could be solved (with no knowledge of the schema, no notion of the project's priorities, and no strength in the language), that's not going to change.

You can either be patient, you can provide a pull request, or you can find another "neat" app that solves your problem.

ESWBitto commented 9 years ago

@snipe, Thank you for making an awesome open source solution!

Anywho, I wanted to add to this conversation on custom fields. I've read the entire discussion and I see now where your going with it. One thing that I've noticed with poking around testing out the UX is when creating an asset. If you don't have the model or the supplier already configured. You have to stop what your doing and then add them and then come back to creating the asset again.

How much trouble would it be to branch into creating a new model or supplier (or custom field) while still trying to add your asset?

I think you mentioned something like a wizard? Would love to hear your $0.02 on it.

snipe commented 9 years ago

@ESWBitto - the ability to add a model while creating an asset is already on deck, though not in this ticket. :)

uberbrady commented 9 years ago

I'm going to start messing with this. I might end up painting myself into a corner or something, but I'd like to at least make a stab at it.

I am planning on doing it this way:

And finally, the scary bit: I think I want to do them as additional fields, with a particular name-scheme. E.g., if you create a new field within one of your fieldtypes named "case color", we actually do an sql ALTER TABLE Assets ADD COLUMN (dynamic_case_color TEXT) or maybe something like "dynamic_17" for the 17th custom field (that way it's not name-specific). Maybe I'll do the former, and just not let you rename the field.

@snipe - any thoughts? Any of this you need to veto, or want changes for?

snipe commented 9 years ago

No, that's all about the way I was planning on handling it as well. We'll also need to dynamically add those fields to the listing JSON output and the "show fields" DataTables javascript.

snipe commented 9 years ago

(Or somehow handle that intelligently, models may have different custom fields, and so it will be challenging to normalize that into a table view when viewing all assets)

uberbrady commented 9 years ago

My thought for that one is simply "don't" - but maybe that's too simplistic.

Maybe, just to get a version of this out, I'll start with only showing them on an individual item's page. A later revision can perhaps include them in the columnar list-view.

snipe commented 9 years ago

@uberbrady - No, I get that, but we also want to build this in such a way where getting to that point won't be completely horrific.

I'm going to want some unit tests on this, I think, since a bug in the "delete custom fields" functions could be destructive to other data.

I really hate handling this via alter table, but the only other alternative is a custom fields table which will put us into even more join hell.

snipe commented 9 years ago

Or maybe we just don't handle that on the listing page, but instead show that on the model's asset listing page: http://demo.snipeitapp.com/hardware/models/1/view