Feature Request/Proposal: custom types for specific fields

[bradleypeabody]

Still getting familiar with SQLBoiler but I don't see a way to specify a custom type for a particular field.

The situation I'm running into is there are various database fields for which I've already implemented custom encoding logic for and I would like to use this approach with SQLBoiler. Example:

type User struct {
    UserID string `db:"user_id"`
    UserName string `db:"user_name"`
    // ...
    Settings UserSettings `json:"settings"`
}

type UserSettings struct {
    ShowDashboard bool  `json:"show_dashboard,omitempty"`
    NotificationsEnabled bool `json:"notifications_enabled,omitempty"`
    // whatever else - fields can be added here without changing schema, all of
    // this is encoded into a string on the Settings field on User
}

// these implement the database driver encoding using json.Marshal/json.Unmarshal
func (us *UserSettings)  Value() (driver.Value, error) { /* ... */ }
func (us *UserSettings)  Scan(value interface{}) error { /* ... */ }

However, I don't see a way to have SQLBoiler let me use UserSettings as the Go type for the "Settings" field (which in the db is "settings" and is just a string/text/varchar.

Some options of how to solve:

• Make a way to provide mappings so you can in a config file or at command line do something like field_types.user.settings = "path/to/mypackage.UserSettings" (this allows the import to be written out to the file as well, or it could be omitted to not provide an import.

• Or maybe there is an option to just let people define their own database structs? I would have no problem maintaining both the schema and the database structs - the code that SQLBoiler generates is like 97% other stuff and manually maintaining these structs is not a huge burden (especially if were an option to dump out at the console what SQLBoiler thinks it should be and I can just copy and paste from that to make what I need).

Or other ideas?

[bradleypeabody]

Related: https://github.com/volatiletech/sqlboiler/issues/606

[aarondl]

I think this issue has been answered in 606. We do support custom types and imports. Few extra notes follow:

Or maybe there is an option to just let people define their own database structs? I would have no problem maintaining both the schema and the database structs - the code that SQLBoiler generates is like 97% other stuff and manually maintaining these structs is not a huge burden (especially if were an option to dump out at the console what SQLBoiler thinks it should be and I can just copy and paste from that to make what I need).

30% of the code sqlboiler generates is the struct. Hardly worth using if you toss out that part. You can modify the templates if you don't like the way it looks. You can change the database driver you're using if you don't like the types it emits (or use type replacement/custom imports to change out types).

[bradleypeabody]

@aarondl Okay and thank you. I want to apologize for any parts of my message in #606 that were overly presumptuous - and I definitely should have read through the documentation more thoroughly first as the custom types feature certainly helps.

I would however like to open a dialog that could potentially result in some sort of solution to allowing full customization of types. Part of my frustration stems from the fact that SQLBoiler is a really awesome tool but not having a workable solution to customizing the model struct types makes it unusable for certain use cases (use cases that are very important to me and I believe others too).

The type replacement feature, if I understand it correctly, does not allow changing the types on specific fields that are of the same type in the database. Nor various other cases like adding other fields to a model struct tagged db:"-"

30% of the code sqlboiler generates is the struct.

I'm sure the exact figure depends a lot on the project and how many fields, but running SQLBoiler on a test case with a few tables for me produced 2966 lines of code, exactly 31 of which were the three structs that corresponded directly to the tables. This is upwards of 95% of the code being things other than these three model structs - and that's without the tests. With tests this figure is closer to 99%. (Again, I'm sure it varies for different sized tables, but still - SQLBoiler does a lot of stuff besides just generate the model structs - I and I think that's a good thing.)

As I see it, the model types are different from the rest of the code in the package for one specific reason: They are much more likely to interact with other parts of the system.

Generally there will be only one, maybe two max, layers doing SQL query building and execution. However, the model objects will end up being referenced potentially by many other areas of the application, not just the storage layer. After all, if in using SQLBoiler one arrives at a way to represent a User, why not at least attempt to make that representation consistent across the entire app. My point is there may be many consumers of these model structs, where as fewer for the rest of the elements of the generated SQLBoiler output.

So let me phrase the question this way - would you be open to a proposal for a feature that met the following requirements:

• Only involved one or more options to disable part(s) of the generated output (potentially printing it for developer reference instead of writing it to the file).
• Perhaps something simple to accommodate the fact that developers might need to manually maintain other files in this same directory.
• Doesn't otherwise add functionality or complication.

I would think that this would be mutually beneficial to the SQLBoiler project and to users like me who want to further customize their model types. Such a feature would not seem to pose undue maintenance strain on the project, and yet allow you to tell people like me "look, you can use X feature to define these objects manually, break a leg".

Is that more realistic and mutually agreeable? And if not is there a specific concern you have about this approach? - perhaps requirements can be modified to help with that concern.

[aarondl]

Hi @bradleypeabody. It's not a really intriguing proposal because it's geared at allowing sqlboiler to be good at the thing I'm actively discouraging. I said in my last reply that we won't add features to allow this workflow and I've still seen nothing to say that it's a good idea to change that.

The type replacement feature, if I understand it correctly, does not allow changing the types on specific fields that are of the same type in the database. Nor various other cases like adding other fields to a model struct tagged db:"-"

You can match against column name if you need to get that granular, so specific field's types are able to be changed.

Adding fields that don't belong to the database is not a use case I'd like to support. When it comes to doing things like serializing JSON for an API, or something else like that where you might want that extra field, it's recommended you create your own struct with the fields that you need. You can also use struct embedding to avoid repetition if all you'd like to do is add.

Remember that from the tool's perspective generation should be idempotent. When you go and add a database field to your schema, and re-run sqlboiler, how is it supposed to add that new database field to your custom struct it either doesn't know about or if it does know about, can't possibly understand? This gets us into having to parse the Go language with the ast libraries and do diff'ing on the struct and is way too much effort to support something that probably shouldn't be done anyway.

The struct that you get from sqlboiler is meant for a single purpose: Serializing data in and out of the database. If it's used for anything past that, it's mostly luck that it worked. It's not a good idea to try to be all things, it's best that it does the one thing and does it well.

Only involved one or more options to disable part(s) of the generated output (potentially printing it for developer reference instead of writing it to the file).

You can already edit the templates to your heart's content, see the --templates command line option. You can do whatever you'd like to them including removing the struct definition. You can add other files and generate copies of the struct that are meant to be mv'd somewhere and modified, there's plenty you can do.

Because this flexibility already exists, I don't think there's a reason to add any features for this?

Perhaps something simple to accommodate the fact that developers might need to manually maintain other files in this same directory.

No. I've said it a dozen times before including in my reply to you in #606, there will be no options or supported workflows that include editing files in the generated package. It's meant not to be touched. It sounds like you've already done your research so I won't repeat my reasoning here. It's largely a philosophical difference of opinion. And sqlboiler is squarely in the camp of "thou shalt not touch the generated code". It keeps everything simple, easy, has good boundaries, easy to understand what will happen when you run the command. The only downside is that you can't attach methods to the model structs, which doesn't bother me since there's no actual need (the only time where a method is required to be on a struct in Go is in order to satisfy an interface, and embedding can solve this issue).

I would think that this would be mutually beneficial to the SQLBoiler project and to users like me who want to further customize their model types. Such a feature would not seem to pose undue maintenance strain on the project, and yet allow you to tell people like me "look, you can use X feature to define these objects manually, break a leg".

The feature itself is potentially very complicated as well which makes the proposal even less desirable. Importantly: There seems to be a general misconception here that we're trying to be all things to all types of users/developers but it's very wrong. sqlboiler was designed from the beginning as an opinionated project that was created to solve a specific problem in a specific way that works well for us and we're not interested in taking in other tool's workflows or other user's workflows because it does add maintenance burden and complication (of which I'm basically the sole remaining receiver).

It's true that in the beginning we had hoped that the Go community would flock to sqlboiler because there are many good reasons to use it over other libraries (like GORM). But as the project has aged and most bug reports are people not being able to use it with X esoteric feature of mysql or want support for camel cased table names or other features that don't concern us it's also true that the initial inclination to make it the "one" ORM has disappeared entirely as it simply eats time for no payoff. Such is the nature of open source maintenance I suppose.

That's to say that: In order for us to support this workflow (which we don't want/need/use and also view as bad practice) we would need to devote more time to engaging in the design and implementation and no doubt maintenance of this feature. It's a very dangerous feature for users in general because of the ability for files to accidentally get deleted/overwritten either from bugs or user error which is no doubt likely to spawn many Github issues and much more time sunk into telling people to read the documentation or apologizing profusely for having cost them time and energy to recode the file that was deleted (yes, yes VCS and all that, but it's easily possible it's not yet added to the index).

So perhaps you see why from our perspective the proposal is unappealing in a variety of ways and why we won't be accepting it in any form. I do apologize as I appreciate you looking into sqlboiler and attempting to work with it and help change it in ways you think would improve it. I also respect the amount of time and energy you've taken to create your issues, it's obvious it comes from the right place. But in the end the point is to reduce maintenance, make it a tool of orthogonal features (this would be counter to that) and to prescribe one good way of working with a database in Go and so it simply doesn't fit.

[bradleypeabody]

Thanks very much for taking the time to respond in such detail. When you put it like that, I understand now where you are coming from. I get that you have to draw a line somewhere in terms of what workflow you're going to support and field all of the requests and other communication that derives from these decisions.

You can already edit the templates to your heart's content, see the --templates command line option. You can do whatever you'd like to them including removing the struct definition.

Good point - I hadn't thought of that.

In any case, I appreciate all the effort you've put in to the project and despite our differences of opinion, I definitely think SQLBoiler is indeed an awesome piece of software!