GORM v2 is on going

[jinzhu]

Hello All,

GORM v2 is under active development (https://github.com/jinzhu/gorm/tree/v2_dev), going to release in the next two months.

Before that, I am NOT going to merge any pull requests based on the master branch.

V2 will be overwritten from scratch with similar public API, it would focus on performance, improve usability and fix fragile designs. We will provide a migration guide to help users migrate before the release.

With the new architecture, it opens the possibility to have a better migration tool, a static code generator that generates type-safe code with all the features of GORM, visualizes models and its relationships to help a new team member quickly understand your project, and even support Redis & MongoDB...

Your code review or suggestions would be much appreciated, please comment to this thread, thank you.

[cocktailer]

Any ETA?

[glebtv]

Would be nice if dirty tracking would be possible, since fields are not plain types anymore, this is a long-requested feature: #612 #2147 #2952

612 even has an implementation suggestion

And just to clarify - will db.User.First() from your example return a user.User (with string type name) or a db.User (with StringField type name)?

[kiwi2019]

Can you add support for Read-write separation ？？？

[fishyww]

@jinzhu hi~ Do you have a release plan for version v2?

[yalva]

Any plans for spatial data types in V2?

[jinzhu]

Hello everyone, I have released a public testing version "v0.2.5" for v2, which is used on some internal production projects already.

If you want to test it, please use the import path: "gorm.io/gorm", version: "v0.2.5", feedbacks welcome!

I am still working on the release note and writing documents, the final release date will depend on feedbacks, If everything goes well, it will be the time we reach 20k stars!

[jinzhu]

@jinzhu Is there a support for generating and querying additionnal fields on join table (Many2Many relation) ?

We have better support for this in v2, it will be easier to use, refer method 'SetupJoinTable', and waiting for the documents for details.

[napei]

Official mongo support?

[jinzhu]

If type safety is your main concern and a motivation behind this feature, I believe a more sturdy type safety would be provided by generating code based on reverse engineering you sql schema, not by reading your go structs.

Added it supports.

[ipfans]

@jinzhu Any support plan for batch actions? e.g. insert records etc.

[jinzhu]

Official mongo support?

Possible to support it in v2, but I am not going to release a MongoDB driver when releasing v2.

[jinzhu]

@jinzhu Any support plan for batch actions? e.g. insert records etc.

Supported it, just use db.Create(&users)

[fr3fou]

Hello everyone, I have released a public testing version "v0.2.1" for v2, which is used on some internal production projects already.

If you want to test it, please use the import path: "gorm.io/gorm", version: "v0.2.1", feedbacks welcome!

I am still working on the release note and writing documents, the final release date will depend on feedbacks, If everything goes well, it will be the time we reach 20k stars!

is there any way to see documentation for v2?

[mshomali]

Hello everyone, I have released a public testing version "v0.2.2" for v2, which is used on some internal production projects already.

If you want to test it, please use the import path: "gorm.io/gorm", version: "v0.2.2", feedbacks welcome!

I am still working on the release note and writing documents, the final release date will depend on feedbacks, If everything goes well, it will be the time we reach 20k stars!

Official Oracle support?

[flyfy1]

Hi @jinzhu , I can see that the v2 code has been pushed to master. Is there a ChangeLog stating what's diff for v1 => v2?

[mtfelian]

Pushed to master?

If it is backward incompatible, it will break many things. Major releases should be released in separate folder module like /v2, according to https://blog.golang.org/v2-go-modules but i don't see v2 subdir

[jinzhu]

Pushed to master?

If it is backward incompatible, it will break many things. Major releases should be released in separate folder module like /v2, according to https://blog.golang.org/v2-go-modules but i don't see v2 subdir

V2 will use different import path, and the codebase already separated, so merged into master won't break things. V1 will keep using codebase from https://github.com/jinzhu/gorm

[leefernandes]

Syntax to group conditions something akin to the following would be nice to see in v2

db.Where(
  db.Where("pizza = ?", "pepperoni").And(db.Where("size = ?", "small").Or("size = ?", "medium")),
).Or(
  db.Where("pizza = ?", "hawaiian").And("size = ?", "xlarge"),
).Find(&pizzas)

output:

select * from pizzas where (pizza = 'pepperoni' and (size = 'small' or size = 'medium')) or (pizza = 'hawaiian' and size = 'xlarge')

upperdb demonstrates a neat implementation for composing conditions. https://upper.io/db.v3/getting-started#composing-conditions-db-or-and-db-and

db.And(
  db.And(
    db.Cond{"age >": 21},
    db.Cond{"age <": 28},
  ),
  db.Or(
    db.Cond{"name": "Joanna"},
    db.Cond{"name": "John"},
    db.Cond{"name": "Jhon"},
  ),
)

( (age > 21 AND age < 28) AND (name = 'Joanna' OR name = 'John' OR name = 'Jhon') )

And hopefully composed conditions would support complex subqueries, not just basic operands/operators.

[axetroy]

@jinzhu I found that v2 has released a new version

Is there any changelog to show the difference with v1?

[kiyonlin]

Is it more reasonable to use db, err := gorm.Open(sqlite.New("gorm.db"), &gorm.Config{})? Because now sqlite.Open is not connected to the database, while returning just a gorm.Dialector implement.

[zouhuigang]

Is batch insertion supported? Upgrade now!!

[Kisesy]

@axetroy @kiyonlin @zouhuigang https://github.com/go-gorm/gorm/wiki/GORM-V2-Release-Note-Draft-CN

[jinzhu]

I am working on the English version Release Note, it will come out sometime this week.

[RichardLindhout]

The api looks really nice. Are there plans for adding benchmarks?

[jinzhu]

The API looks really nice. Are there plans for adding benchmarks?

I have modified https://github.com/yusaer/orm-benchmark to test GORM v2's performance with prepared statement cache enabled and w/o default transaction (like others), although the suit seems to have some issues, the results are unstable at each run, in general, the results still pretty good.

[frederikhors]

It would be amazing to have benchmarks against go-pg.

[longhaoteng]

@jinzhu Hi, mysql string type in the table changed from varchar to longtext, is this a bug or a feature?

and

type Model struct {
    ID        uint      `json:"id" gorm:"primary_key"`
    CreatedAt time.Time `json:"created_at"`
    UpdatedAt time.Time `json:"updated_at"`
}

type Test struct {
    Model
    Name     string `json:"name"`
}

db.AutoMigrate(&Test{})

Error Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(3) NULL,updated_atdatetime(3) NULL,namelongtext,PRIMARY KEY (id))' at line 1

v1's datetime becomes datetime(3),what should I pay attention to?

[jinzhu]

It would be amazing to have benchmarks against go-pg.

Tested against it, performance is similar if not better.

[jinzhu]

@jinzhu Hi, mysql string type in the table changed from varchar to longtext, is this a bug or a feature?

and

type Model struct {
  ID        uint      `json:"id" gorm:"primary_key"`
  CreatedAt time.Time `json:"created_at"`
  UpdatedAt time.Time `json:"updated_at"`
}

type Test struct {
  Model
  Name     string `json:"name"`
}

db.AutoMigrate(&Test{})

For string fields without size, not a primary key, no index defined and don't have default values, its data type changed to longtext to avoid incautiously trimming for MySQL, you can define a size for it, it was 256 in v1.

Error Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(3) NULL,updated_atdatetime(3) NULL,namelongtext,PRIMARY KEY (id))' at line 1

v1's datetime becomes datetime(3),what should I pay attention to?

What's your DB version?

[jinzhu]

Draft Release Note released:

https://github.com/go-gorm/gorm/wiki/GORM-V2-Release-Note-Draft

[tanweerdev]

Postgres ErrorCodes support please. https://www.postgresql.org/docs/10/errcodes-appendix.html

I am available to discuss and work on this ErrorCodes support. would it be fine to discuss ErrorCodes support here or should I start a new issue? @jinzhu

[longhaoteng]

@jinzhu

What's your DB version?

Version : 5.5.60-0+deb7u1
OS : Raspberry Pi
Hub : https://registry.hub.docker.com/r/hypriot/rpi-mysql

[jinzhu]

Postgres ErrorCodes support please. https://www.postgresql.org/docs/10/errcodes-appendix.html

I am available to discuss and work on this ErrorCodes support. would it be fine to discuss ErrorCodes support here or should I start a new issue? @jinzhu

Hello, @tanweerdev Let's start a new issue, this issues is too long to discuss. thank you.

[jinzhu]

@jinzhu

What's your DB version?

Version : 5.5.60-0+deb7u1
OS : Raspberry Pi
Hub : https://registry.hub.docker.com/r/hypriot/rpi-mysql

datatype datetime(3) is supported since MySQL 5.6, is there any reason using 5.5?

[longhaoteng]

datatype datetime(3) is supported since MySQL 5.6, is there any reason using 5.5?

hypriot/rpi-mysql is the most used mirror image of Raspberry Pi, it hasn't been updated for more than two years, you can see that there are 5m users, there's no better option at this time. V2 has many advantages to drive upgrades, v1 is supported below 5.6 version, if only because the data support this causes upgrades, that is a regret.

[jinzhu]

datatype datetime(3) is supported since MySQL 5.6, is there any reason using 5.5?

hypriot/rpi-mysql is the most used mirror image of Raspberry Pi, it hasn't been updated for more than two years, you can see that there are 5m users, there's no better option at this time. V2 has many advantages to drive upgrades, v1 is supported below 5.6 version, if only because the data support this causes upgrades, that is a regret.

Upgrade https://github.com/go-gorm/mysql and use it like:

import (
  "gorm.io/driver/mysql"
  "gorm.io/gorm"
)

db, err := gorm.Open(mysql.New(mysql.Config{
  DSN: "gorm:gorm@tcp(localhost:9910)/gorm?charset=utf8&parseTime=True&loc=Local", // data source name, refer https://github.com/go-sql-driver/mysql#dsn-data-source-name
  DisableDatetimePrecision: true, // disable datetime precision support (added since mysql 5.6)
  DefaultStringSize: 256, // add default size for string fields, by default, will use db type `longtext` for fields without size, not a primary key, no index defined and don't have default values
}), &gorm.Config{})

[longhaoteng]

@jinzhu good job, thanks.

[popsUlfr]

v2 looks great :)

This might be a stupid question but there's no Close() on DB anymore, I was wondering how the connection to the database is closed or when ? Also wondering if there's a way to access the underlying sql.DB object like in v1 ?

[jinzhu]

@popsUlfr

Also wondering if there's a way to access the underlying sql.DB object like in v1 ?

Added DB method https://github.com/go-gorm/gorm/commit/e487f355a0838bbc158c5c7d848b35753d290884#diff-1f97616c9377ded874040d08c43d6165

This might be a stupid question but there's no Close() on DB anymore, I was wondering how the connection to the database is closed or when ?

As GORM using DB connection pool, so for an application, usually it is not necessary to use the Close method, so we removed it to avoid misuse

[popsUlfr]

@jinzhu Thank you!

As GORM using DB connection pool, so for an application, usually it is not necessary to use the Close method, so we removed it to avoid misuse

In the case of sqlite3 for instance, wouldn't that be problematic if let's say the database is locked in exclusive mode but my application is done doing work with it and wants to release the resources to let another application open it ? Also some operations might be scheduled to run when closing a connection to the database.

Or if generally the work my application is doing with the database is short-lived and it wants to release the resources associated with the connection manually ?

It seems weird to me to reach the end of my application and potentially leave a database connection still open. In my opinion it still makes sense to expose a Close() but maybe document the fact that it might not be necessary in most cases. Although with you exposing the underlying *sql.DB object, one may close it this way which is fine as well.

[jinzhu]

Although with you exposing the underlying *sql.DB object, one may close it this way which is fine as well.

Let's use this then ;)

[leefernandes]

in v2 I'm having an issue with uuid default value function gen_random_uuid() failing against postgres because gorm is wrapping it in single quotes in the CREATE TABLE query.

CREATE TABLE "harumphs" ("id" bigserial,"created_at" timestamptz,"updated_at" timestamptz,"deleted_at" timestamptz,"ok" uuid NOT NULL DEFAULT 'gen_random_uuid()',PRIMARY KEY ("id"))

type Harumph struct {
    gorm.Model
    Test uuid.UUID `gorm:"type:uuid;not null;default:gen_random_uuid()"`
}

in v1 it works because the migrator did not wrap the default in single quotes:

CREATE TABLE "harumphs" ("id" bigserial,"created_at" timestamptz,"updated_at" timestamptz,"deleted_at" timestamptz,"ok" uuid NOT NULL DEFAULT gen_random_uuid(),PRIMARY KEY ("id"))

I believe it may be related to the migrator checking field.DataType when building a SQL string. I was hoping because my model uses uuid.UUID which is a [16]byte that it would not assign DataType of string but it is. https://github.com/go-gorm/gorm/blob/master/migrator/migrator.go#L77

[leefernandes]

I'm seeing the following AutoMigrate err on fields of type pq.StringArray. This type works in gorm v1.

unsupported data type: &[]

type Harumph struct {
    gorm.Model
    Things pq.StringArray `gorm:"type:text[]"`
}

However if I define my own StringArray type with basic Scan & Value implementations, the unsupported data type err does not occur:

type StringArray []string

func (m StringArray) Value() (driver.Value, error) {
    return json.Marshal(m)
}

func (m *StringArray) Scan(src interface{}) error {
    b, ok := src.([]byte)
    if !ok {
        return errors.New("StringArray.Scan byte assertion failed")
    }

    var value StringArray
    if err := json.Unmarshal(b, &value); err != nil {
        return err
    }

    *m = value

    return nil
}

[leefernandes]

Here's a scenario that worked in v1 but is no longer working in v2. An embedded struct could implement Scan & Value interface methods. err & example below.

[error] unsupported data type: [123 34 67 111 110 116 101 110 116 34 58 110 117 108 108 125]
panic: runtime error: invalid memory address or nil pointer dereference
/gorm.io/gorm/schema.(*Schema).ParseField(0xc0000d8000, 0xc41f85, 0x8, 0x0, 0x0, 0x1042580, 0xd14580, 0xc41f8f, 0x12, 0x58, ...) 
/gorm.io/gorm/schema/field.go:297 +0xe65
/gorm.io/gorm/schema/schema.go:117 +0x59f
/gorm.io/gorm/statement.go:315 +0x5f

type Harumph struct {
    gorm.Model
    Embedded `gorm:"type:jsonb;"`
}

type Embedded struct {
    Content interface{}
}

func (m Embedded) Value() (driver.Value, error) {
    return json.Marshal(m)
}

func (m *Embedded) Scan(src interface{}) error {
    b, ok := src.([]byte)
    if !ok {
        return errors.New("Embedded.Scan byte assertion failed")
    }

    var value Embedded
    if err := json.Unmarshal(b, &value); err != nil {
        return err
    }

    *m = value

    return nil
}

A workaround is not embedding the struct, but for a particular pattern I've been using on a project, the embed is preferred.

Non embed works:

type Harumph struct {
    gorm.Model
    Named Embedded `gorm:"type:jsonb;"`
}

[popsUlfr]

Byte slices ([]byte) also throw unsupported data type: &[] errors when migrating.

[jinzhu]

@ItsLeeOwen @popsUlfr

Fixed all those issues, big thanks for your report.

Please upgrade GORM to v0.2.7

[popsUlfr]

Fixed all those issues, big thanks for your report.

It works good job.

With sqlite I noticed that any type information is completely discarded when creating a table :

CREATE TABLE `sqlar` (`name` ,`mode` ,`mtime` ,`sz` ,`data` ,PRIMARY KEY (`name`))

sqlite has dynamic typing sure, but storing the type intents has value.

Here's the model by the way (using sqlite archive) :

// SQLAR https://www.sqlite.org/sqlar.html
type SQLAR struct {
    // name of the file
    Name string `gorm:"COLUMN:name;TYPE:TEXT;PRIMARY_KEY"`
    // access permissions
    Mode uint32 `gorm:"COLUMN:mode;TYPE:INT"`
    // last modification time
    MTime time.Time `gorm:"COLUMN:mtime;TYPE:INT;AUTOUPDATETIME"`
    // original file size
    SZ uint64 `gorm:"COLUMN:sz;TYPE:INT"`
    // compressed content
    Data []byte `gorm:"COLUMN:data;TYPE:BLOB"`
}

// TableName sqlar
func (SQLAR) TableName() string {
    return "sqlar"
}

Also a question about AUTOUPDATETIME : sqlite does not have the shortcut like mysql has with ON UPDATE CURRENT_TIMESTAMP but it could be done with a trigger. Does gorm inject the update timestamp during an update statement automatically ?

[jinzhu]

Hi @popsUlfr

Need to upgrade the database driver, for SQLite, the latest version is v1.0.2

Also a question about AUTOUPDATETIME : sqlite does not have the shortcut like mysql has with ON UPDATE CURRENT_TIMESTAMP but it could be done with a trigger. Does gorm inject the update timestamp during an update statement automatically?

Yes

[popsUlfr]

Need to upgrade the database driver, for SQLite, the latest version is v1.0.2

Ok done.

This is with gorm.io/driver/sqlite@v1.0.2.

I did a simple create, updates and select but the AUTOUPDATETIME creates confusion when trying to read back the value :

ar := models.SQLAR{
    Name:  "myfile.txt",
    Mode:  0666,
    MTime: time.Now(),
    SZ:    uint64(len(content)),
    Data:  []byte(content),
}
db.Create(&ar)
db.Model(&ar).Updates(map[string]interface{}{"sz": len(newContent), "data": []byte(newContent)})
db.First(&ar)

Debug() mode output :

SELECT count(*) FROM sqlite_master WHERE type='table' AND name="sqlar"
CREATE TABLE `sqlar` (`name` TEXT,`mode` INT,`mtime` INT,`sz` INT,`data` BLOB,PRIMARY KEY (`name`))
INSERT INTO `sqlar` (`name`,`mode`,`mtime`,`sz`,`data`) VALUES ("myfile.txt",438,"2020-06-18 12:19:07.178",735,"<binary>")
UPDATE `sqlar` SET `data`="hello world",`sz`=11,`mtime`=1592475552 WHERE `name` = "myfile.txt"
SELECT * FROM `sqlar` WHERE `sqlar`.`name` = "myfile.txt" ORDER BY `sqlar`.`name` LIMIT 1

sql: Scan error on column index 2, name "mtime": unsupported Scan, storing driver.Value type int64 into type *time.Time

EDIT : also just realised that the types are there with v1.0.2 :) nice

[jinzhu]

MTime time.Time `gorm:"COLUMN:mtime;TYPE:INT;AUTOUPDATETIME"`

The type is not necessary, or set it to datetime or time.