volatiletech / sqlboiler

Generate a Go ORM tailored to your database schema.
BSD 3-Clause "New" or "Revised" License
6.73k stars 544 forks source link

Eager loading with many-to-many relationship generates wrong type #176

Closed jfernstad closed 7 years ago

jfernstad commented 7 years ago

What version of SQLBoiler are you using (sqlboiler --version)?

v2.5.0

If this happened at runtime what code produced the issue? (if not applicable leave blank)

    response, err := models.As(DB,
        Where("id = ?", aID),
        Load("Bs.Setting"),
    ).One()

Please provide a relevant database schema so we can replicate your issue (Provided you are comfortable sharing this)

screen shot 2017-07-12 at 10 43 28

Using MySQL.


CREATE TABLE IF NOT EXISTS `A` (
  `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
);

CREATE TABLE IF NOT EXISTS `B` (
  `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
);

CREATE TABLE IF NOT EXISTS `Settings` (
  `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
  `b_id` INT UNSIGNED NULL,
  `value` VARCHAR(45) NULL,
  PRIMARY KEY (`id`),
  INDEX `pk_setting_b_id_idx` (`b_id` ASC),
  CONSTRAINT `pk_setting_b_id`
    FOREIGN KEY (`b_id`)
    REFERENCES `B` (`id`)
);

CREATE TABLE IF NOT EXISTS `A_B` (
  `a_id` INT UNSIGNED NOT NULL,
  `b_id` INT UNSIGNED NOT NULL,
  PRIMARY KEY (`a_id`, `b_id`),
  INDEX `pk_b_id_idx` (`b_id` ASC),
  CONSTRAINT `pk_a_id`
    FOREIGN KEY (`a_id`)
    REFERENCES `A` (`id`),
  CONSTRAINT `pk_b_id`
    FOREIGN KEY (`b_id`)
    REFERENCES `B` (`id`)
);

Further information. What did you do, what did you expect?

Hi, I'm trying to load all the information necessary for a "fat" response for a webservice and I want to Load data a few tables separated from the request. The tables above are hopefully equivalent to what I have (can't share, sorry).

I'm hoping to get Settings for all Bs that have a relationship with A but this query generates the following runtime panic:

interface conversion: interface {} is *models.BSlice, not *[]*models.B

This is happening deep in (again equivalent) function models.bL.LoadBSetting.

In my mind *BSlice and *[]*B are basically the same?

Will gladly accept any suggestions to workarounds.

aarondl commented 7 years ago

Hey there. I wasn't able to reproduce this on the master branch.

Here's the schema I used (I had to name it differently:

CREATE TABLE IF NOT EXISTS `videos` (
  `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
);

CREATE TABLE IF NOT EXISTS `tags` (
  `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
);

CREATE TABLE IF NOT EXISTS `settings` (
  `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
  `tag_id` INT UNSIGNED NULL,
  `value` VARCHAR(45) NULL,
  PRIMARY KEY (`id`),
  INDEX `pk_setting_b_id_idx` (`tag_id` ASC),
  CONSTRAINT `pk_setting_b_id`
    FOREIGN KEY (`tag_id`)
    REFERENCES `tags` (`id`)
);

CREATE TABLE IF NOT EXISTS `video_tags` (
  `video_id` INT UNSIGNED NOT NULL,
  `tag_id` INT UNSIGNED NOT NULL,
  PRIMARY KEY (`video_id`, `tag_id`),
  INDEX `pk_b_id_idx` (`tag_id` ASC),
  CONSTRAINT `pk_a_id`
    FOREIGN KEY (`video_id`)
    REFERENCES `videos` (`id`),
  CONSTRAINT `pk_b_id`
    FOREIGN KEY (`tag_id`)
    REFERENCES `tags` (`id`)
);

insert into videos () values ();
insert into videos () values ();
insert into tags () values ();
insert into tags () values ();
insert into tags () values ();
insert into video_tags (video_id, tag_id) values (1, 1);
insert into video_tags (video_id, tag_id) values (1, 2);
insert into video_tags (video_id, tag_id) values (2, 1);
insert into video_tags (video_id, tag_id) values (2, 3);
insert into settings (tag_id, value) values (1, 'hello');
insert into settings (tag_id, value) values (2, 'there');
insert into settings (tag_id, value) values (3, 'world');

The meat of the code I used to test:

    response, err := models.Videos(db,
        qm.Where("id = ?", 1),
        qm.Load("Tags.Settings"),
    ).One()

    spew.Dump(response)

Run gives this output:

SELECT * FROM `videos` WHERE (id = ?) LIMIT 1;
[1]
select `a`.*, `b`.`video_id` from `tags` as `a` inner join `video_tags` as `b` on `a`.`id` = `b`.`tag_id` where `b`.`video_id` in (?)
[1]
select * from `settings` where `tag_id` in (?,?)
[1 2]
(*models.Video)(0xc42000d4a0)({
 ID: (uint) 1,
 R: (*models.videoR)(0xc42000d580)({
  Tags: (models.TagSlice) (len=2 cap=2) {
   (*models.Tag)(0xc42000d620)({
    ID: (uint) 1,
    R: (*models.tagR)(0xc4200f9050)({
     Settings: (models.SettingSlice) (len=1 cap=1) {
      (*models.Setting)(0xc4200195c0)({
       ID: (uint) 1,
       TagID: (null.Uint) {
        Uint: (uint) 1,
        Valid: (bool) true
       },
       Value: (null.String) {
        String: (string) (len=5) "hello",
        Valid: (bool) true
       },
       R: (*models.settingR)(<nil>),
       L: (models.settingL) {
       }
      })
     },
     Videos: (models.VideoSlice) <nil>
    }),
    L: (models.tagL) {
    }
   }),
   (*models.Tag)(0xc42000d640)({
    ID: (uint) 2,
    R: (*models.tagR)(0xc4200f9080)({
     Settings: (models.SettingSlice) (len=1 cap=1) {
      (*models.Setting)(0xc420019600)({
       ID: (uint) 2,
       TagID: (null.Uint) {
        Uint: (uint) 2,
        Valid: (bool) true
       },
       Value: (null.String) {
        String: (string) (len=5) "there",
        Valid: (bool) true
       },
       R: (*models.settingR)(<nil>),
       L: (models.settingL) {
       }
      })
     },
     Videos: (models.VideoSlice) <nil>
    }),
    L: (models.tagL) {
    }
   })
  }
 }),
 L: (models.videoL) {
 }
})

At commit: 0b027fa01c78dcc2036564fb7dabbd1229509821 (current master, no code changes)

jfernstad commented 7 years ago

I'm stumped. It crashes with your schema as well.

SQLBoiler commit: cc47da44 MySQL version: 5.7.18 golang version: go version go1.8.1 darwin/amd64 SQLBoiler command: ./sqlboiler --basedir ./vendor/github.com/vattle/sqlboiler --wipe --no-tests --no-hooks mysql

sqlboiler.yml contents, we use goose to migrate db tables:

blacklist:
  - goose_db_version
# note: tinyint-as-bool will treat tinyint(1) as bool instead of int8 - the display width (1) is significant!
tinyint-as-bool: true
mysql:
  dbname: $MYSQL_DB
  host: $MYSQL_HOST
  port: $MYSQL_PORT
  user: $MYSQL_USER
  pass: $MYSQL_PASSWORD
  sslmode: false

There doesn't seem to be any relevant code changes from HEAD of master @ c43e8561 affecting this.

I had to change the name of a constraint in the video_tags table before the server would accept the tables because of colliding names.

CREATE TABLE IF NOT EXISTS `video_tags` (
  `video_id` INT UNSIGNED NOT NULL,
  `tag_id` INT UNSIGNED NOT NULL,
  PRIMARY KEY (`video_id`, `tag_id`),
  INDEX `pk_b_id_idx` (`tag_id` ASC),
  CONSTRAINT `pk_video_tag_id`
    FOREIGN KEY (`video_id`)
    REFERENCES `videos` (`id`),
  CONSTRAINT `pk_tag_tag_id`
    FOREIGN KEY (`tag_id`)
    REFERENCES `tags` (`id`)
);

And here is the SQL tables as read back from the MySQL server:

mysql> show create table videos;
+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table  | Create Table                                                                                                                                          |
+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| videos | CREATE TABLE `videos` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1 |
+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+

mysql> show create table settings;
+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table    | Create Table                                                                                                                                                                                                                                                                                                                                                  |
+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| settings | CREATE TABLE `settings` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `tag_id` int(10) unsigned DEFAULT NULL,
  `value` varchar(45) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `pk_setting_b_id_idx` (`tag_id`),
  CONSTRAINT `pk_setting_b_id` FOREIGN KEY (`tag_id`) REFERENCES `tags` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1 |
+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

mysql> show create table tags;
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                        |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| tags  | CREATE TABLE `tags` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1 |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------+

mysql> show create table video_tags;
+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table      | Create Table                                                                                                                                                                                                                                                                                                                                                                           |
+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| video_tags | CREATE TABLE `video_tags` (
  `video_id` int(10) unsigned NOT NULL,
  `tag_id` int(10) unsigned NOT NULL,
  PRIMARY KEY (`video_id`,`tag_id`),
  KEY `pk_b_id_idx` (`tag_id`),
  CONSTRAINT `pk_tag_tag_id` FOREIGN KEY (`tag_id`) REFERENCES `tags` (`id`),
  CONSTRAINT `pk_video_tag_id` FOREIGN KEY (`video_id`) REFERENCES `videos` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Code:

    response, err := models.Videos(e.Managers.DB,
        Where("id = ?", 1),
        Load("Tags.Settings"),
    ).One()

Crash log:

SELECT * FROM `videos` WHERE (id = ?) LIMIT 1;
[43]
select `a`.*, `b`.`video_id` from `tags` as `a` inner join `video_tags` as `b` on `a`.`id` = `b`.`tag_id` where `b`.`video_id` in (?)
[43]
2017/07/14 10:49:04 http: panic serving 127.0.0.1:57062: interface conversion: interface {} is *models.TagSlice, not *[]*models.Tag
goroutine 4216 [running]:
net/http.(*conn).serve.func1(0xc422038000)
    /usr/local/Cellar/go/1.8.1/libexec/src/net/http/server.go:1721 +0xd0
panic(0x1756d00, 0xc4221a8440)
    /usr/local/Cellar/go/1.8.1/libexec/src/runtime/panic.go:489 +0x2cf
gitlab.project.com/project-cloud/project-webservice/models.tagL.LoadSettings(0x1c09720, 0xc422175d60, 0x0, 0x17c5fc0, 0xc4221d81a0, 0x0, 0x0)
    /Users/joakim/Projects/go/src/gitlab.project.com/project-cloud/project-webservice/models/tags.go:235 +0x9f8
reflect.Value.call(0xc42196e380, 0xc42098a060, 0x13, 0x180cf1b, 0x4, 0xc420331c38, 0x4, 0x4, 0xc422166250, 0x170f840, ...)
    /usr/local/Cellar/go/1.8.1/libexec/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0xc42196e380, 0xc42098a060, 0x13, 0xc420331c38, 0x4, 0x4, 0xc4221d8230, 0x199, 0xc42196e380)
    /usr/local/Cellar/go/1.8.1/libexec/src/reflect/value.go:302 +0xa4
gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries.loadRelationshipState.callLoadFunction(0x1c09720, 0xc422175d60, 0xc4221de180, 0xc4221d8080, 0x2, 0x2, 0x1, 0x17c5fc0, 0xc4221d81a0, 0x16, ...)
    /Users/joakim/Projects/go/src/gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries/eager_load.go:184 +0x40d
gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries.loadRelationshipState.loadRelationships(0x1c09720, 0xc422175d60, 0xc4221de180, 0xc4221d8080, 0x2, 0x2, 0x1, 0x17c5fc0, 0xc4221d81a0, 0x2, ...)
    /Users/joakim/Projects/go/src/gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries/eager_load.go:101 +0x598
gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries.loadRelationshipState.loadRelationshipsRecurse(0x1c09720, 0xc422175d60, 0xc4221de180, 0xc4221d8080, 0x2, 0x2, 0x0, 0x178df20, 0xc421f7a880, 0x199, ...)
    /Users/joakim/Projects/go/src/gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries/eager_load.go:213 +0x209
gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries.loadRelationshipState.loadRelationships(0x1c09720, 0xc422175d60, 0xc4221de180, 0xc4221d8080, 0x2, 0x2, 0x0, 0x17f93a0, 0xc421f7a880, 0x0, ...)
    /Users/joakim/Projects/go/src/gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries/eager_load.go:117 +0x4c8
gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries.eagerLoad(0x1c09720, 0xc422175d60, 0xc422048b90, 0x1, 0x1, 0x17f93a0, 0xc421f7a880, 0x0, 0x0, 0x0)
    /Users/joakim/Projects/go/src/gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries/eager_load.go:57 +0x166
gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries.(*Query).Bind(0xc42188e2c0, 0x17f93a0, 0xc421f7a880, 0x0, 0x0)
    /Users/joakim/Projects/go/src/gitlab.project.com/project-cloud/project-webservice/vendor/github.com/vattle/sqlboiler/queries/reflect.go:114 +0x1cb
gitlab.project.com/project-cloud/project-webservice/models.videoQuery.One(0xc42188e2c0, 0xc422175d60, 0xc421f7a860, 0x3)
    /Users/joakim/Projects/go/src/gitlab.project.com/project-cloud/project-webservice/models/videos.go:92 +0x74

This seems to be the panicing code, tags.go:235:

    if singular {
        object = maybeTag.(*Tag)
    } else {
        slice = *maybeTag.(*[]*Tag) // <<-- Panic here
        count = len(slice)
    }

Instead of posting the contents of tags.go, settings.go and videos.go, I'll post their file length for easier matching.

File Lines of code
tags.go 1593
settings.go 1167
videos.go 1282

Actually, here is a zip of the 3 files. generated_files.zip

Code is failing both in tests (using Ginkgo) and using a request to the webservice (using Postman).

I'll keep testing but if you have any other ideas, I'm open.

aarondl commented 7 years ago

I think you're on to something. I diff'd your files against my generated files and here's the things I found:

33a34,43
> var SettingColumns = struct {
>   ID    string
>   TagID string
>   Value string
> }{
...

The above shows that this commit is missing: f5e53ac

---
>   var queryMods []qm.QueryMod
191,192c194,195
<       qm.InnerJoin("`video_tags` as `b` on `a`.`id` = `b`.`tag_id`"),
<       qm.Where("`b`.`video_id`=?", o.ID),
---
>       qm.InnerJoin("`video_tags` on `tags`.`id` = `video_tags`.`tag_id`"),
>       qm.Where("`video_tags`.`video_id`=?", o.ID),
196c199,204
<   queries.SetFrom(query.Query, "`tags` as `a`")
---
...

The above shows that these commits are missing: 35563d1 2168a70

The second commit there 2168a70 is right next to a commit 466ea1f which fixes an occurence of the issue you're seeing.

Something to remember is that sqlboiler finds itself in your GOPATH and uses its templates from there. So it's possible that's why this part is happening. Maybe you have an old version of sqlboiler hanging around somewhere (maybe vendored?). Everything points to the fact that there's old code -somewhere-. Even if your sqlboiler binary itself reports v2.5.0 the most important part is actually the template files on disk, and the libraries on disk (in this case github.com/vattle/sqlboiler/queries).

jfernstad commented 7 years ago

Oh wow. Our setup compiles SQLBoiler before every run from the vendored directory. We do this for CI but of course, I never let the CI run through this code because it failed locally.

I'll sanitize my directories and try again, after vacation. 😁

Thanks for your time and input!

aarondl commented 7 years ago

No problem. Re-open this if you continue to see problems :)