volatiletech / sqlboiler

Generate a Go ORM tailored to your database schema.
BSD 3-Clause "New" or "Revised" License
6.73k stars 544 forks source link

how to scan table and parse to struct #294

Closed gensmusic closed 6 years ago

gensmusic commented 6 years ago

If you're having a generation problem please answer these questions before submitting your issue. Thanks!

What version of SQLBoiler are you using (sqlboiler --version)?

v2.6.0

If this happened at generation time what was the full SQLBoiler command you used to generate your models? (if not applicable leave blank)

If this happened at runtime what code produced the issue? (if not applicable leave blank)

What is the output of the command above with the -d flag added to it? (Provided you are comfortable sharing this, it contains a blueprint of your schema)

Please provide a relevant database schema so we can replicate your issue (Provided you are comfortable sharing this)

Further information. What did you do, what did you expect?

Currently I scan table by

rows := AccountsG().QueryP()
for rows.Next() {
    var act Account
    if err := rows.Scan(
        &act.ID,
        &act.GUID,
                    .... a lot of columns here !!!
        ); err != nil {
        fmt.Println("scan got err", err)
        continue
    }
          fmt.Println(act)
    }

Is there any built-in function to build to the struct ???

ceshihao commented 6 years ago

Please refer to https://github.com/volatiletech/sqlboiler#select

Use .One() or .All()

aarondl commented 6 years ago

See above comment. Thanks for jumping in @ceshihao :)

In addition to that, see the section on finishers in general which shows you all the ways you can "end" a query: https://github.com/volatiletech/sqlboiler#finishers

gensmusic commented 6 years ago

Sorry I didn't make myself clear. I mean, how to iterate a talbe with size like 1000000000 and scan data to struct. I don't think I can use All() in this case.

aarondl commented 6 years ago

Oh wow haha. Yeah that's a very different question. So I looked at the code a bit to see if that would work, but unfortunately Bind(rows *sql.Rows, obj interface{}) which is what would normally help you do this consumes all the *sql.Rows even if it's only scanning one object. Really it should break after it's descended into here: here.

I've corrected this on both dev and v3 branches, however neither of those are technically released yet, but dev is pretty stable right now since all our efforts are on v3 so I have no problem saying you could use the dev branch.

Having said that the performance of calling Bind over and over like that will actually be worse than hand crafting because there are a number of duplicated actions each call:

  1. It determines if your struct is a slice, slice of pointers, or struct.
  2. Columns are fetched from *sql.Rows
  3. Mapping is fetched from cache (created on the first time through)

Columns isn't going to change after every Next() and the mapping is always going to be the same. So if you need raw performance I suggest you mirror the bind call here by doing the following:

  1. Get the columns from the *sql.Rows
  2. Make a struct mapping using MakeStructMapping (this maps all of your column names to paths to pointers)
  3. Make a bind mapping using BindMapping (this maps all of the columns you care about (ie. the columns from the sql.Rows) to paths to pointers)
  4. Use PtrsFromMapping on your object to retrieve the pointers for the struct to scan into.

But of course you may want to benchmark to see if you care that much about the performance implications first :)

gensmusic commented 6 years ago

Thank you very much for your answer. It helps a lot. I love you guys.