Open dardoguidobono opened 6 years ago
We used to cache collection columns, but that got us in trouble and we decided to remove that feature.
Imagine this scenario:
I think this is important, but we don't have a solution for the above case yet. What do you think?
By the way, this is not a cache, but if you keep a reference to a collection, like this:
col := sess.Collection("table")
Then you'll be able to use it like:
col.Find()...
col.Insert()...
without having to find the table primary keys again.
Let's keep this discussion open, would be great to see different options (and we'll hopefully find a solution).
In other environments (thinking of rails and sqlalchemy here), migrations are run out-of-band from the client application, and the assumption is that the client will either be down during migrations or the migrations will be compatible with running apps, which will be rolling-restarted soon after the migrations run. Sacrificing everyday performance for safety during deployments doesn't seem like a great tradeoff.
That said, I'm also not a fan of the hidden caching and do like making collection reuse up to the application rather than managed by upper/db. In other words, I like the col.Find()
/ col.InsertReturning()
approach. Unfortunately there's currently no way I can see to perform multiple collection operations inside a transaction.
A couple rough first thoughts:
well with my proposal the collections are cached in the session not in the tx and are only queried once. if i apply the patch i dont see any further queries using the same session.
D.
when clone a database:
cachedCollections
cannot be reusedgithub.com/upper/db/mysql/database.go
line:219
func (d *database) NewDatabaseTx(ctx context.Context) (sqladapter.DatabaseTx, error) {
clone, err := d.clone(ctx, true)
if err != nil {
return nil, err
}
clone.mu.Lock()
defer clone.mu.Unlock()
connFn := func() error {
sqlTx, err := compat.BeginTx(clone.BaseDatabase.Session(), ctx, clone.TxOptions())
if err == nil {
return clone.BindTx(ctx, sqlTx)
}
return err
}
if err := d.BaseDatabase.WaitForConnection(connFn); err != nil {
return nil, err
}
return sqladapter.NewDatabaseTx(clone), nil
}
Hello: I create several transactions per second and after that use the Collection method of the tx. that triggers the PrimaryKey() and the sql to this method every time..
Can we just cache the collection Structs ?
Code snippet:
That outputs:
2018/03/23 15:02:07 Session ID: 00001 Query: SELECT "pg_attribute"."attname" AS "pkey" FROM "pg_index", "pg_class", "pg_attribute" WHERE (pg_class.oid = '"twitter_analysis"'::regclass AND indrelid = pg_class.oid AND pg_attribute.attrelid = pg_class.oid AND pg_attribute.attnum = ANY(pg_index.indkey) AND indisprimary) ORDER BY "pkey" ASC Time taken: 0.00359s Context: context.Background
2018/03/23 15:02:07 Session ID: 00002 Transaction ID: 00001 Query: SELECT "pg_attribute"."attname" AS "pkey" FROM "pg_index", "pg_class", "pg_attribute" WHERE (pg_class.oid = '"twitter_analysis"'::regclass AND indrelid = pg_class.oid AND pg_attribute.attrelid = pg_class.oid AND pg_attribute.attnum = ANY(pg_index.indkey) AND indisprimary) ORDER BY "pkey" ASC Time taken: 0.00162s Context: context.Background
2018/03/23 15:02:07 Session ID: 00003 Transaction ID: 00002 Query: SELECT "pg_attribute"."attname" AS "pkey" FROM "pg_index", "pg_class", "pg_attribute" WHERE (pg_class.oid = '"twitter_analysis"'::regclass AND indrelid = pg_class.oid AND pg_attribute.attrelid = pg_class.oid AND pg_attribute.attnum = ANY(pg_index.indkey) AND indisprimary) ORDER BY "pkey" ASC Time taken: 0.00635s Context: context.Background
maybe by adding : nd.cachedCollections = d.cachedCollections inside NewClone function of database?
Also the the caching to be effective needs a close change in the close method so it only clear the cache of collections if the database if there is no tx