setup_connection, create_table, and other common functions

[thorwhalen]

In many storage systems there is a "connect" and a "create location" concern.

Examples of what I mean by these terms:

• connect: sql, mongo and s3 need to create a connection to the "server". In a local file system, no connection is needed as such.

• create location: In sql and mongo we need a DB and a table. In a local file system, we need a root directory to have been created in order to write in it. In s3, a bucket needs to be created, but we can write to any/path/in/this/bucket (as far as permissions allow) without creating directories (there are no actual directories in s3, only a pretense of them.

My wish is to have common names for common functionalities throughout different storage systems, so what should we call the methods implementing the "connect" and "create location" concern? I propose connect and mk_location, but invite comments/proposals.

Note that we can synonymize these methods in the particular dialect of a context, such as

mkdir = mk_location  # in local files, ftp, or ssh
create_table = mk_location  # in sql
create_collection = mk_location

How do we deal with the fact that some contexts need multiple locations to be made (a db and a table for example)?

What should the interface of mk_location be?

Does https://www.python.org/dev/peps/pep-0249/ have any opinions to guide us?

[Kulv3r]

@thorwhalen what if we just go with a simple generic approach of having setup and teardown methods? And actual implementation is left to specific requirements of each backend.

[thorwhalen]

Sure. It seems to be the pattern. But I'd like to see a clear view of what it means for different systems (file(-like) systems, sql, nosql, timeseries, graphdbs, etc.) and what the sub-patterns are for these different classes of storage systems are.

Also, does it make sense to use the terms __enter__ for setup and __exit__ for teardown, under the philosophy of staying close to python builtins and idioms, or rather use the language of DBs (only if they're consistent).

[Ranix]

Using the "adapter pattern design" as approach to solve this, the easy way to follow up. is with the words CONNECT, DISCONNECT.

Really don't mater the end point of the connector(DB, Storage, etc) only the type of connector you need to solve your specific problem.

[Ranix]

For the basic functions operations in the engines, we can use the CRUD standard definition.

https://en.wikipedia.org/wiki/Create,_read,_update_and_delete

[thorwhalen]

Great pic and table. I will reuse these, for sure.

Know though, that the current focus of py2store is on list-like (item collections) and dict-like (key-value) operations, and not exactly those listed. Instead of update we want list (iterate over keys -- or elements of a collection). Having an update method is useful and probably will add that perspective in the near future. For now though, we can make an update with a combination of the existing base methods.