i2mint / py2store

Tools to create simple and consistent interfaces to complicated and varied data sources.
MIT License
11 stars 2 forks source link

Dirs, files, and key-dependent value transformation #77

Open thorwhalen opened 3 years ago

thorwhalen commented 3 years ago

See ZipReader in slib.s_zipfile.py. This store has two value categories: Directories and Files. Both categories are distinguishable by the keys, through the "ends with slash" convention.

When a file, the value return is bytes, as usual.

When a directory, the value returned is a ZipReader itself, with all params the same, except for the prefix which serves to specify the subfolder (that is, prefix acts as a filter).

See filesys.py. It has some file stores and some dir stores. Separate. In what ways could we make stores that do both, like ZipReader does (but not necessarily how it does -- perhaps there's a better way, a more specifically appropriate one, or on the other hand, a more general (and reusable) pattern.

See new_s3.py. Same question/task.

Read misc.py and understand how it works. Perhaps this is a better approach to the problem. It fits since it solves the general problem of returning a different kind of object as a function of a property of the key (in misc.py that property is the extension -- but a more general setup is present there already).

Perhaps the approach that misc takes should be generalized to a general pattern/utility/decorator that can be used to solve the dir vs. file problem for zip, local files, and s3.