Closed tombaeyens closed 3 years ago
- sodasql (why was this again?)
Because you can then refer to it as a meta package in for example your 5 min tutorial
. It would basically install a default subset of modules, like eg the postgres connector.
You can go without the package but then your 5 min tut
would state something like:
Install
pip install soda-sql-base soda-sql-postgres-connector soda-sql-snowflake-connector ...
vs
Install
pip install soda-sql
, which installs thebase
package along with for example thepostgres
connector.
I think the DX should be more like:
pip install soda-sql[all]
to install everything and pip-install soda-sql[posgres]
or `pip install soda-sql[snowflake]' etc. The AFAIK we can do this using using setuptools extras.
My 2 cents: I am asking myself who would need ever to download all the warehouse supports? I thinks that’s a degenerate case that we should maybe not optimise for by doing it with pip install soda-sql
. I agree with Vinjay on this: perhaps better to leave soda-sql as core dependency only and as people to fetch warehouse they wish to use.
As discussed during Engineering Meeting:
A prospect today mentioned we should consider binary packaging. Did we evaluate that option? If we support that, it seemed to give him more confidence that our tool would not clash with other tools like the AWS CLI.
In my opinion, binary packaging would mean we have to bundle all the modules together, not sure if that's a worthy goal to pursue because there will be significant amount of dependencies that many users might not need/use (e.g., people using redshift, are also getting all other deps). OTOH, since they will be isolated, it is only going to increase the payload size rather than conflicting with the python environment.
For generating installer, we can use http://www.pyinstaller.org
Splitting code into modules shouldn't be causing any issues with this goal though.
Meeting notes: we discussed and will initially go for splitting up the modules only and not pursue the binary packaging route that was mentioned.
@vijaykiran I m closing sodadata/soda-sql#314 (install fails on Python 3.9.2) as a duplicate of this issue. If that is a wrong assumption, please reopen it.
Done!
Folder structure to be discussed
Libraries use dashes in the names, no underscores. Folder names as indicated below all use underscores.
Also, before starting I would like to understand how we'll deal with versioning the individual libs and how this impacts the release process.
TODO investigate if dialects can be extracted. Passing data from core lib to dialects is no prob. But The dialects should not invoke methods on core stuff. That would cause circular dependencies. To be investigated before starting this.