Closed nj1973 closed 6 months ago
We need to do this pretty soon because installing the hdfs package on requires requests-kerberos which, on Debian 11, caused me to have to install:
sudo apt-get -y install krb5-config libkrb5-dev gcc python3-dev
This is unnecessary seeing as we are limited to a BigQuery target at the moment. This issue will simplify installation greatly.
Some of the packages we require to be installed pull down a number of dependencies, an example being the Snowflake client. If we know we are an Oracle/GCP shop it would be nice to not need to install packages related to Snowflake, Synapse or Hadoop.
I wondered if we could do this with sections in the
pyproject.toml
file. For example the basic installation only installs core dependencies. We then need to additionally install optional-dependencies for the distributions we are interested in. e.g.: