Update setup.py with git based build/release tagging

[vijaykiran]

Currently building the local version uses a hardcoded __dev__ as a version, it is better to use the git SHA/ref as a version.

• [ ] update setup.py to pickup the git ref and set SODA_SQL_VERSION
• [ ] Update documentation on how to build/install locally
• [ ] Make sure that CI/CD uses the same process

[dirkgroenen]

Make sure that CI/CD uses the same process

Are you proposing we stop using semantic versioning and instead use commit hashes?

[vijaykiran]

No, semver allows adding build metadata. The version can be x.y.z-{build-info} can still be used without any issues. Here the build-info will be commit/ref info.

https://semver.org/#spec-item-10

[milanaleksic]

Is this a well-known practice (meaning are there other packages that do it?).

For example sodasql doesn't seem to have any dependency with that naming scheme:

(.venv) milan@MilanMBP (server_api_adaptations) ~/SourceCode/sodasql → pip list
Package                    Version   Location
-------------------------- --------- -------------------------------
appdirs                    1.4.4
asn1crypto                 1.4.0
attrs                      20.3.0
azure-common               1.1.25
azure-core                 1.8.2
azure-storage-blob         12.5.0
bleach                     3.3.0
boto3                      1.15.18
botocore                   1.18.18
cachetools                 4.2.0
certifi                    2020.12.5
cffi                       1.14.3
chardet                    3.0.4
click                      7.1.2
colorama                   0.4.4
coverage                   5.4
cryptography               3.2.1
distlib                    0.3.1
docker                     4.4.1
docutils                   0.16
Faker                      5.3.0
filelock                   3.0.12
future                     0.18.2
google-api-core            1.24.1
google-auth                1.24.0
google-cloud-bigquery      2.5.0
google-cloud-core          1.5.0
google-crc32c              1.1.0
google-resumable-media     1.2.0
googleapis-common-protos   1.52.0
grpcio                     1.34.0
idna                       2.10
importlib-metadata         1.7.0
iniconfig                  1.1.1
isodate                    0.6.0
jeepney                    0.6.0
Jinja2                     2.11.2
jmespath                   0.10.0
keyring                    22.0.1
MarkupSafe                 1.1.1
more-itertools             8.6.0
msrest                     0.6.19
oauthlib                   3.1.0
oscrypto                   1.2.1
packaging                  20.9
pip                        21.0.1
pkginfo                    1.7.0
pluggy                     0.13.1
proto-plus                 1.13.0
protobuf                   3.14.0
psycopg2-binary            2.8.5
py                         1.10.0
pyasn1                     0.4.8
pyasn1-modules             0.2.8
PyAthena                   1.10.7
pycparser                  2.20
pycryptodomex              3.9.8
Pygments                   2.7.4
PyJWT                      1.7.1
pyOpenSSL                  19.1.0
pyparsing                  2.4.7
pytest                     6.0.2
pytest-cov                 2.10.1
pytest-html                3.1.1
pytest-metadata            1.11.0
python-dateutil            2.8.1
python-dotenv              0.13.0
pytz                       2020.1
PyYAML                     5.3.1
readme-renderer            28.0
requests                   2.23.0
requests-oauthlib          1.3.0
requests-toolbelt          0.9.1
rfc3986                    1.4.0
rsa                        4.6
s3transfer                 0.3.3
SecretStorage              3.3.0
setuptools                 47.1.0
six                        1.15.0
snowflake-connector-python 2.3.6
soda-sql                   0.0.0     /Users/milan/SourceCode/sodasql
tabulate                   0.8.7
tenacity                   6.2.0
text-unidecode             1.3
toml                       0.10.2
tox                        3.15.2
tox-docker                 2.0.0
tqdm                       4.56.0
twine                      3.3.0
urllib3                    1.25.10
virtualenv                 20.4.2
webencodings               0.5.1
websocket-client           0.57.0
zipp                       3.4.0

[vijaykiran]

Releases are based on tags hence you don’t see it in the pypi. The idea is not to replace the release version numbers, although there’s no hard and fast rule.

I meant using a updated script that will give reasonable build information. The tagged releases will have the same semantics, but we can add additional metadata to other builds.

I suggest using https://github.com/python-versioneer/python-versioneer - please see the rationale in the read me of the project.

[dirkgroenen]

I might be off so please correct me if needed, but what's the actual gained benefit when using eg versioneer (or any other solution for that matter)? We already have tags which are associated to a build commit and can therefore always resolve back to a specific moment in time.

Would it be mostly for people who decide on 'building' their own version from master and then start reporting bugs, so we know what 'version' it is their using - or do you have other things in mind?

[vijaykiran]

The proposal of this issue to make the versioning part of the build process itself (setup.py) rather than in GitHub actions .

Indeed we have a tag associated but it is only added during the CI phase. This means installing from local checkout will never have any reasonable tag nor version number. Also, as you may know tags can be (re)attached to any commit ref,

And, yes - as you pointed out it is very helpful when devs are reporting bugs (as described in versioneer readme file I linked above).

Thee reason to use versioneer is just DRY :) and configure the behaviours we want.. We can also use just git commands to populate the version (as other projects do in references below).

References: https://github.com/apache/airflow/blob/d2577e42a93f2f1d25ba3759e561f2e1e70b5f1d/setup.py#L129 https://github.com/apache/superset/blob/538443893b07b561dbce5f27f54dbd5eaa65824e/setup.py#L28 (uses json)

[tombaeyens]

Looks like there is consensus. It s probably best to plan this in after we have the demo.soda.io up and running. Is that ok Vijay?

[vijaykiran]

Agreed, not an urgent issue for now :)

[tombaeyens]

I think we have versioning properly covered now.