search

DataDog / datadogpy

The Datadog Python library
https://datadoghq.com/
Other
609 stars 302 forks source link

Why obscure build system? #718

Closed takeda closed 2 years ago

takeda commented 2 years ago

Is there a reason why datadog package adopted an obscure build system?

Python currently has:

And that's ignoring many, many of of other build systems that were created and died.

Adding a new project, especially that it's only few months old makes it hard to use it with tooling that automates packaging. In my case I use https://github.com/nix-community/poetry2nix which has supports the above, but this is brand new and locks me now from using datadog package.

Looking at setup.py and pyproject.toml I don't see anything that poetry (which is becoming more and more popular recently) doesn't handle. Could this step be reconsidered?

ofek commented 2 years ago

Hello there! I can answer your questions.

Is there a reason why datadog package adopted an obscure build system?

Our goal is to remove all setup.py files since their use is now heavily discouraged and build with a PEP 517 compatible build system while defining metadata with PEP 621.

setuptools

Disadvantages:

Flit

flit-core is the build backend behind Flit.

Disadvantages:

Poetry

poetry-core is the build backend behind Poetry.

This was actually a no-go option for us due to its persistent unwillingness to adopt new standards if they are deemed suboptimal (see comments on PEP 621 and PEP 665)

Hatchling

In my case I use https://github.com/nix-community/poetry2nix which has supports the above

This project never used Poetry. Does it have special logic for other backends?

Could this step be reconsidered?

Not without more compelling reasons than those used to justify the switch.

takeda commented 2 years ago

This project never used Poetry. Does it have special logic for other backends?

One of goals of Nix is full reproducibility. So instead of downloading a pre-made wheel it actually rebuilds every package. poetry2nix tries to maintain that principle so it rebuilds every package, because of it, it needs to understand various build systems:

https://github.com/nix-community/poetry2nix/blob/master/overrides/build-systems.json

many packages also have non-python dependencies (or are broken), so they need overrides like these:

https://github.com/nix-community/poetry2nix/blob/master/overrides/default.nix

Going back to the reasoning. PEP 665 was rejected and PEP 621 appears to be purely about project metadata. I don't see poetry's author immediately dismissing it, but just saying that it will take time to implement and also pointing that it doesn't support all use cases.

As an user I was extremely frustrated when PPA pushed everyone out of setuptools, even though setup.cfg was fully declarative and could be decent if it was simply improved. The new toml configuration did not cover all setuptools functionality and caused many projects to depend on external libraries to read toml.

They apparently did not learn from the past, did not research all tools and did not ensure that their implementation covers all use cases. So I actually sympathize with the author and appreciate trying not to break things for me. After all as a developer, I do not care about politics involving my build tool. I want it to work and not learn about incompatible change when I'm pushing a fix to production.

I also think that creating a new build tool, because you disagree with somebody's opinion about how metadata is provided in a build configuration file a bit weird as far as standardization goes. This is especially exacerbated, because as I see you work for DataDog, so essentially DataDog decided to not go with any existing build standard and and create their own for their own packages.

I know I won't convince you, but this is very weird and ironically anti-standard way of doing things only hurting anyone else who tries to use and package this module/package.

ofek commented 2 years ago

it needs to understand various build systems

I see you opened https://github.com/nix-community/poetry2nix/issues/564, I'll add hatchling to https://github.com/NixOS/nixpkgs tomorrow

PPA pushed everyone out of setuptools

Not true, rather users were given alternatives https://www.python.org/dev/peps/pep-0517/#abstract

caused many projects to depend on external libraries to read toml

True, will be fixed by https://www.python.org/dev/peps/pep-0680/

They apparently did not learn from the past, did not research all tools

This is ignoring the numerous lengthy discussions that took place which included people from academia, industry, and other packaging ecosystems like Linux & Conda

and did not ensure that their implementation covers all use cases

Sure, though do you have examples? Pragmatically speaking, a standard that initially covers the majority of use cases and can be updated over time is better than one that initially covers all present and future use cases but will never materialize.

so essentially DataDog decided to not go with any existing build standard and create their own [...] very weird and ironically anti-standard way of doing things

  1. You're conflating standards and tools. The PEPs made it so there is no standard tool but rather any tool is valid as long as the standards are implemented.
  2. Hatch as a project predates my employment at Datadog 😉
ofek commented 2 years ago

I mentioned you in https://github.com/NixOS/nixpkgs/pull/163088 🙂

takeda commented 2 years ago

@ofek I see, I guess I was missing some background, but I disagree about setuptools, it pushed me away. And I really tried to stick to it and pip-tools. Pip progressively made it harder for me.

Thanks for the commit to nixpkgs, I guess if hatch and hatchling would be included in nixpkgs then perhaps adding support to poetry2nix would be very easy.

It looks though like your PR was merged, but right now it only is adding you as a maintainer. Will you be making a new PR to add these packages?