nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
637 stars 116 forks source link

Track transitive dependencies via pip-tools #252

Closed nchammas closed 4 years ago

nchammas commented 6 years ago

Advantages to using pip-tools:

Disadvantages:

@Perados, what do you think?

This doesn't simplify the problem of packaging. I'm also not sure if it fully addresses the issue you raised, since the compiled dependencies don't affect the package that gets published to PyPI. This may be OK, though, since you were interested in building your own packages reliably. And if the other approaches out there like Poetry and Pipenv prove to be a lot more work, then at least we have something here we can use in the interim.

Fixes #248. Prompted by the discussion here: https://github.com/nchammas/flintrock/issues/169#issuecomment-386199046

nchammas commented 4 years ago

Tested this end-to-end with a full run of the test suite against AWS.