First Class Python Support (4000USD Bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 4000USD bounty on this issue, payable by bank transfer on merged PRs implementing this. Standard bounty terms apply

---

Python is common in industry, and lacks good build tools. Mill could a be a good tool to help manage large multi-module Python projects (see What Makes Mill Unique?) providing automatic caching and parallelism for the various workflows (dependency resolution, typechecking, testing, packaging, publishing) that are core to the local development experience to help keep them fast and responsive.

The goal of this ticket is to generate a set of Python build examples that match the Java/Scala/Kotlin equivalents. We already have an example PythonModule for demo purposes, but we would need to flesh it out using the equivalent Python tools and libraries.

For the purposes of this ticket, each examples should match as closely as possible the Java/Scala/Kotlin equivalents, to provide a useful minimal-but-still-educational code example, along with associated english documentation and explanations. You should read through the relevant sections of the Building Java with Mill documentation before proceeding, even if you don't know Java, just to get a feel for what the documentation and examples for each section should cover.

1. [ ] example/pythonlib/basic/ (500USD)

   • [ ] 1-simple/: A minimal Python module demonstrating typechecking/running/testing/pex
     • We should also demonstrate how to use the Python REPL hooked up to your module
   • [ ] 2-custom-build-logic/: A Python module with custom build logic
   • [ ] 3-multi-modules/: Multiple inter-related Python modules

2. [ ] example/pythonlib/dependencies/ (500USD)

   • [ ] 1-pip-deps/
   • [ ] 2-unmanaged-wheels/
   • [ ] 3-downloading-unmanaged-wheels/
   • [ ] 4-repository-config/: examples of how to use alternate PyPI mirrors or repositories

3. [ ] example/pythonlib/linting/ (1000USD)

   • [ ] 1-black/: equivalent to 1-scalafmt but using Black
   • [ ] 2-code-coverage/: not sure what the popular Python code coverage lib is, but they should have something
   • [ ] 3-ruff/: using Ruff to lint the Python code

4. [ ] example/pythonlib/module/ (500USD)

   • [ ] 1-common-config/
   • [ ] 2-custom-tasks/
   • [ ] 3-override-tasks/
   • [ ] 4-compilation-execution-flags/
   • [ ] 5-resources/
   • [ ] 6-pex-config/

5. [ ] example/pythonlib/testing/ (500USD)

   • [ ] 1-test-suite/: we should have examples of Python being used with at least two different testing frameworks: pytest and unittest
   • [ ] 2-test-deps/
   • [ ] 3-integration-suite/

6. [ ] example/pythonlib/publishing/1-publish-module/, example/pythonlib/basic/4-realistic/ (500USD)

   • publishing/1-publish-module should demonstrate how to publish a Python package to PyPI
   • basic/4-realistic is only doable after all the previous bullets are done, so I'm grouping its bounty together with publishing/

7. [ ] example/pythonlib/web/ (500USD)

   • [ ] 1-hello-flask/: hello world website using Flask framework
   • [ ] 2-todo-flask/: TodoMVC webapp using Flask framework

8. [ ] example/pythonlib/web/ (500USD)

   • [ ] 3-hello-django/:hello world website using Django framework
   • [ ] 4-todo-django/: TodoMVC webapp using Django framework

[himanshumahajan138]

@lihaoyi Ready for this but need your Guidance (in between)

I think we should do this in parts...

[lihaoyi]

@himanshumahajan138 yes, will need to think about the best way to approach this. It would be a pretty large scale effort and will definitely need to be broken down into parts

[himanshumahajan138]

Aaah! Great approach for subparts

Working on this as it will require huge code base so will start from small to big

Let's do this ✨

[himanshumahajan138]

@lihaoyi I want(have) to create a Proper PythonModule.scala using the Last PR Code(#3882) and Need some time so fully create it for each of the above-mentioned task

So, I request to avoid Clashes if anyone trying to do this please once comment... (and i know bounties are available for everyone, Still if possible, assign me as one hand code contains flow 🤞)

[mpkocher]

@lihaoyi Perhaps It would be useful to clarify how pyproject.toml fits (or doesn't fit) into your vision for adding Python support to mill.

https://peps.python.org/pep-0621/

[lihaoyi]

@mpkocher I don't actually know yet, we'll probably need to figure that out as we flesh out the implementation

[lihaoyi]

For anyone reading, I merged https://github.com/com-lihaoyi/mill/pull/3970 which sets up the basic scaffolding:

• Code in pythonlib/src/
• Unit test code in pythonlib/test/src/, run via ./mill pythonlib.test
• Unit test code in example/pythonlib/, run via ./mill example.pythonlib.__.local.test

From there you should be able to iterate on the examples necessary for this ticket's bullet points to flesh out the necessary functionality

[himanshumahajan138]

@lihaoyi that's great you just gave the foundation brick for the building ✨

[lihaoyi]

Also the example Python module walkthrough is mandatory reading for anyone who wants to start on this https://mill-build.org/mill/extending/example-python-support.html

[himanshumahajan138]

@lihaoyi i will work on this expanding the previous code just after completing the kotlin springboot

[sideeffffect]

There's this new Python tool https://rye.astral.sh which may be a source of inspiration for Mill support for Python. What is innovative about Rye, compared to previous Python tools is that it's aiming to be a one-stop tool for anything related to working with a Python project:

• not just managing packages/dependencies, or just
• managing a venv, but crucially also
• managing a Python version/implementation (there's e.g. PyPy besides CPython, etc)
• running the project
• running the tests in the project
• publishing the package to a repository
• running formaters
• running linters
• running type checkers
• etc...

If Mill covered well a broad spectrum of workflows people working with Python need to do, coupled with Mill's goodies, like implicit parallelism and aggressive caching, it could be a tempting proposal for at least part of the Python community. Good luck with this! :crossed_fingers:

[himanshumahajan138]

@lihaoyi actually i was working and stuck at some testing part as per the official python docs i have seen that python use unittest and pytest library for this purpose but when i checked the test folder i saw that you used some other way (something like done in javalib and others)

Now the question is what way we have to follow ? and if you are targeting something like javalib and others then can you give some guidance or some link related to testing docs for mill

by the way i have used unittest for now when i will raise PR we will Groom it there for sure...

Thanks...

[lqhuang]

There's this new Python tool https://rye.astral.sh/

Features of rye have all merged into uv https://docs.astral.sh/uv/. uv is promising and definitely the best practice for now.

Python use unittest and pytest library for this purpose

Yeah, unittest is a built-in standard library. For a demo project, it's quick and easy, though almost every main stream libs adapt pytest as test runner.

[himanshumahajan138]

Python use unittest and pytest library for this purpose

Yeah, unittest is a built-in standard library. For a demo project, it's quick and easy, though almost every main stream libs adapt pytest as test runner.

Yes! but the way @lihaoyi is doing (check the other examples in javalib and others) i am concerned about that coz i didn't got any docs and unable to get a spike of learning 😅

I think @lihaoyi have to provide some Guide and through some light on it

[jodersky]

Cross-posting a comment from #3992 here.

I'll work on the missing fundamentals (test support, repl etc), then @himanshumahajan138 will take care of the basic examples, while I work on the dependencies part

[jodersky]

I'll work on dependencies next, now that #3992 is almost done