The future of community modules (the `testcontainers-*` packages)

[alexanderankin]

The testcontainers-* packages, e.g.:

• https://pypi.org/project/testcontainers-arangodb/
• https://pypi.org/project/testcontainers-postgres/

were added as part of the last release, but it introduces a lot of complexity, for what could be just a lot of optional dependency groups in one package.

As it stands, the dependency tree is something like:

testcontainers
+--- testcontainers-core -> *
+--- [postgres] (to represent "group" if testcontainers[postgres] or testcontainers[minio,postgres] is requested to be installed)
     +--- testcontainers-postgres (*)
          +--- testcontainers-core (*)
          +--- sqlalchemy (*)
          \--- psycopg2-binary (*)

but it could look like:

testcontainers
+--- [postgres] (to represent "group" if testcontainers[postgres] or testcontainers[minio,postgres] is requested to be installed)
     +--- sqlalchemy (*)
     \--- psycopg2-binary (*)

downsides:

• of course users may get import errors, and we should then wrap those to educate users on how to install testcontainers[postgres] rather than just testcontainers.
• temporary pain while we lack 100% parity with testcontainers-java (log + port waiting, rather than importing sqlalchemy)

upsides:

• better developer experience on both sides (library and client side)
• we can publish a single package to pypi, we still have yet to hear back from Till, and he is the only maintainer on those packages.

[alexanderankin]

all three currently available maintainers have a consensus (2 strongly, and one slightly) leaning towards the consolidated package approach but I wanted to make this decision publicly recorded in github issues, as well as to let people weigh in for consideration as well. thank you.

[totallyzen]

hey @alexanderankin I've modified the title and pinned this as a major issue. We need to talk more about this with everyone as it isn't just about package layout and dependencies, it also means we have a LOT of exponential testing to do and that is becoming prohibitive with the costs to running those.

Comments on what to do are welcome!

[covatic-john]

Hey happy to help where I can

[totallyzen]

Hey @covatic-john thanks for the offer! We'll definitnely take that up! I need to wrap my head around how we can split some of the work.

• [x] We're fairly settled to have only one package with extras at this point. It's just too complex to manage PyPI otherwise.
• [ ] What remains is increasing quality and decide on what "official" container flavours to support and how.
• [ ] Specifically the work with mypy is a pain in the back because of the many modules we already have.
• [ ] Splitting the work by modules might help to parallelise the work.

In addition, having chatted with Testcontainers staff they are also keen on reducing the number of "official" modules and try to make the generic containers flexible and easy to use (improving wait conditions for example).

If you fancy comparing tc-java, tc-go and tc-python for features it would be awesome as it gives us some ideas on what to aim for. To be clear, no expectations, just giving you ideas on what you could help with.

Cheers!

[covatic-john]

I’ll take look :) have a great weekend

On 16 Feb 2024, at 09:22, Bálint Bartha @.***> wrote:

Hey @covatic-john https://github.com/covatic-john thanks for the offer! We'll definitnely take that up! I need to wrap my head around how we can split some of the work.

We're fairly settled to have only one package with extras at this point. It's just too complex to manage PyPI otherwise. What remains is increasing quality and decide on what "official" container flavours to support and how. Specifically the work with mypy is a pain in the back because of the many modules we already have. Splitting the work by modules might help to parallelise the work. In addition, having chatted with Testcontainers staff they are also keen on reducing the number of "official" modules and try to make the generic containers flexible and easy to use (improving wait conditions for example).

If you fancy comparing tc-java, tc-go and tc-python for features it would be awesome as it gives us some ideas on what to aim for. To be clear, no expectations, just giving you ideas on what you could help with.

Cheers!

— Reply to this email directly, view it on GitHub https://github.com/testcontainers/testcontainers-python/issues/412#issuecomment-1948024744, or unsubscribe https://github.com/notifications/unsubscribe-auth/BC3KZVP2A5I264BORCCMSB3YT4QOXAVCNFSM6AAAAABB3KFVTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBYGAZDINZUGQ. You are receiving this because you were mentioned.

--

This email and any attachments are confidential, may contain privileged material and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Covatic. If you are not the intended recipient of this email, you must neither take any action based upon its content, nor copy nor show it to anyone. 

Covatic is a limited company registered in England & Wales. Registration number:

10422855. Registered office: 412-413 Scott House, The Custard Factory, Gibb Street, Birmingham, B9 4AA.