IDEA: unify rust neon_local and python test fixtures

[kelvich]

Since now neon_local is meant only for local development and not for distribution we can unify them. What about having a python class to handle compute and storage start/stop and rewrite neon_local in python?

[kelvich]

WDYT @hlinnaka ?

[hlinnaka]

Works for me.

That said, I'd also love to have Helm chart that installs the whole shebang into a kubernetes cluster. Including pageserver, safekeepers, proxy, mock S3 server, etcd, and the cloud console. If we had that, we could perhaps replace our python test harness with that too. That's a more ambitious goal, though.

[yeputons]

A wild idea: rewrite test fixtures to Rust instead :)

[hlinnaka]

A wild idea: rewrite test fixtures to Rust instead :)

That's not a crazy idea either. pytest is a nice framework, but I think doing more with Rust would allow us to do more "white-box" testing.

[LizardWizzard]

AFAIK there were some reasons against using rust for integration tests, so back then they were rewritten in python, and that's the reason why we have this split between rust that remained in the local control plane and python tests. I like both ideas. If people are ok with recompiling test code I'm more towards using rust everywhere. (No more problems with different python versions and virtualenvs yay). So in general I vote for having local control plane/cli and tests written in the same language so the code is shared through some library. If the language is rust we can share even more with actual binaries.

[phoenix24]

have Helm chart that installs the whole shebang into a kubernetes cluster. Including pageserver, safekeepers, proxy, mock S3 server, etcd, and the cloud console.

This would be super useful, and should be a separate issue.

[cicdteam]

Works for me.

That said, I'd also love to have Helm chart that installs the whole shebang into a kubernetes cluster. Including pageserver, safekeepers, proxy, mock S3 server, etcd, and the cloud console. If we had that, we could perhaps replace our python test harness with that too. That's a more ambitious goal, though.

When I wrote charts for proxy and console I had idea to write charts for all components and one uber chart that includes all them as subcharts (so you all install whole stack by single command). It useful for tests and for community who wants to try Neon locally. But what to do with console? It couldn't be public chart. But without console how to create/start/delete projects? I have no answer.

[hlinnaka]

Works for me. That said, I'd also love to have Helm chart that installs the whole shebang into a kubernetes cluster. Including pageserver, safekeepers, proxy, mock S3 server, etcd, and the cloud console. If we had that, we could perhaps replace our python test harness with that too. That's a more ambitious goal, though.

When I wrote charts for proxy and console I had idea to write charts for all components and one uber chart that includes all them as subcharts (so you all install whole stack by single command). It useful for tests and for community who wants to try Neon locally. But what to do with console? It couldn't be public chart. But without console how to create/start/delete projects? I have no answer.

We have got to make the web console public too, or at least extract all the needed parts and publish them. Need to discuss the exact mechanics of that..

[cicdteam]

My view - for hobby projects and local tests I prefer have/use some cli tool to manage computes instead of make console public.