Suggestion: Rewrite in Golang

[etiennedi]

The current version of the CLI tool resulted as a "we need something quick" prototype and is currently written in Python. I'm proposing to rewrite it in Golang for the following reasons:

Why Golang?

• UX The UX of a software product should generally not be affected much by the language its written in, however, we are seeing a major UX impact at the moment: Because we cannot compile the CLI into a single binary, the user has to have a python environment and even install specific dependencies. Especially as the tool grows, version conflicts of either Python itself or the used dependencies could lead to issues, as we have very little control over the user's environment. With Golang we could cross-compile for all systems (darwin, linux and even Windows) and the single binary would contain everything that's needed.

• Developer Synergies Currently our stack of long-term maintained products and applications sees Golang as it's first priority. While Python is also common in our stack, it is commonly used for one-off solutions, import and data preperation scripts, etc. As a result we have a lot of knowledge, examples and processes around what makes our Golang development high-quality (testing knowledge and examples for all levels, CI discipline, examples of proper architecture patterns such as Clean Architecture in Weaviate, etc).

• Code Synergies We have quite a bit of code dealing with weaviate in Go already that might be reused. For example, the auto generated REST client (from go-swagger) is versioned together with weaviate, so we could easily reuse the code. If we were to adjust to API changes in the current client a lot of effort of the development of the CLI tool would be creating different clients for different weaviate versions. There are other synergies, such as the used types which are defined in entities/. According to Clean Architecture this represents the innermost circle, meaning these are rules that are "enterprise wide", i.e. they span beyond a single application. They'd therefore be safe to reuse in tools directly related to Weaviate, such as the CLI.

• CLI Currently untested I would not argue that testing in Golang is any better or worse than in Python - in fact I don't know much about properly testing Python. However, I would make the argument that the lack of tests in the current implementation are tech debt which needs to be mitigated. Writing tests to test code that has already been written can often times be more expensive than rewriting (I've heard a major US consultancy that starts with a "T" quote numbers saying 2-3 times as much effort as writing the code in the first place). Therefore in order to increase the test coverage we need to invest considerable effort anyway. At this point it might be cheaper - and we'd end up with the better tests - if we simply rewrote it altogether.

• Great support for CLI tools in Golang With cobra and potentially viper there is amazing support for CLI tools with a great UX in Golang. Especially with the suggestions mentioned in #14, cobra could improve the DX (and UX, too) quite a bit.

Why not Golang?

• Effort to rewrite The only real argument I currently see against a golang-based rewrite is that a rewrite takes time. However, considering the fact that the CLI tool overall is rather lightweight at the moment and we need to invest time into adding tests and potentially changing the UX-related features from #14, my argument would be that we need to spend this time anyway.

• Happy to add other counter-arguments, please respond in a comment and I'll add them here

cc @bobvanluijt @fefi42

[etiennedi]

By the way I opened this issue not because I think we should do this urgently. Instead I want us to have the discussion soon and be aligned early. Then we can find a point in time when such a rewrite would make sense.

[bobvanluijt]

I’m completely agnostic on this. So, if @fefi42 agrees I’m all for it.

PS: For what it's worth, Python was used for prototyping because of the gcloud cli-software also uses P.

[fefi42]

For now my Go-lang skills are very small so for high priority features it might be unavoidable to stay on python for a little while until I build these skills. From an engineering perspective I totally see the side that we want to have one technology stack. On the other side it is probably way easier to find programmers that have experience with python than with go.

[etiennedi]

For now my Go-lang skills are very small so for high priority features it might be unavoidable to stay on python for a little while until I build these skills.

Fair point. As mentioned above, I don't see any urgency in doing such a rewrite urgently. We also shouldn't postpone it too long though, as the tool is still relatively small and would be considerably easier to rewrite now than later. Completely agree with still adding features in Python if they are high prio.

A more interesting change could be the UX changes outlined in #14. Depending on what the strategy is on when to publicly announce and promote the CLI (@bobvanluijt?), I'd strongly suggest to do the UX improvements which are all breaking changes first.

I don't have a strong preference of whether we do "Tech-Rewrite and UX changes" in a single run or do them separately. If we go "UX changes first. Then Rewrite", we'll probably touch everything twice though. Having said that because of the small code-base the cost would be limited.

From an engineering perspective I totally see the side that we want to have one technology stack

cool 👍

On the other side it is probably way easier to find programmers that have experience with python than with go.

I'm not too worried about this. Go is designed to be easy to be picked up. The skills that you need to write good Go code (proper architecture patterns, isolation, abstraction, testability, test coverage etc.) aren't language specific. Someone who writes high-quality Python code will also write high-quality Go code after a bit of training.

For what it's worth, Python was used for prototyping because of the gcloud cli-software also uses P.

I was hoping I could find a blog post or something about why they made that decision, but can't find anything. My - completely made up- theory would be that Python was quite common among Infra engineers for automation already. So they got started with what was common. Not so sure if Google would still choose Python today if they started building the tool from scratch. ;-)

[fefi42]

The CLI is now pip installable and uses the python client as a backend. This make the installment and publishing very easy and results in an extremely slim codebase within the CLI project itself. The CLI should even work on windows although I have not had the chance to test that.

From a one technology stack perspective python currently plays a big role within SeMI and will probably do so for a while with solution facing challenges.