Design discussion

[boxed]

Moved from https://github.com/pypa/pip/issues/4768#issuecomment-499151294

[boxed]

@csdummi said "But what is wrong with my project? What is the right idea git has?"

Your project seems very nice to make CLI apps but git is pluggable. You can shadow commands, add commands, put in aliases and probably more. All without touching the code of git. In fact git itself is implemented as a bunch of stand alone programs.

This is what I want to do with p.

[CSDUMMI]

I read your p.py and saw that you filter from the PATH variable? Is that useful? Wouldn't it be nicer to have a config file like the .gitconfig, where the path to each standalone binary would be put? Then find_available_commands() loads that file and p searches only through them.

[boxed]

Well no, because that would make it much more cumbersome to add a command. And I use that data on the entire file system to find executables that I p calls so I need that anyway.

[CSDUMMI]

Good, another question: Do you plan on implementing p with a little bit more OOP?

[boxed]

I don't think so. There doesn't seem to be much use for it at this point and it can make it harder to port to another language which is the plan.

[CSDUMMI]

Of which language are you thinking?

[CSDUMMI]

Just a question on the side: I'm trying to write the Documentation/UserGuide.md installation part, on which operating systems does p currently work?

[boxed]

Re languages: C# and Java are the two big missing ones. It's more complex for Java because not everyone uses maven. But go, kotlin, CMake, C configure/make, etc would be needed for MVP I think.

[boxed]

Re OS: I've only used it on macOS but it should be the same on linux. But it's missing parts like installing the python tool chain via brew (and installing brew!) so it's not complete there either.

[CSDUMMI]

https://github.com/boxed/p/issues/4#issuecomment-500213178 Do you mean by that, you want to rewrite p in another language?

[boxed]

Yes. When it's totally clear what the functionality should do it makes sense to rewrite the prototype in some native language so binaries can be shipped without the massive dependency that python3 is.

[CSDUMMI]

And in which language do you think p should be rewritten?

[CSDUMMI]

A p config command, like the git config command

p needs command to change global settings, install language packages and do other things, that we add later. I think there should be a binary called p-config include in p, that can do stuff like this.

[boxed]

Re language: don't know. http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-retrospective/ makes a strong case for OCAML, but things have changed since then so I'd have to reevaluate.

[CSDUMMI]

Three options for the language issue:

First option

We could leverage the architecture of p and differentiate between parts of p, each doing one job and written in one language. But not all parts use the same language, just each part uses one.

Con

This also means that programmers, that only know one or two languages are fixed to support p in a single part of p. And it divides the developers and makes it impossible for one person to keep up to date on every part of the project and all their dependencies.

The second option

We could also split p into three parts (and possibly more):

• p core (p and p.py in this repo)
• p languages (Everything to support one language for p)
• p config (Configuration should be like p core, but limited to a fixed set of commands, not changed by installing p language packages)

Each one of them can choose different languages, and is moved to different repositories.

Third option

We fix p and all parts of p to a single language and don't allow anything else into the build of p.

[boxed]

I think option 2 is what I'm talking about, except that "p config" is a part of "p core". As for many repositories, well.. let's not go crazy. If a language can be supported by a 5 line .ini file then it makes sense to have a separate repo for it.

[CSDUMMI]

I don't think every language should be supported in a 5 line file of whatever format. There will be such languages where you have to implement a whole new binary to support p, compile it and put it into the package file. Such files are huge and to support them these language packages will need whole repos. But at first I will implement p config and these two commands:

$  p config install <language-package> # Download and install language support package
$  p config remove <language-package> # Uninstall and remove language support package

This includes defining a package format and that should be able to handle even binaries and compresses them over the network communication.

[CSDUMMI]

I want to make another argument for using different languages in p-core and p-config. p-core doesn't do networking, all of the installing, downloading and uploading is done by language support. All p-core is doing is:

• accessing environment variables
• reading files
• evoke other programs
• regex matching

For these jobs, you don't need a very high level language. We could optimize this part of p perhaps even to work in C.

But p-config is totally different, it needs to:

• evoke other commands
• download files (networking)
• upload files (networking)
• write to files
• append to files
• do string manipulation
• compress files
• decompress files

I wouldn't want to do that in a low level language, like C. . If we want to optimize p-core and p-config is included in p-core, we couldn't use a low level language for the basic jobs p-core does, because that would complicate the p-config implementation as well.

Conclusion

That's why I'd use different languages in p-core and p-config, because one can be optimized by using lower level languages efficiently, while the other needs many high level libraries for networking and such things.

[boxed]

Well, if we go with OCaml, that's a super HIGH level language, but that also produces very fast native code. It's just that I am more used to python so will prefer that for a prototype.

[CSDUMMI]

I don't want to change to another language yet, but I think that p-core can be better optimized, if it doesn't include p-config, due to all the string manipulation, networking, etc. that p-config has to work with. I don't know OCaml and can't judge about its performance, but I think it wouldn't be a disadvantage to separate p-core and p-config and then decide about a language in both of them independently and with an eye to performance vs development.

[CSDUMMI]

I think we should honestly think about using C for p-core. We can optimize p-core with C and letting it call binaries written in higher level languages to do things, like networking, string manipulation, etc. Python programs can be optimized by just implementing it in C. And p-core doesn't do too complex things, that we can implement it in C.

[CSDUMMI]

C++, not C necessarily, but I think a fast and low level language would be able to do what p-core needs to do.

[CSDUMMI]

I tried to rewrite some part of p in C++ on the rewrite_C branch. I've got to say it is very complex and that we have to mix C and C++ stdlib. And sometimes it is even worse, because C and C++ have no functions to work with the filesystem efficiently. I thus conclude that we would increase our work by using C or C++ by a factor of 10+. We need to look somewhere else to get a good language to rewrite p in.

[boxed]

Yea, C/C++ is worse than one imagines. I used to be a C++ programmer for 9 years. Those were dark days.

[CSDUMMI]

I now made an attempt at writing p in Haskell and though simplified somethings, it seems like Haskell would be much easier then C and only a little bit more complex then Python. rewrite_haskell branch

[CSDUMMI]

Summary of the rewrite of p in Haskell

I wrote a simplified p-core version in Haskell. Haskell is a functional language and can be compiled or interpreted. And Haskell uses the stack project manager, and you can run p-core, either by executing the p/p link or typing stack run, given that you installed stack. I didn't implement auto detection yet, but if you have the right binaries installed in ~/.p/packages/<language>/<commands>, you can execute the command by typing p <language> <command>.

To conclude I think Haskell has better support for third-party libraries and in general Haskell is better than C++, but probably a little more complicated than Python.