Refactor commands API

[jasonrute]

Sorry. I did this work a long time ago but never made a PR. ~I think its ready to go, but I'll look it over again tomorrow and add a better description of the work.~

Update: This is now ready to go

The main point of this PR is as follows:

• Fix all the little bugs in the commands.py file. There are many paths which were broken. For example, sometimes certain fields were not returned in the JSON, and those where made optional. Other times typos or incorrectly named fields would cause a path never to work. I think e.g. there was a field file which was incorrectly named file_name.
• Don't break if extra fields come back in the JSON. As the widgets experience taught us, lean might in the future add additional fields, and as Gabriel said: "the unspoken assumption with JSON-based protocols is that it is always safe to add new fields, and we explicitly reserve the possibility of adding new fields in the future".
  • Update: I think this is still broken in the qt server but can be fixed in another PR.
• Some requests like "sleep" and "long_sleep" don't get responses. Fixed that by adding some information to the request type and checking for that when sending the response.
• Redesign the Response hierarchy (solves Issue #6). At the first level is what we know from the JSON itself: AllMessagesResponse, CurrentTasksResponse, OkResponse, ErrorResponse. Then later, the trio server (which knows what command it sent), can further refine the response (if it is an OkResponse to the specific response via to_command_response. This requires knowing what response was sent, so it must be done by the server. Also if an ErrorResponse was returned, then send raises an error. (As I comment in the code, ErrorResponse usually mean you sent a bad request, so throwing an error is appropriate.)
  • Update: As for the qt server, I just did the minimal thing to give it the same behavior as before. It show also implement a similar logic.
• Add lots and lots of tests, solving Issue #11. This was very tedious, but I think I went down all the branches in the tests.
  • Update: QT server is not tested, but trio server and command.py are.
• A few other bug fixes or improvements to the Lean server, such as fixing the trio server kill method which was broken. Also fixes issue #2 where it freezes if you sync an unchanged file.
  • Update: Again, didn't address this for the QT server.
• ~(I think @PatrickMassot added static type checking to a different PR which got combined with this one, hence the py.typed file.)~ Nevermind. That was already in master. (I was working on a divergent branch. I just merged in Master.)

Still TO DO:

• [x] Resolve conflicts
• [x] Check and if necessary, update the QT server.
• [x] Run mypy ~and maybe a linter~ (I used PyCharm's built in tools, but haven't run mypy yet).
• [x] Check compatibility with the newest version of lean (this was last tested by me with Lean 3.9.0). Unfortunately, there are no integration tests which actually call Lean, but I do have an uncommitted file which makes lots of different types of calls to the lean server.

Do do in another PR:

• Support widgets (issue #18). I added no code in support of widgets. It is a terribly complex interface which makes me question our decision to hard code the lean interface (instead of just leaving it as JSON).

[jasonrute]

Ok. This is ready to go. What do you think @PatrickMassot ?

[jasonrute]

@Julian I'd be interested in your thoughts on this as well.

[Julian]

Support widgets (issue #18). I added no code in support of widgets. It is a terribly complex interface which makes me question our decision to hard code the lean interface (instead of just leaving it as JSON).

I mentioned I had evil ideas somewhere -- said evil ideas are I'm somewhat confident I can make this just parse the typescript interfaces into Python classes rather than reimplementing them which would super simplify things I suspect, (and do so in a way that benefits the Python ecosystem beyond this project). I have a POC of parsing Typescript interfaces but haven't had time to turn it into a real thing the past few weeks. But.. at some point I think I can do that. So my thoughts if any on that track are probably it's worth doing the bare minimum to churn along here and then probably try the sledgehammering :).

Will have a look at the rest at some point, probably after this weekend though since I keep procrastinating something else I should be working on.

[jasonrute]

I can make this just parse the typescript interfaces into Python classes rather than reimplementing them which would super simplify things I suspect

I admit I don't see the benefit of this @Julian. If we implement the classes, then we have (1) documentation, (2) a static type system that we can use mypy, PyCharm, or VSCode to inspect and catch errors, as well as (3) in editor tab completion. If we don't want any of that, then we can just use Python's usual approach of converting JSON into dicts, lists, floats, ints, bools, etc. One tool that sort of bridges those divides is ProtoBufs, but that requires the interface to be documented somewhere in a .proto file and then the python file is automatically created. (But I don't think the server protocol is compatible with ProtoBuffs, or is it documented in a similar way.)

[Julian]

Fair enough obviously I'll try and convince you before doing anything. And yeah if you're after the static typing tooling then it's definitely a tradeoff (the tradeoff being these kinds of issues with Widget -- where the server is speaking a different protocol than you are, and you get to now reimplement a chunk of annoying objects). But yeah it's not terribly important either. Will worry about reviewing this as-is first certainly.

[PatrickMassot]

Thanks a lot for all your work. Let's merge this, we can still improve stuff later.