Testing discussion/braindump

[twolfson]

As mentioned in #99, I have been trying to find a reasonable way to test curses on and off. This issue will start as a braindump/rant and then continue as a discussion for what we should test and when.

For the repo, my initial reaction to testing is:

1. Get a high level test or 2 in place first
   • For example, something that launches Suplemon, presses a key, and verifies a key was written to the window
   • This serves as a good baseline to make sure that no PR ever fully breaks Suplemon
2. Test inner API parts on piece by piece basis
   • For example, verify that running the window command "move cursor left", moves the cursor left (mind blown)
   • This gives us more coverage with less setup/teardown (and maybe headaches) for each test
   • The negative side here is that if an internal API changes, then the test will need to be updated/rewritten; resulting in wasted time/effort

With those 2 goals in mind, I started attempting to take down the goliath that is "How do we test starting up and entering keys into a text editor?"

My different attempts at this can be found in a gist, scattered throughout its history. Unfortunately, nothing worked great.

https://gist.github.com/twolfson/e24af2d3d5424540492f

Here are the rough/quick notes:

• Starting a child process to a curses application yielded non-useful errors for cbreak
• As a result, I went to a simpler script that outputs letters (no stdin worries for now)
• We could start that up as a child process
• Unfortunately, there's no EOF. As a result, when we used subprocess.PIPE with stdout.read()/stderr.read(), the Python test.py would hang unless a byte count is entered (e.g. stdout.read(200))
• Then, I was enlightened by a blog post saying we could provide an in-memory file descriptor instead (StringIO in Python 2, should be BytesIO in Python 3)
• This kind of/sort of worked -- it would write out content if I had an time.sleep(1). However, if I used while 1 loop that looked for changes on the StringIO, it would never yield anything =(
• Then, I tried to see if we could dodge the subprocess entirely by pointing curses to a StringIO instead of sys.stdout
• Unfortunately, it wanted a file descriptor number which I didn't have
• As a result, I created a pty via pty.fork() but after getting that stood up, we were still getting output on stdout
• I wasn't sure if it was a problem in my setup or curses
• I really wanted that to work
• Then, I tried overwriting sys.stdout with a StringIO but curses also went around that
• I now believe curses ignore the stdout provided to it via curses.setupterm but am not sure why

I have a few more thoughts beyond that:

• See if there's any way to mock curses (similar to FakeRedis)
• See if there's any way to read state from curses (e.g. get the contents of what curses is showing)
• Find out what's really going wrong with pty/curses

END OF BRAINDUMP/RANT

Now we should both be on the same page. From here, I suggest we get at least some tests in place for API's which are never going to change. One very clear one:

• Config's JSON parser (which does comment filtering)
  • https://github.com/richrd/suplemon/blob/v0.1.40/suplemon/config.py#L99-L108
  • Although, I think this probably already exists as an open source module
  • I believe it's sometimes called JSON5
  • https://pypi.python.org/pypi/json5/0.2.4

After that, I suggest we start building our equivalent of sublime_api. Then, we can write tests for those since we know Sublime's API is quite solid.

[richrd]

Great to hear you've been testing testing and thanks for the detailed report! :+1:

On the braindump:

I agree with 1. and 2. These tests would definitely be good to have. Right now I've been rather busy so I'll have to dive deeper into this when I have proper time.

And wow, curses seems to be a big pain for testing :/. You seem to have tried many things, most of which I wouldn't have thought about. Respect for that!

Mocking curses is probably possible, but it would be hard to cover all the situations that might occur:

• Resizing the terminal
• Different terminals interpret some key combinations differently For instance Ctrl - Page Up and Ctrl - Page Down result in different input on different terminals. They are used to push line(s) up and down but they perform other operations on some terminals.
• You can't draw characters outside the terminal bounds
• On some terminals colors entirely or more than eight colors aren't supported
• Trying to input or display special characters on python 2 curses fails
• Some others problems that I don't remember (and probably more that I haven't seen yet)

At least it's possible to retrieve characters from the curses windows with window.inch (https://docs.python.org/3/library/curses.html#curses.window.inch). That might be helpful.

Other stuff: I hadn't heard about JSON5. That seems great and perfect for the config use case. On the other hand I would like to have minimal external dependencies and the possibility of installing Suplemon directly from a .tar.gz. Maybe JSON5 can be packaged within Suplemon...

Then about sublime_api: It would be great to have it implemented, but I'm afraid that many of it's features might be hard to implement or such that they can't be done in the terminal (will at least need fallbacks or mocks). One of my big concerns is that I've been a bit lazy and haven't implemented regions (at least yet). I feel they will require lots of work and testing. Another thing that would be nice is bracket matching, but I don't yet have much ideas on how to implement it.

But I do agree with you about the API. I'll need some time to work on the core first and break up the App class etc though.

Thanks again for your help and interest :)

[richrd]

Update:

I came across this: http://sublime-text-unofficial-documentation.readthedocs.org/en/latest/extensibility/syntaxdefs.html . One point there scares me a little in regard to implementing sublime_api:

Several editing features in Sublime Text make extensive use of this fine-grained contextual information.

[twolfson]

What makes you worried about it? Context is pretty useful with respect to any key stroke =/

[richrd]

Well, mostly the amount of work required to implement this "fine grained contextual information" in regard to the syntax parsing. I agree that it's important, but I'm afraid I might not have the time (or skills in the worst case) to get it done. For now Pygments has provided nice syntax highlighting, but I'm not sure how well it could be utilized in this context.

I'll need to look into that more and think about it.

[twolfson]

In Sublime, syntax parsing is typically handled by .tmLanguage files.

https://github.com/twolfson/sublime-files/blob/3083.0.0/Packages/ASP/ASP.tmLanguage

These are grammars designed by TextMate.

https://manual.macromates.com/en/language_grammars

We should be able to either find an existing library or use something that can interpret well defined grammars (e.g. Bison).

One hurdle that we should get over though is using external dependencies. I know you don't want to install anything via pip. However, there are options like what requests does where it keeps those external packages as part of its releases:

https://github.com/kennethreitz/requests/tree/v2.7.0/requests/packages

[richrd]

Oh, good to know how the grammars work. An existing lib would of course be great, I'll see if I can find a Python lib for that.

On dependencies: I'm not at all against external dependencies, but my goal is to have Suplemon work without them. It's fine if that means that some features won't work, as long as it's usable. Even though only Python 3.3 or higher is supported officially I personally (and unfortunately) need Suplemon to be usable even on Python 2.6 :(. I hope to get rid of this need in the future.

That said, I'll look for existing solutions for interpreting the syntax. I don't think I'll have the resources to do it myself. Lets see :+1:

[ghost]

For Config and everything related, uses only std lib: https://github.com/ssato/python-anyconfig#python-anyconfig

For Auto-Save Auto-Backup every N time, uses only std lib: https://github.com/dbader/schedule#usage

[richrd]

Now that I'm on the rewrite train, I have a terminal backend implementatin for abstracting away curses and the terminal (https://github.com/richrd/suplemon/tree/rewrite-dev/suplemon/backends). The only backend implementation right now is curses, but a mock backend should be easy to write for testing. Hopefully that would fix this https://gist.github.com/twolfson/e24af2d3d5424540492f#file-main-py-L14

[Consolatis]

You could always use an actual terminal emulator like tmux for tests:

window=$(tmux new-window -d -P -c "/path/to/suplemon/repo" -n suplemon-test);
tmux send-keys -t "$window" "python3 suplemon.py" ENTER;
sleep 1;
if tmux capture-pane -t "$window" -p -S 0 -E 0 | grep Suplemon >/dev/null; then
    printf "\x1b[32mSuplemon initialized\x1b[m\n";
    tmux send-keys -t "$window" c-q y;
    sleep 1;
else
    printf "\x1b[31mSuplemon init failed\x1b[m\n";
fi
tmux kill-window -t "$window";

Some simple example tests here: https://gist.github.com/Consolatis/72058e71cfba310bf1a6d8794e391f65#file-test_suplemon_tmux-sh

[richrd]

Thats a nice idea! The examples work perfectly for me. I wonder if there's a way to make travis run these. EDIT: Seems like it's possible, this looks helpful https://maori.geek.nz/testing-microservices-with-pmux-and-travisci-8d3c42ce995c

[Consolatis]

Not sure you need pmux for that, it could make writing the tests easier though. If the travis instance has tmux available it should also work to simply create a new tmux session.

I updated the example script to do just that, also included a test for opening files: https://gist.github.com/Consolatis/72058e71cfba310bf1a6d8794e391f65#file-test_suplemon_tmux-sh

If the script exists inside the repo it could figure out the repo path automatically.

[richrd]

Yeah, it looks like pmux isn't necessary. I've been quite busy with other things lately, but when I get some spare time I'll try to add a basic run test to travis with this method. After that's done more tests can be added.

EDIT: This will be a lot easier in the future, when the rewrite is working, since it abstracts the curses IO completely.

[richrd]

Closing for now. I aim to add tests to the rewrite branch.