Is it possible to run tests in IEx shell and then analyze the results for neotest?

[scottming]

First of all, thank you very much for your work, which has given me a new experience in testing elixir in nvim. But I don't know if you have the same feeling that doing Unit Testing in elixir is much slower than in other languages, I know it's not caused by neotest or Lua.

The main reason is that every time you run a mix test you need to start the whole app, whereas if we just run the test in iex, it would be much faster.

Here is a project I generated with mix new elixir_demo, you can see that even the simplest project that is brand new takes 1.5 seconds to run each test using the command line, whereas with iex it just takes 0.015 seconds, which is a 100x improvement, and if we could do that, then developing elixir projects in Nvim would be a superb experience.

References:

• ExUnit tests from a module instead of command line/mix - Questions / Help - Elixir Programming Language Forum https://elixirforum.com/t/exunit-tests-from-a-module-instead-of-command-line-mix/22485
• TestIex — Easier Test Driven Development in Elixir | by Daniel Olshansky | Medium https://olshansky.medium.com/testiex-easier-test-driven-development-in-elixir-761cd1a9f11
• How to speed up the start of mix test? - Questions / Help - Elixir Programming Language Forum https://elixirforum.com/t/how-to-speed-up-the-start-of-mix-test/43498/15

[jfpedroza]

Hi. Thanks for reaching out. I think this is something worth exploring.

It would require starting an Elixir process, probably when the Neotest client is started and then in every run, communicate with it, sending it the test to run.

This is similar to using an Elixir test watcher, which at the moment doesn't work for me, but I'm investigating it. Test report should work for both.

Running this way should be an alternative and not replace the current approach.

Here is a project I generated with mix new elixir_demo, you can see that even the simplest project that is brand new takes 1.5 seconds to run each test using the command line.

It takes 0.5 seconds in my machine:

[scottming]

Thanks for your reply, Jhon. I have some experience with Elixir, but I am a newbie to Neovim, so can you give me more details?

Right now the module works fine and only needs to call two functions: start/0, test/1

> Test.start() # https://github.com/scottming/test_iex/blob/scott/lib/test_iex.ex#L13
> test_file = "test/file_test.exs"
> Test.test(test_file) #https://github.com/scottming/test_iex/blob/73686cec0b2d6a804f95a69abeec72ab67ac9d3c/lib/test_iex.ex#L42

So the next step is to add a genserver to the TestIex project to wrap TestIex module.

But I have a lot of questions about neovim?

1. How do I start the TestIex genserver at the same time as neotest start?
2. How do I connect to the genserver when after using the require("neotest").run.run() command?

Is there any code I can refer to regarding the above two questions? and which module do I need to modify?

[rcarriga]

Since conversation on this has already started here, I'll just comment on this thread.

How do I start the TestIex genserver at the same time as neotest start? How do I connect to the genserver when after using the require("neotest").run.run() command?

What you're looking to do is create a new strategy which will start a remote process and send the commands to it as tests are run. I'm happy to help with figuring out the process handling but I'm not sure how much the adapter will have to do.

I can imagine something like this being useful for other languages (e.g. as you said, python) but I'd say for a first attempt it can be written specifically for this use case which will hopefully make things easier.

Here's the flow:

1. User runs a test with iex strategy
2. neotest-elixir detects the iex strategy and provides whatever arguments are needed for it (similar to how nvim-dap integration works with debugger configuration)
3. The strategy is called and starts the process, runs the test and returns output.

Then when the user runs another test, the strategy has already started the process so it just sends the command to the existing process. If you would prefer to have the iex process start on startup rather than the first run, it might be more complex so I'd advise against it for simplicity but there are options such as exposing a public API from the strategy directly.

[scottming]

Hi, @rcarriga for this question,

but I'm not sure how much the adapter will have to do.

I spent some time exploring, which led me to discover that there may not be many changes to the neotest-elixir adapter.

I tried reusing these modules: formatter.ex, json_encoder.ex, then setting the neotest envs(NEOTEST_OUTPUT_DIR, NEOTEST_WRITE_DELAY), and making some changes to testiex, and then run it in iex shell.

Then I got the same result as require("neotest").run.run()， I got three result files.

But as for lua/nvim/neotest, how to start and hold the process with a strategy, I'm not capable enough and I don't know how to do it。

[jfpedroza]

Yes, most of the adapter can be reused. The change is in the build_spec function and adding the strategy, of course.

I would like to give this a go, if you don't mind.

[scottming]

If you can help, that would be great. Thank you, Jhon

[jfpedroza]

@scottming I made great progress in #14. It's mostly complete, but still needs testing and some tuning. Check it out.

You can use it by adding strategy = "iex" to require("neotest").run.run(), like so: lua require("neotest").run.run({vim.loop.cwd(), strategy = "iex"})

[jfpedroza]

Also note that the process is not IEx yet, so you can't run tests manually from the output buffer.

[scottming]

thank you so much, let me try it first, ~I will give you feedback as soon as I can~

@jfpedroza , left some comments on the PR#14.

[scottming]

Hello, @jfpedroza and @rcarriga , thank you for your great work on the neotest/neotest-elixir project.

And for this feature I have found a more stable and robust approach that is less error than looping and managing the processes by the adapter, I will test it in the next few days and if there are no problems I will share the code.

[scottming]

Hi, added #23

[jfpedroza]

Hey. I haven't been able to work on the other PR recently. I'll take a look at yours when I can.

[scottming]

Thanks. Take your time.