search

editorconfig / editorconfig-core-test

Testings for EditorConfig Core
http://editorconfig.org
BSD 2-Clause "Simplified" License
27 stars 22 forks source link

CLI tests with dummy core pass unexpectedly on Windows #38

Open dail8859 opened 4 years ago

dail8859 commented 4 years ago

The CLI tests only validate that the executable returned a 0 exit code, it does not validate the textual output of the command.

My app output something such as "dummy output" and it still passes the two version switch tests. I do not know much about CMake, however it looks like this function needs to accept parameters.

https://github.com/editorconfig/editorconfig-core-test/blob/48610d43b7455af12195473377f93c4ceea654f5/CMakeLists.txt#L105-L116

cxw42 commented 4 years ago

Thanks for the report! Would you please attach or point us to your test code, so we can reproduce the problem? The set_tests_properties line sets the regex to check in the output. Much appreciated!

dail8859 commented 4 years ago

I do not have any easy shareable example currently but it should not be hard to verify with any of the other cores.

Shouldn't new_ec_cli_test accept parameters like the other new_xxx functions?

cxw42 commented 4 years ago

It does! cmake_parse_arguments is the function that extracts the args. CMake syntax is very strange.

What project are you working on? You have opened a number of issues recently, and we could better assist if we knew the big picture.

In the mean time, I am closing this issue in the absence of a test case.

dail8859 commented 4 years ago

CMake syntax is very strange.

Indeed it is!

What project are you working on?

I'm writing an editorconfig core for Qt5

in the absence of a test case.

I'm using Windows for this test...

  1. mkdir mytest
  2. cd mytest
  3. git clone https://github.com/editorconfig/editorconfig-core-test
  4. Create CMakeLists.txt with this text: cmake_minimum_required(VERSION 3.17) project(mytest NONE) enable_testing() set(EDITORCONFIG_CMD ${CMAKE_CURRENT_SOURCE_DIR}/test.bat ) add_subdirectory(editorconfig-core-test) message( ${CMAKE_CURRENT_SOURCE_DIR} )
  5. Create test.bat with this text echo "nothing"
  6. cmake .
  7. ctest -C Release -R switch (no clue why I have to include the -C Release on my system but it complains if it is missing...the -R will just run the two version switch tests)
  8. The two tests incorrectly pass
cxw42 commented 4 years ago

https://github.com/editorconfig/editorconfig-qtcreator/issues/3

dail8859 commented 4 years ago

Yep. Very aware of that issue. Even commented in it :)

dail8859 commented 4 years ago

Oh never mind. You are probably just linking the issue and not pointing it out to me specifically :)