MarlinFirmware / Marlin

Marlin is an optimized firmware for RepRap 3D printers based on the Arduino platform. Many commercial 3D printers come with Marlin installed. Check with your vendor if you need source code for your specific machine.
https://marlinfw.org
GNU General Public License v3.0
15.94k stars 19.08k forks source link

🧑‍💻Improve make tests-all-local #27040

Closed sjasonsmith closed 2 weeks ago

sjasonsmith commented 2 weeks ago

Description

This resolves several issues encountered when running make tests-all-local. These were found when operating inside Windows Subsystem for Linux, but likely impacted other development environments as well:

  1. Tests continued after failing a TEST_TARGET. This made it difficult to notice when a failure occurred. By adding a test for status and an exit(1) call on failure, the test sequence now stops as soon as a failure occurs.

  2. PlatformIO frequently failed to transition from one test target to the next, with errors indicating missing frameworks. This seems to be resolved by adding a 5 second delay when transitioning from one TEST_TARGET to the next.

  3. Test the local Python to ensure pyyaml is available. If not, instruct user how to install it. I chose not to install it automatically, since this is the user's default Python and not a virtual environment.

  4. Fix broken file reference, and do not require Python files to be made executable to work.

Requirements

Benefits

Simplified running compilation tests locally to test changes outside of a PR.

sjasonsmith commented 2 weeks ago

@thisiskeithb would you be able to test this on macOS to make sure I haven't broken your workflow while fixing my own?

sjasonsmith commented 2 weeks ago

There are apparently some more changes I need to make. I think I edited some other files that I forgot to check in.

thisiskeithb commented 2 weeks ago

I've not had good luck with running most of Marlin's scripts, so I usually send them off to GitHub and let CI handled them.

I still have the same Permission denied error when trying to run make tests-single-local / make tests-all-local unless I run chmod +x get_test_targets.py before they'll attempt to run. Could all of these scripts be set to executable by default like we do for files in Marlin/buildroot/tests/? Or somehow automate it? Switching branches / testing out all kind of forks makes this a pain.

Once I make get_test_targets.py executable and run make tests-single-local TEST_TARGET=FYSETC_F6, it has several macros.h warnings, but works:

Passed
All tests completed successfully

Running make tests-all-local, it fails with a new error (same error I get before this PR):

Traceback (most recent call last):
  File "/Marlin/buildroot/share/scripts/get_test_targets.py", line 11, in <module>
    with open(yaml_file) as f:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '.github/workflows/test-builds.yml'

I documented some of these issues & potential fixes in https://github.com/MarlinFirmware/Marlin/pull/26904#issuecomment-2027326052, but it would be really handy if there was a set of instructions somewhere that had the list of prerequisites on what all needs to be installed / set to executable or automate it so this process is easier.

Tested on macOS Monterey

thisiskeithb commented 2 weeks ago

After your latest commits... Success!

Passed
All tests completed successfully

Tested on macOS Monterey

sjasonsmith commented 2 weeks ago

After your latest commits... Success!

Passed
All tests completed successfully

Tested on macOS Monterey

Thanks for testing it out, and I'm glad we could fix the macOS failure!