search

MarlinFirmware / Marlin

Marlin is an optimized firmware for RepRap 3D printers based on the Arduino platform. Many commercial 3D printers come with Marlin installed. Check with your vendor if you need source code for your specific machine.
https://marlinfw.org
GNU General Public License v3.0
16.05k stars 19.15k forks source link

[FR] No unit tests? #8051

Closed fmckeogh closed 6 years ago

fmckeogh commented 6 years ago

Is there any reason in particular there aren't any unit tests? I understand embedded projects are the best candidates for unit tests, but it is certainly possible using one board to run code with a unit test flag set, and another to measure output and check for discrepancies.

Roxy-3D commented 6 years ago

What do you envision? I think this has been discussed before and the problem is it would be more work to define the correct behavior than the benefit of having automated testing.

fmckeogh commented 6 years ago

I was planning on starting to develop a system for doing automated unit tests, with two boards hooked up to a server that polls for new PRs and then comments the results of the tests, but I wanted to know if it would be worth the time and effort.

I wouldn't necessarily start with documenting all correct behaviour, but instead with simple tests, such as given a sample GCode file will this pull request give the correct motor output, and expanding to more complex tests later on.

I'm new to the 3D printing community and want to contribute, but if it's been discussed and deemed unnecessary then I have my answer. :)

Roxy-3D commented 6 years ago

I'm new to the 3D printing community and want to contribute, but if it's been discussed and deemed unnecessary then I have my answer. :)

Unnecessary is the wrong word. But part of the problem is the code (and behavior) is evolving so fast, even the documentation doesn't stay up to date. It would be very difficult to keep the test cases current and up to date.

fmckeogh commented 6 years ago

Okay, I understand. Thanks!

oysteinkrog commented 6 years ago

I actually disagree, having tests that verify that every g-code does the correct thing would be awesome.

Tannoo commented 6 years ago

How can an automated system tell if any g-code worked correctly? Wouldn't that require someone to watch the results? Otherwise, there is Travis.

fmckeogh commented 6 years ago

A test board would be hooked up to a server and its motor outputs connected to another monitoring board. The outputs would be checked to match with known working outputs for a certain file.

Travis only checks software tests and compilation issues.

Roxy-3D commented 6 years ago

and its motor outputs connected to another monitoring board.

And the LCD signals. And the Power FET's to heat the nozzles and bed. And then there needs to be fake readings to simulate the various voltages the thermistors will provide to the boards heat system. And fake inputs to simulate the end stop switches. As well as fake signals to pretend somebody is turning the encoder wheel and selecting different menu options.

And all of these fake signals need to be coded with reasonable behavior that can simulate the real life hardware. And then of course... In order to test the failure paths in the firmware, this same simulation code needs to be coded to provide different types of failure (that are going to be tested for proper behavior).

And what exact setup do we do this for? Do we do it for every possible setup of end switches and thermistors and nozzles and single and dual stepper motors on an axis? Do we do it for Cartesian and Delta and Scara?

I actually disagree, having tests that verify that every g-code does the correct thing would be awesome.

Nobody said it wouldn't be awesome. I just don't know how to do it.

bobc commented 6 years ago

The company I work for make motor drives, and we test all hardware with a real load before shipping. This is to verify manufacturing quality, not software, but the principle is similar. We spend a lot of time and money building the test equipment, that is partly because it needs to be very robust in the production, they can't afford downtime. However, even then we only test a subset, and the motor drive is relatively simple, it just needs to run the motor up to a known speed. If the motor reaches speed in a certain time, that is a test pass.

In the 3d printer case, you would need to at least provide a load for the motor outputs, if not actual motors. The second board would need to have extra ADC inputs to monitor the coil voltage. You would get megabytes of data. You could then compare that data against a reference set. However, you need to allow for variations due to noise and differences in timing, so you have to be pretty fuzzy with the matching. When a mismatch occurs, you have to trawl through a lot of data to figure out if it is a real error or a false positive.

If you just took digital data, either step/dir signals, or even measure position from the printer, you have a similar problem - you have to compare huge data fingerprints. It is only useful for regression tests, you have to have a known good reference to start with. If the software changes, you have to re-capture all the data.

I've never seen any hardware project attempt unit tests like this. It is way harder than just running a couple of boards back to back. I think that it is a huge amount of work for something that is of not much value.

drmulligan commented 5 years ago

What do you envision? I think this has been discussed before and the problem is it would be more work to define the correct behavior than the benefit of having automated testing.

Sorry for the necrocomment but I cannot find anywhere else to post this. For a legacy project, which this is from the point of view of implementing unit tests, you start with asserting current behaviour as correct. At first it becomes a change detector allowing devs to decide what the actual correct behaviour is as tests fail due to changing behaviour. At that time they will have the perspective to be able to choose the best assertions. I've used this approach before at work and it worked wonderfully. We had the safety net to know when our changes had unintended consequences and gave us the foundation to continue forward. Testing isn't easy. It is most often harder to do than to write the feature. It doesn't take long for the benefits to outweigh the effort.

marcus-sonestedt commented 4 years ago

Seems more feasible now with the HAL than in older versions. The hardware could be mocked and compute that moves are being done. I've done similar things by compiling my Arduino code as a win exe and also reading/mocking output pins and simulate real hardware.

Agree with @drmulligan that just verfiying that current behaviour is preserved is a good start for easing confidence.

Sadly, current projects I'm on use old forked versions, so I can't constibute tests on the current version, and these old don't have the HAL and are thus a pain to inject mocking into.

Awesome project nevertheless! Much appreciated!

drmulligan commented 4 years ago

We would need to attract someone with experience unit testing a large multi target embedded project in order to kick this off.
One major advantage that can come of isolating code, as in using the extract method refactoring, to test might be to achieve a better level of decoupling from the HAL. This might be too much to hope for as it's an advantage of test driven development more than legacy testing.

github-actions[bot] commented 4 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.