search

Optiboot / optiboot

Small and Fast Bootloader for Arduino and other Atmel AVR chips
Other
1.09k stars 399 forks source link

Need testing volunteers... #205

Open WestfW opened 7 years ago

WestfW commented 7 years ago

The list of chips that are nominally supposed to be supported by Optiboot seems to be growing quite long; much larger than I'm able to test myself. I'd like to build a network of people who are willing to test new Optiboot commits, especially on the less common chips and/or under less common circumstances. If you're able to do that, add a comment to this issue, along with which chips you'd be able to test, and any special requirements or features that hold particular interest (MCUSR pass-through, write-to-flash, etc.) Here's a list of chips I'd like to have "covered": ATmega8 ATmega88, ATmega88p ATmega168, ATmega168p, ATmega168pb ATmega328, ATmega328p ATmega328pb ATmega1280 ATmega2560 ATmega32, ATmega16 ATmega644p ATmega1284p (specific implementations thereof including Bobuino, Wildfire, etc) ATmega16m1, 32m1, 64m1 ATtiny84, 44, 24 ATtiny85 ATtiny1634 ATtiny861 ATtiny87, 167 ATtiny88 ATtiny162

SpenceKonde commented 7 years ago

ATtinyx7, 841, 1634, 828 (these are the chips that I currently include a version of Optiboot for in my core)

Is there a working Optiboot version for the tinies without hardware serial? I wasn't aware of one. If there is, I ought to include it in my core (which would mean I'd test on those too)

MCUdude commented 7 years ago

.. Are the ones that I have in my drawer and works with Optiboot. I can also test the write-to-flash feature using this example sketch.

per1234 commented 7 years ago

@MCUdude, do you know which branch and commit of majekw's fork your cores use? It's a bit anecdotal but I think it's worth noting that those modifications to optiboot have been in use by many people for almost a year now in some cases without finding any specific issues.

I'm happy to help out but the most of the parts I have are already covered by MCUdude:

I'd very much like to see MCUSR preserved and ATmega2560 support. I haven't found a use for write-to-flash yet but it seems like it will eventually come in handy.

MCUdude commented 7 years ago

@per1234 I've been using the Super master branch for all microcontrollers except for ATmega2560/2561, which I've used the m2560 branch. Not sure if the Super master contains all necesary code for the 2560/2561 stk500 workaround..

IIRC I also did some mods to the file to fix some simple errors for the older AVR generation. Re-did the makefile, added more devices to the pin_defs.h file, and minor tweaks to the optiboot.c file.

Nobody have reported any bootloader related issues that didn't turned out to be a HW issue 👍

SpenceKonde commented 7 years ago

Something worth noting w/regards to x7 series ATTiny: It's on that list above, but there's no way the code in the master branch of this repo will work on the x7. I don't think it'll even compile - the x7 doesn't have a normal UART, it has a LIN UART, with a bunch of fancypants features anyone not using LIN doesn't care about. All the registers are differently named and need to be treated slightly differently. I've done that in the version of optiboot in my core, but I never handed it back to westfw to integrate with his repo.

It sounds like we're suffering from fork-itis here, where the people maintaining cores (me and Hans, at least) are working with forks of the bootloader created when we started working on the core, and which we've never updated since.... That's certainly the case for me.

MCUdude commented 7 years ago

I'd love to have one Optiboot branch that supported all chips, with a proper, vanilla makefile (chip; not board/platform oriented) where it's possible to add new hardware without having to host a fork myself.

Personally, I like to store compiled hex files in separate folders, where the name of each hex file contains the build options. This way the boards.txt file gets a lot cleaner with multiple HW options.

WestfW commented 7 years ago

those modifications to optiboot have been in use by many people for almost a year now ... without finding any specific issues.

Well, yes, exactly. And if I manually merge the majekw and etc pull requests and make other changes, who can pop in and confirm "yep, it doesn't look like you broke anything!"

MCUdude commented 7 years ago

I suggest you create a branch where new devices and functionality can be merged, while keeping the current master branch untouched. If someone's doing a PR to the "new" branch, somebody, other than the person who's doing the PR, should test and verify it works and/or doesn't break anything.

When do you consider code "bug free" and "safe"? tested by 5 people? 50 people? 5000 people?

per1234 commented 7 years ago

I think a branch only causes more confusion and "fork-itis". Just do testing on each PR's fork, merge once tested, do a release when appropriate. People wanting the most stable code should be using a release, not the tip of the master branch. Tip is for beta testing.

per1234 commented 7 years ago

if I manually merge the majekw and etc pull requests and make other changes, who can pop in and confirm "yep, it doesn't look like you broke anything!"

Well there are three of us committed already but it would be good to get more coverage.

@awatterott or @elektor-labs any chance you might be willing to do testing of ATmega328PB?

@thomasonw would you be willing to do testing of ATmega64M1 and ATmega32M1?

thomasonw commented 7 years ago

@per1234 - Yes, I can test the ATmega64M1 (Has a port been rolled into the core optiboot yet?). I will have to look around for a board I can make up for the ATmega32M1 / 16M1 (and a CPU), but should be able to do a go/no-go test on it as well.

Is there an official test procedure we should be using? Or just do go/no-go testing a few times :-)

(Also a note, I will be traveling the next month. Have access to computer, but will not have any target test devices I can use. Expect to be back the end of May)

elektor-labs commented 7 years ago

@per1234 - yes, we can test on the 328PB, no problem.

awatterott commented 7 years ago

We can test ATmega328PB, ATmega328P, ATtiny841, ATtiny85. The testing on different MCUs would be a good starting point to integrate our auto baudrate detection into mainline Optiboot.

majekw commented 7 years ago

My collection of chips for test is not so impressive:

But returning to testing topic, I think Optiboot should go into continuous integration. I checked options and what I found:

  1. Simple 'it compile' should be quite easy achieved by Travis CI with PlatformIO, but Optiboot should set some reference Arduino version (with it's avr-gcc shipped) as compiling on every possible compiler could be difficult. Using Travis we can be sure at least every commit and pull request compile on every target.
  2. 'Matrix compile' - I was thinking about setting up whole environment for compilation on almost every possible version of avr-gcc shipped with Arduino. As a result I expected information not only about 'fail' or 'compile', but also about sizes of every target compared to for example last release or last commit in branch. It should be very useful to keep size within limits.
  3. Real CI - run Optiboot code (after every commit) inside simulator performing some functional tests (for example checking for timeout from watchdog, checking every possible reset cause and bootloader behavior etc). It's most difficult (all test must be written, new environment with simulator should be set up and configured), but I think it's currently the way such things are developed. Right now I checked some simulators and it seems really possible to do. But not easy.

Of course, testing on real hardware also should be done, but after some significant changes. It could be also useful to make some rules for pr (for example: new features always with example/test sketch, or at least with documentation) and giving feedback (maintainer job).

@WestfW: if you want to merge all this things, please set some timeline or order of accepting it, so everybody could resolve merge conflicts in correct time (some pr conflict between each other, so that's why order is important).

I could get this all CI topic, but I'm not very experienced with it, so if anybody have more skills, I would be glad to cooperate/learn from him/her.

per1234 commented 7 years ago

@majekw I think it's a great idea.

Testing every avr-gcc version would be cool but, though I'm usually a big proponent of backwards compatibility, I'm not convinced it's really necessary. I think it would be reasonable to just specify in the documentation that the latest version of the Arduino IDE should be used and test using the included version of avr-gcc. I think the idea of comparing the size to the last commit is excellent. So I would suggest a hybrid of 1 and 2. Of course 3 would be best.

I have two bootloader projects I'm involved with that I've been thinking of setting up CI for. It would be really wonderful if this could be done in a way that lends itself to reuse.

I've been working on something like this for use with Arduino sketches to test libraries or hardware packages with different versions of the Arduino IDE so I do have a little experience with Travis CI but only a very little. One thing I've done in that project is to count the number of warnings in the compiler output and display that in the report so that you can get a quick idea of whether any new warnings have been added by the commit.

majekw commented 7 years ago

@per1234 Your script is great and it almost covers 1. and 2. :+1: I'll try to modify it and test on my fork.

I'm afraid that for full matrix and size comparison Optiboot should have repository with hexs for all possible targets. It would be good to move them somewhere to not clutter sources. Good example how it could be done are in @MCUdude's cores.

per1234 commented 7 years ago

I'm glad if it can be of use. I wanted to generate a report of the compile size, warning count, etc. of every verification. The idea was to make it easy for me to see how the changes compare to the previous commit. Ideally the report would list the deltas but I haven't opened that can of worms yet. I just output it in tab separated form so I can throw it into a spreadsheet and use that for the comparisons. I guess the options for automatically generating the deltas in Travis CI would be either to have the previous report available online for it to access or to checkout the previous commit and redo each test. The latter would be easier but will double the compilation times since they all would need to be done twice.

The problem I ran into was if you use Travis CI in the normal way, where you set up a matrix, each test runs as a separate job in an isolated virtual machine. That's great because it prevents each test from interfering with the others. The problem is that I haven't found any way to generate a unified report for the entire build, rather than one for each job. So I decided to not use the matrix and just do all tests in one job.

The trouble with running all tests in one job is that some versions of the Arduino IDE interfere with each other. For example, when 1.6.2 is first run it moves its bundled hardware cores to the .arduino15 folder, where they get used by all other IDE versions after that. I'm experiencing other similar problems that I haven't tracked down the cause of yet. I'm very hampered by the fact that I have no prior experience with using Arduino IDE with Linux (I usually use Windows), with Arduino CLI, or Travis CI (which does some things behind the scenes). The most common failure mode of the Arduino IDE when run in this environment is to hang, so I get no useful feedback on the cause of the problem. I'm doing some tests on a Raspberry Pi now to see if I can at least eliminate the Travis CI variable from the equation but I'm also new to using the RPi so I don't know if it will be any better. I don't think interference between compilations would be an issue for the purposes of testing optiboot builds as there should be no need to use the Arduino IDE.

I suspect the way to get a unified report from isolated matrix tests would be to use the deployment feature of Travis CI. I would probably prefer to deploy to GitHub Pages. I believe this would result in a report file for each job being uploaded but I haven't come up with a plan for how to combine the reports.

I've done some more work that is in this branch: https://github.com/per1234/arduino-ci-script/tree/keeper-commits but it's not passing all tests yet. The rest of the branches other than master are just me flailing. I think the master branch will serve at least as a demonstration of what I'm talking about with creating a script that can be used for any project that generates a report of the results of the compilation for each target. Most of the other work is specific to the Arduino IDE and thus irrelevant to this project.

I think making the hex files generated by each CI build available would be a very good idea. That way we can all test using the same files to eliminate any variables that might be introduced by different build environments and this could also be used to automate releases. Travis CI does have the deployment feature that could be used for this: https://docs.travis-ci.com/user/deployment/ There are a slew of deployment provider options, none of which I've ever heard of. It makes sense to me to keep it on GitHub if possible. There is the option of deploying to GitHub pages: https://pages.github.com/ Which can be created as a branch of a repository. One thing to keep in mind is that Travis CI apparently does a force push (https://github.com/travis-ci/travis-ci/issues/7562) so I guess that means only the hex files from the most recent build can be available, definitely not ideal.

I'm happy to provide any assistance on the CI project, though it's debatable if I have much to offer. Do you think it would make sense to open a new issue specific to discussion of continuous integration of optiboot? We could move these comments over there to avoid getting the tester thread off track.

MCUdude commented 7 years ago

Anything new @WestfW? I think we're all ready to contribute, but you're the one in charge here 😉

majekw commented 7 years ago

Hello All.

I made some initial work with Travis-CI for Optiboot. At the beginning I focused only on compiling using avr-gcc shipped with Arduino as it's easiest to install by everyone on any platform, it's portable and it's easy to say to someone just 'install Arduino version X and everything should be fine'. So, my progress:

Results:

Worst thing is that there is no version which can compile properly all targets :-( Attiny84 is broken on all versions, eeprom support needs to be rewritten for newer versions. Safest version as for today is 1.6.9.

I will do more tests and then I'll prepare clean pull-request. Optiboot definitely needs it (same as real maintenance).

majekw commented 7 years ago

I made PR: https://github.com/Optiboot/optiboot/pull/212 with basic Travis-CI support. Until above issues are fixed, I think is enough to test against only one avr-tools version. I also opened issue dedicated to CI in Optiboot: https://github.com/Optiboot/optiboot/issues/213

termuxinator commented 5 years ago

Optiboot 8.0 works with bare bones ATtiny85, it should be supported, I don't have time to pull/merge but I could help someone configure it...