search

MarlinFirmware / Marlin

Marlin is an optimized firmware for RepRap 3D printers based on the Arduino platform. Many commercial 3D printers come with Marlin installed. Check with your vendor if you need source code for your specific machine.
https://marlinfw.org
GNU General Public License v3.0
16.21k stars 19.22k forks source link

RTOS under Marlin discussion #10549

Closed Roxy-3D closed 4 years ago

Roxy-3D commented 6 years ago

A number of people have suggested having an RTOS (Real Time Operating System) under the Marlin firmware would help in a number of ways. We would be able to isolate some functionality into separate tasks and prioritize that logic’s use of resources.

One obvious example would be the ADC (Analog to Digital Conversion) of the nozzle and bed temperatures. Breaking this out of the main logic path would be a good thing. This task could be prioritized at a very low priority and just consume short bursts of CPU cycles that nothing else can use. There are a few questions I would like to discuss here.

I’ve started playing with the idea of putting an RTOS under Marlin. I’ve been using FreeRTOS for the work. Amazingly, I’m making good progress doing that using an AtMega-2560 board. If it can be made to work in that environment, for sure it can be made to work with the 32-bit platforms. The big problem is it will consume some resources that are already scarce on the 8-bit AVR. It will make it that much more difficult to add powerful new features to the AVR’s.

Does it even make sense to do this for the 8-bit AVR’s ? The limited resources of the 8-bit AVR’s causes one last question to be asked:

As a community, we need consensus to the direction we head. Everybody’s thoughts are welcome.

GMagician commented 6 years ago

I'm using a Mega2560 and what I'll say is going against me but I think that 8 bits has done its season and new improvments need to be focused on 32 platforms.

autonumous commented 6 years ago

@GMagician I have to agree with you.

but i like the idea

Roxy-3D commented 6 years ago

I'm using a Mega2560 and what I'll say is going against me but I think that 8 bits has done its season and new improvments need to be focused on 32 platforms.

I strongly believe that at some point in the future, we are going to have to fork the code base. I'm not even saying we will have to leave the 8-bit stuff behind. But it is very possible at some point in the future we can implement things on the 32-bit platforms that are impossible to implement on the 8-bit platforms. In that case, it may be the two code bases still track each other 'pretty much' but 8-bit doesn't have everything the 32-bit machines have.

I'm one of the biggest proponents of 'not forking the code bases' until we have no choice. And in fact, that is why I'm messing around with this on the AtMega-2560. If that can be made to work, we can have an RTOS under Marlin on the 32-bit processors without forking the code base.

It may be an RTOS actually helps the AVR 8-bit platforms. The reason is it would let us change the interface to the LCD code. In an RTOS environment, we could send messages to the LCD code on what to do and not repaint the display in critical, time sensitive loops. If something else of higher priority needs the CPU... The LCD can wait a 1/4 second until the CPU has nothing more important to do. Right now, we jump through all kinds of hoops to address this issue. Like complex (and ugly) logic to repaint only a portion of the LCD screen on each pass through the idle loop.

I'm using a Mega2560 and what I'll say is going against me but...

I hear you! And the worst case scenario is we freeze the feature set of the 8-bit world. (I don't think we need to do that.) And in that worst case world, the Marlin feature set is pretty robust.

32-bit hardware is getting so cheap... And development time is so precious... In a way, it is a shame to slow down the Marlin feature set development because the platform (hardware and RTOS) are not there to make the developer's job easier. (I'm not really in this camp... But it is a fair perspective to discuss.)

ejtagle commented 6 years ago

The only reason for not using an RTOS is SRAM consumption. Each thread requires a separate stack, and under AVR SRAM could be the problem. Regarding all the other things, a RTOS WILL simplify Marlin architecture, and will undoubtly reduce CPU cycles consumption and will help alot in managing resources. The main point of a RTOS is that tasks only run when needed, and there is task priorization. So, instead of polling for flags and implementing a lot of finite state machines, as it is done right now, a simple task will do the same with much better module isolation On the "bad" side, as you start using threads, you will have to start worrying about thread synchronization, race conditions, mutexes, etc.

ManuelMcLure commented 6 years ago

Considering that a 72MHz STM32 Cortex-M3 with 512K flash is almost the same price as a 16MHz ATMega 2560, I expect that the prices for 32-bit printer controller boards will fall precipitously in the near future. I don't think it makes a lot of sense to spend a lot of effort on an RTOS for 8-bit boards. And I'm another person who's currently running their printer on a Mega 2560.

Roxy-3D commented 6 years ago

The only reason for not using an RTOS is SRAM consumption. Each thread requires a separate stack, and under AVR SRAM could be the problem.

Agreed! If we did put an RTOS under Marlin on the 8-bit platforms, we can't go crazy defining lots of little tasks that all have stack space hard allocated for them. That is one good thing about the current Marlin code base. Functions can create local storage, and when they are done running, the stack shrinks back to where it was before the function was entered. That doesn't happen with tasks.

With that said... With UBL running a 10 x 10 mesh (with 100 floating point numbers dedicated to holding the mesh data) I have 3100 bytes free on my machine with a 20x4 LCD Panel. I have 2752 bytes free on my machine with a Graphical LCD Panel. (And I turn on all the debug stuff and other things that cost memory.)

So... If we are strategic about it... And maybe find ways to keep the low priority tasks with very small stack sizes... And maybe combine a few low priority functions into the same task so they share a stack space... It might not be a 'Show Stopper' that keeps us from having a shared code base between the 8-bit and 32-bit processors.

thinkyhead commented 6 years ago

👍 Upvote for the next-generation of Marlin (3.0?) to be built on an RTOS. And I also like the Klipper approach, offloading more of the work to the host so the firmware can be simplified and extend the useful life of your old AVR RAMPS.

Roxy-3D commented 6 years ago

Upvote for the next-generation of Marlin (3.0?) to be built on an RTOS.

We can't even get 1.1.8 closed out... 3.0 is probably 5 years from now. How about a 2.1 time frame?

thinkyhead commented 6 years ago

Since it would be a complete change of architecture, it would not be a descendent of 2.x. It would truly be a new firmware built completely from the ground up.

ejtagle commented 6 years ago

Something i don´t like about RTOS is their dynamic memory allocations. I have developed on several embedded systems, and when you start allocating memory, creating and destroying tasks, etc, eventually you end in a situation of memory fragmentation, where you have free memory but not in a contiguous block, so you cannot satisfy a relatively "big" request, and your system crashes. I found out that you must keep at least 10% or even more of free memory to reduce that risk Certainly, the static memory allocation system that Marlin uses is better for "non virtual memory equipped" MCUs. Yes, a raspPI has at least a memory paging mechanism, thus does not have this problem. But smaller MCUs do not have it...

Roxy-3D commented 6 years ago

Since it would be a complete change of architecture, it would not be a descendent of 2.x. It would truly be a new firmware built completely from the ground up.

No... I don't think this is true... We can slip it underneath Marlin and slowly do surgery on it to turn functionality into tasks. The first version would have FreeRTOS 'root kitted' under Marlin. And maybe with a few 'sample' extractions done. The ADC stuff would be trivial and simplify a bunch of stuff. And the LCD code would be pretty simple and beneficial to extract as a low priority task.

I don't think very many lines of code in Marlin change between v2.0.0 and v2.1.0. And maybe they don't happen more than 6 weeks apart in time. What ever v2.0.0 gets frozen at, it shouldn't be that hard (6 weeks) to finalize the RTOS work. v2.1.0 is identical feature set as v2.0.0, but it has an RTOS under it.

GMagician commented 6 years ago

@ejtagle memory fragmentation can't be solved also dotnet has this problem, and it works with a lot of memory (yes it also alloc/dealloc big chunks of it). Microsoft has done something to manage this but their solution can't be applied to critical systems for different reasons. I agree with both @Roxy-3D (do it sooner acting a surgery on current 2.0) and with @thinkyhead (can't go on with 2.x branch), but 2.x can replace perfectly 1.x so last one can be freezed after 1.1.9 is released. 2.0 can have a very short life turning quickly to 3.0.

ejtagle commented 6 years ago

Don´t take it wrong: I actually like using RTOS for complex project. I even wrote an small one myself a long time ago (it was a cooperative multitasking OS, as those have less problems when there are low resources available, such as RAM Memory fragmentation is a problem if available RAM is not much and more than 10 threads running at the same time... But i am not opposed to switching to an RTOS.

Roxy-3D commented 6 years ago

Well... Putting all religion aside... Suppose we did make v2.1 an RTOS version of v2.0??? That implies both the 32-bit and the 8-bit stuff is building with an RTOS in place. If somebody has a problem where v2.1 doesn't quite meet their expectations? Well... The answer is simple: v2.0 is the exact same functionality. Move to that until we resolve your issue on v2.1.1.

And if somebody has an AVR part that can withstand another 10KB of program memory... The RTOS is there for them.

Roxy-3D commented 6 years ago

Something i don´t like about RTOS is their dynamic memory allocations. I have developed on several embedded systems, and when you start allocating memory, creating and destroying tasks, etc, eventually you end in a situation of memory fragmentation, where you have free memory but not in a contiguous block, so you cannot satisfy a relatively "big" request, and your system crashes.

memory fragmentation can't be solved also dotnet has this problem, and it works with a lot of memory (yes it also alloc/dealloc big chunks of it).

Yes. But Marlin is a special case to all this. Right now, the only malloc() in Marlin is in the M100 memory corruption checker. It used to actually do a malloc() to find the heap. But (without looking) I think it is using sbreak to know where the free memory is.

And what I'm doing is starting some threads, and then I try to start Marlin with 8000 bytes of stack space. If the task fails to start, I try to create it with 7900 bytes of stack. Etc. Etc. What ends up happening is no matter how the user configure their printer... The Marlin task finds all the memory. We can reserve more malloc() memory for other tasks if that makes sense. But right now, the bulk of the logic is in the Marlin task and it doesn't do malloc()'s.

GMagician commented 6 years ago

@ejtagle, @Roxy-3D my purpose was only to put a note to memory managements, but I really like the RTOS idea and I'll do my best to help to go on, even if I have no all knowledge you have

ejtagle commented 6 years ago

RTOS is the way to go. But memory management and thread cost must be taken into accoumt from the start

thinkyhead commented 6 years ago

Suppose we did make v2.1 an RTOS version of v2.0??? That implies both the 32-bit and the 8-bit stuff is building with an RTOS in place.

I don't see an RTOS being at all possible on AVR. It depends so much on the speed of the processor to have the luxury of underlying task management concurrent with stepping at the required rates, especially when dealing with 32x and up micro-stepping. I believe it would have to be limited to the 32-bit architectures.

I'm not being religious when I say that the RTOS comes along with certain best practices that essentially tank all the C-like optimized code we've been writing so far. It would —as it were— "raise the bar" and call upon us to employ formal design patterns in a way that Marlin has eschewed up to now in favor of a very Imperative approach.

Grogyan commented 6 years ago

I would add that using an RTOS would also enable the potential for a simplified version of OctoPrint to be used concurrent. OnDevice slicing would still not be an option.

Roxy-3D commented 6 years ago

I'm continuing to make good progress. Memory is tight. Right now I think I'm fighting a memory corruption issue in settings._load(). I'm working to narrow down the problem.

Right now, I've got the main Marlin loop running as a high priority task. I'm waiting to do the surgery to cut out the LCD panel and other idle() task activity until I can get all the EEPROM data loaded.

I'm getting more and more convinced this can be done without any serious architectural changes. Which brings me to the reason for this post. Right now we have two branches running in parallel. We really need to get down to one main branch. But in the short run, maybe it makes sense to create a new bugfix-2.1.0 branch with the intent being it exactly tracks bugfix-2.0.0 except for the fact it has FreeRTOS slipped under it.

That would do a few things.

@thinkyhead Can you live with a 3rd branch as we work to get down to just one main branch?

Spawn32 commented 6 years ago

I feel so exited over this @Roxy-3D :) great work..

Roxy-3D commented 6 years ago

Still a lot to do... But I'm at the point where I believe it can be done and without architectural changes. It is keeping the build environment clean that is my biggest concern.

Oh! Incidentally... I'm doing all the debugging using a Max7219 8x8 LED Matrix. It runs as a separate task and gets fed the debug information via a message queue. It runs as a super low priority task that is just a little bit higher priority than the idle task. So... it is real easy to believe the RTOS is doing what it is supposed to do just because there would not be debug information showing up on the display otherwise.

thinkyhead commented 6 years ago

Can you live with a 3rd branch as we work to get down to just one main branch?

Sure. Make a branch to experiment with RTOS. Once it becomes stable it can merge with 2.x.

Roxy-3D commented 6 years ago

Actually... I'm doing that right now. I have it working in bugfix_1.1.x but that is the wrong place. I'm trying to get it moved over to bugfix_2.0.0 right now. You are going to be surprised how little the Marlin code changes.

Have you ordered one of those Max7219 LED Matix things off of eBay? You definitely want one for this purpose. It is $2.00 well spent! https://www.ebay.com/itm/MAX7219-Dot-Matrix-8x8-Led-Display-Module-MCU-Control-For-Arduino-Innovation/222709799381?epid=1672606113&hash=item33da89d9d5:g:QmgAAOSwbw1aATcc

thinkyhead commented 6 years ago

I will mos def get a Max7219. They make the printers look extra cool.

p3p commented 6 years ago

You are going to be surprised how little the Marlin code changes

RTOS is the way forward as more asynchronous features are added to Marlin, but without substantially changing the architecture of Marlin to event based systems (user interaction, display, communications, ect) all adding an RTOS underneath does is increase overhead. If the intention is to move Marlin 2.0 to an RTOS base we have a lot of work to do to planning the subsystems and the interactions between them if we want to make it more than just a 'feature'.

I did experiment with having the LPC176x platform running on an RTOS, all it did was start Marlin as the user-space application so it was a bit pointless, and things were complicated enough with attempting to get Arduino compatibility for libraries.

Roxy-3D commented 6 years ago

but without substantially changing the architecture of Marlin to event based systems (user interaction, display, communications, ect) all adding an RTOS underneath does is increase overhead.

This is a little bit similar to the effort 18 months ago to start turning things into objects. It should help us clean up certain things. And admittedly, there is some over head. Think about the graphical LCD update activity. That is very expensive and burns a lot of the CPU cycles. If that can be done as a very low priority item when nothing else important is happening, that is a big win. I'm actually thinking I can get good results going the opposite direction. I'm thinking initially we could run the LCD logic as a high priority task that sleeps very often in its timer driven loop so it doesn't consume the CPU for anything but very short bursts of time.

So with a few example surgeries showing benefit... It can continue like the conversion to an object based model did. One step at a time.

Any way... If it isn't a clear win, we can revert the changes pretty easily.

thinkyhead commented 6 years ago

If that can be done as a very low priority item when nothing else important is happening, that is a big win.

In fact, at this time display updates do get throttled whenever the planner needs priority.

ejtagle commented 6 years ago

Marlin right now implements a Pseudo RTOS (lets call it cooperative RTOS). Uses prioritized interrupts for high priority (Realtime) tasks, and an idle() callback for low priority tasks. The remaining things are run in a different task... ;)

Roxy-3D commented 6 years ago

If that can be done as a very low priority item when nothing else important is happening, that is a big win.

In fact, at this time display updates do get throttled whenever the planner needs priority.

lcd_update() limits how long it uses the CPU and does the screen update in phases. But if idle() gets called... lcd_update() gets called. Right?

Can you explain more about what you are thinking please?

thinkyhead commented 6 years ago

lcd_update() limits how long it uses the CPU and does the screen update in phases.

lcd_update used to block the main loop for all of those screen slices, and on deltas especially it often led to planner starvation and stuttering. So we made two important changes. First, the screen only updates one slice on each call to lcd_update. And second, if it looks like the planner is in danger of starvation (tracked using max_display_update_time and planner.block_buffer_runtime()) then it will skip the update altogether. However, it continues to process buttons so that clicks and wheel rotations won't get missed.

Roxy-3D commented 6 years ago

OK. Thank you. That is how I remembered it.

Roxy-3D commented 6 years ago

I have an initial version of Marlin bugfix-2.0.0 working with FreeRTOS under it. I've broken out some simple tasks to start the process of taking advantage of having a scheduler under our code.

I've asked ThinkyHead to look this over and make what ever changes need to be made: https://github.com/Roxy-3D/RTOS-bugfix-2.0.0-try-3

I need help getting it to build on the 32-bit platforms. But it currently can build code for the 8-bit AVR platforms. It boots up and does basic stuff for both Graphical LCD Panels and 20x4 LCD Panels.

It is possible this gets put in place as bugfix-2.1.0. The goal of the bugfix-2.1.0 branch would be functional equivalence to bugfix-2.0.0 except it has an RTOS under it.

alexxy commented 6 years ago

Btw are you going to use arduino + freertos? or aim is to port to freertos? Also freertos might have posix extensions (threads and others) and emulators that might help in debugging of threads

thinkyhead commented 6 years ago

@Roxy-3D — I've rebased your RTOS branch and pushed it up to https://github.com/MarlinFirmware/Marlin/tree/Marlin_RTOS if you want to grab an up-to-date version.

Roxy-3D commented 6 years ago

Btw are you going to use arduino + freertos? or aim is to port to freertos? Also freertos might have posix extensions (threads and others) and emulators that might help in debugging of threads

Both the bugfix_1.1.x branch and the bugfix_2.0.0 branch are working (limping along) with FreeRTOS.

@Roxy-3D — I've rebased your RTOS branch and pushed it up to https://github.com/MarlinFirmware/Marlin/tree/Marlin_RTOS if you want to grab an up-to-date version.

Thanks! I'll pull it down... Right now... The question I'm struggling with is what to do about the AVR processors. I have them working with FreeRTOS in the mix. But memory is tight. I'm worried it will limit what we can do in the future.

I'm wondering if it makes sense to have an AVR branch without an RTOS. And just put the RTOS under the 32-bit branch. It maybe this decision won't have to be made.... I am finding a few places to save memory. But like I said... Memory is tight.

thinkyhead commented 6 years ago

I think RTOS should be an option, but not the only one.

Roxy-3D commented 6 years ago

I think RTOS should be an option, but not the only one.

It would be possible to use #ifdef's to control that.

Roxy-3D commented 6 years ago

Post moved from elsewhere to here:

I was working to bring up 2.1.x on AVR using generic PlatformIO and so started making some PR submissions for what I thought was an obvious omission of leaving out the actual FreeRTOS library (https://github.com/MarlinFirmware/Marlin/pull/11270) but now that I am digging deeper, I think it may be more complicated than that.

Should I submit PRs against your "Marlin_RTOS" instead of the main one in the Marlin repo ? At least until things seem to be working better ? Or should we keep on the main branch where Scott manages merges so he can apply his massaging or recommended organizational improvements along the way ? I just don't want to cause Scott frustration when things get merged then breaks something as he tends to then unmerge then won't re-merge for what seems like forever. But the branch seems very much in flux. My goal was to see if I could get it to compile, and if so, begin working through all the Travis tests until it passes.

I wanted to know how you felt with the RTOS branch and if dev work should continue on things like your side branch, then you hand submitting PRs to main where Scott can then do his logistics work to cross T's and dot I's, or we just do it live on the main RTOS branch.

With that, can you briefly get me up to speed on the FreeRTOSconfig.h and "variants" issues you mentioned in the readme ? I'm not familiar with them, but I think I'm now wrestling with it:

Code: .piolibdeps/FreeRTOS_ID507/src/variantHooks.cpp: In function 'void initVariant()': .piolibdeps/FreeRTOS_ID507/src/variantHooks.cpp:50:2: error: 'USBDevice' was not declared in this scope USBDevice.attach(); ^ .piolibdeps/FreeRTOS_ID507/src/variantHooks.cpp: In function 'void vApplicationIdleHook()': .piolibdeps/FreeRTOS_ID507/src/variantHooks.cpp:73:6: error: 'serialEventRun' was not declared in this scope if (serialEventRun) serialEventRun(); ^ *** [.pioenvs/at90usb1286_dfu/lib5c8/FreeRTOS_ID507/variantHooks.cpp.o] Error 1 Cool idea trying to get this off the ground sooner rather than never !

Please submit any changes to the actual RTOS branch. The 'Variant' files are for the purpose of adapting FreeRTOS to a given platform. There is a bug in the Variant files because it assumes everybody would want to have a hook to handle malloc() being out of memory. I did not want that. I wanted the initialization code to allocate sufficient memory to each thread... And then use malloc to find out how much memory the main Marlin task could have. The problem is, when you turn off the variant hooks, there isn't a routine to handle a return from the scheduler. (The scheduler should not ever return!!!!)

I have attached the changes required to FreeRTOS to be used to build for AVR's. A good first Pull Request would be to get these pulled into the Marlin branch some how. The changes to configuration are not drastic. But by turning off the malloc() failures going to a hook... It breaks the variant.cpp file. It is patched in the .ZIP file at the end of this message. Here is the change:

void initVariant(void) __attribute__ ((OS_main));
void initVariant(void)
{
#if defined(USBCON)
    USBDevice.attach();
#endif

    setup();        // the normal Arduino setup() function is run here.

    vTaskStartScheduler(); // initialise and run the freeRTOS scheduler. Execution should never return here.

//  vApplicationMallocFailedHook(); // Probably we've failed trying to initialise heap for the scheduler. Let someone know.

DIE:    goto DIE;  // we should never get here!!!
}

I think the way things proceed is the Marlin RTOS thread does not stay in lock step with the main threads. But the places where the FreeRTOS stuff cuts into the code is fairly tame. I think instead of staying lock step... We periodically just merge the changes with the current Bugfix branch and call that the new, updated Marlin RTOS branch.

The problem is... Memory is tight on the AVR processors. It is working. But initially, it may make sense to not create extra threads unless they serve a very real purpose. And even then, try to add new features by controlling whether an RTOS is present or not with #ifdef's.

I wanted to know how you felt with the RTOS branch and if dev work should continue on things like your side branch, then you hand submitting PRs to main where Scott can then do his logistics work to cross T's and dot I's, or we just do it live on the main RTOS branch.

Just to be clear!!! I no longer have a side branch for RTOS activity. It is a real Marlin branch. It probably should be viewed as an 'experimental' branch because we don't know for sure we can do everything we want with AVR's. Let's target all 'improvements' and changes to the https://github.com/MarlinFirmware/Marlin/tree/Marlin_RTOS branch.

FreeRTOS-changes.zip

Sineos commented 6 years ago

I suppose this is known: https://github.com/dc42/RepRapFirmware/blob/dev/RTOS_notes.md

Roxy-3D commented 6 years ago

No... This is the first time I've seen that. And I think our ENTER and LEAVE Critical Section macros just turn off interrupts, so that portion of the document does not apply to us.

boelle commented 4 years ago

should we put this on a 3.0 milestone?

Roxy-3D commented 4 years ago

I don't think so... The reason is, we probably don't want to do this until the code is forked into a 32-bit version and an 8-bit version. And we don't want to do anything to force leaving the 8-bit processors behind. We are trying to keep everything supported by one code base.

boelle commented 4 years ago

yep and we dont know if that would happen before or after 3.0. So i agree

and calling it a feature request would be kind of wrong, so will just skip this one :-)

boelle commented 4 years ago

i will close this one as the debate has died out and it could as well be done on discord

Marlin Discord server. Join link: https://discord.gg/n5NJ59y

if truly wrong we can reopen of course

github-actions[bot] commented 4 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.