Raster issues: Smoothieware Jerkyness / did we need a new protocol, proper discussion

[ghost]

Paraphrasing from a discussion with @wolfmanjm in Oct:

@wolfmanjm said Maybe it is time to reconsider a special LW only command that sends a raster line at a time? This will require some thought and planning, we can't send binary data so it may need to be uuencoded or maybe a run length encoding that is ascii only. Not quite sure what we would do on the smoothie side but it should keep the queue full I would think. It will be a little like the way we handle segmented arcs and segmentation on deltas I think. Worth thinking about.

@openhardwarecoza said That would be a really cool and beneficial implementation! Marlin uses a Base64 implementation: https://github.com/TurnkeyTyranny/buildlog-lasercutter-marlin/blob/master/Marlin/Marlin_main.cpp#L1070 Lasersaur's protocol is documented here https://github.com/nortd/driveboardapp/blob/6409230a68bb653ea43a197ce9c091c9fef01188/docs/protocol.md#raster-data-mode

@wolfmanjm said As with all things Marlin I wouldn't touch it with a barge pole ;) (even if I had any idea what they were doing). The other protocol is also something I would't use. You want to be able to set the speed different on every pixel or run of pixels right? so neither of those allow that . Also I do not want to parse a raster in smoothie either. I was thinking more of a way to specify the same kind of gcode you currently send but in a more compact way so more lines get sent over the serial (or read from sdcard) at a time. then smoothie simply unpacks the commands and expands them into regular planner lines. So LW3 does all the work, and smoothie just unpacks them and handles them like it would regular gcode. Does that make any sense? The lines need to not exceed the USB serial buffer size, but should contain the same info that a sequence of gcode would so the start position, the laser power for each sequence and the feedrate for each sequence. something along those lines.

So with that said, heres some basic groundwork:

• Whatever we end up deciding on needs to be a new standard, at least with grbl @chamnit and Tinyg @giseburt

• All three share the same bottleneck: How best to get streaming raster data over serial

• I want to support all three in LW with the same data format! #yearofthelaser and #laserweb got lasers to go mainstream, so either you help us out to get a good compromise for all, or I may go as far as considering dropping your firmware from the supported list. This isnt the place for "but I can do it better" - all we we want here is a good enough, that works with everything

• At the moment Smoothie has the worst performance in this regard. TinyG status unknown at this point but because their laser code is early its NB that we get them joining the party (discussion) now while they still can. Grbl actually surpassed in 1.1 and is now the best performer on raster (with just plain old gcode by the way)

• Keep this one cival and result focussed. lets get this hashed out quickly. Its not to enrich someone specific, its about making lasering even more accesible and more useful, and in turn about making sure LW3 and the open work we do, ensures your various boards and firmwares, do what users want the best

[giseburt]

Maybe it's best to start with current state. Could you give an example of what's produced now?

G0X0Y0
G1S10X0.1F20000
S20X0.2
S30X0.3

Something like that? That's probably the most compact representation. It's already run-length, in that multiple pixels in a row that would have the same S could simply be merged into one X move, or whatever.

[tbfleming]

Here's what LW4 (lw.raster-to-gcode) produces:

G1 X0.15 Y0.15 S0.0000
S0.5529
X0.30
X0.60 S0.5673
X0.90 S0.5098
X1.20 S0.4654
X1.50 S0.3804

Maybe we could abuse comments?

G1 X0 Y0 S0
X100 (!S sdjDSleru93u4iuREd)      ; base-64 encoded S values

[giseburt]

In g2core we use "active comments," where we put JSON in a parenthesized comment. This is how we avoid the M-code nightmare of Marlin.

For example, we have two somewhat non-standard M-codes:

• M100 is "execute this JSON in sync with the gcode"
• M101 is "wait for this JSON to be true"

So, in a 3D printer, you would set the temperature (with the he1st code, meaning HEater 1 Set Temp) with M100:

M100 ({"he1st":210})

And then wait for it to be At Temp (he1at):

M101 ({"he1at":true})

[ghost]

raster-to-gcode can now export the bitmap height-map ;) https://lautr3k.github.io/lw.raster-to-gcode/dist/example/

[chamnit]

Before we start going down this path, I'd like to just add this.

Personally, I don't think g-code itself is the problem. How the program is received is just as big of a part, and there are a lot of ways to improve the streaming process independently of g-code and LW. For example, just eliminate streaming during a job by using on-board storage. Problem solved. There are also other strategies like making the response time from the controller shorter and executing the raster motions more efficiently.

Generally, I'm a proponent of not bastardizing the g-code standard, because it remains a universal interface for machine control. The main reason it gets bastardized is due to something undefined in the standard. Marlin is guilty on many fronts of this, but it was necessary to control all of the additional stuff on 3d printers.

If we start going down the path of defining new g-codes. I'd like to lay out some basic rules:

• Make a complete description of what the g-code does. There should be no room for any mis-implementation or mis-understanding.
• Complete error-checking definitions. This critical for lasers in particular. You don't want some 3rd-party, poorly-written CAM generator to mistakenly make a command error that can cause a safety hazard or fire.
• It should account for every other modal state possible, like incremental and absolute modes.
• The laser power values should be floating point and not restricted to an 8-bit integer. Otherwise, this can limit usage in the future.
• If existing g-code command letters are used (L P Q), they shouldn't overlap with any other potential future applications. For example, A,B,C,U,V,W are related to other axes and should be excluded from use. This rule doesn't really leave much to use.
• This should be agreed upon by everyone.

Personally, I'd like to avoid having to invent something totally new, like Base64 compression schemes (even though it's a neat idea) or XML/JSON commands. I'd prefer to fit it within the standard, but, AFAIK, there is no way to pass a vector in g-code. So, I may be in the minority on this one.

[arthurwolf]

I really don't think we should be creating a new format here. Bitmap ( BMP ) has been around for decades, it's very compact, very simple, easy to parse from a microcontroller, you can open the generated product with any image editor which is very nice for debugging, and there is heaps of code for it all around. Let's just stream that. Let each microcontroller do what it wants with that afterwards. We just need the host to send bitmap, and the microcontroller to have a way to say when it's ok to send more.

On Laserweb's side, I'm pretty sure skarab would be able to add that in just a few hours by googling "javascript bitmap". Because we need to go back and forth and stream it that way, we just need to mirror every even line, that's not a problem.

It's simple, it's bandwidth and processing-power efficient, it's already been around for ages. We even already have code for it ( for example : https://github.com/logxen/Smoothie/blob/laser_engrave/src/modules/tools/laser_engrave/LaserEngrave.cpp#L324 )

8-bit bitmap is likely good enough for now, but if we want to move beyond that later, we can, there are existing 16-bit standards too.

Then all we need to do, is get the ability to stream binary ( where we now only use ascii ), and bob's your uncle. I'm strongly against base64, it's just a way around using binary, but it's very wasteful. Let's do this right.

Once we have that working, all we need to do is agree on the Gcodes that "setup" a job before it starts ( size, speed, etc ), and the gcode that actually starts binary streaming mode. And the host needs to learn to not send anything but the binary data while it's streaming, which isn't difficult.

This is I think the optimal implementation here, it is very easy on all fronts, except for one hard thing, which is getting our communication systems to learn to handle binary. I think we can do it.

My 2 cents.

On Tue, Dec 20, 2016 at 5:40 PM, Sonny Jeon notifications@github.com wrote:

Before we start going down this path, I'd like to just add this.

[image: standards] https://cloud.githubusercontent.com/assets/948029/21358645/950d2958-c696-11e6-93e1-cd58c43b9419.png

Personally, I don't think g-code itself is the problem. How the program is received is just as big of a part, and there are a lot of ways to improve the streaming process independently of g-code and LW. For example, just eliminate streaming during a job by using on-board storage. Problem solved. There are also other strategies like making the response time from the controller shorter and executing the raster motions more efficiently.

Generally, I'm a proponent of not bastardizing the g-code standard, because it remains a universal interface for machine control. The main reason it gets bastardized is due to something undefined in the standard. Marlin is guilty on many fronts of this, but it was necessary to control all of the additional stuff on 3d printers.

If we start going down the path of defining new g-codes. I'd like to lay out some basic rules:

• Make a complete description of what the g-code does. There should be no room for any mis-implementation or mis-understanding.
• Complete error-checking definitions. This critical for lasers in particular. You don't want some 3rd-party, poorly-written CAM generator to mistakenly make a command error that can cause a safety hazard or fire.
• It should account for every other modal state possible, like incremental and absolute modes.
• The laser power values should be floating point and not restricted to an 8-bit integer. Otherwise, this can limit usage in the future.
• If existing g-code command letters are used (L P Q), they shouldn't overlap with any other potential future applications. For example, A,B,C,U,V,W are related to other axes and should be excluded from use. This rule doesn't really leave much to use.
• This should be agreed upon by everyone.

Personally, I'd like to avoid having to invent something totally new, like Base64 compression schemes (even though it's a neat idea) or XML/JSON commands. I'd prefer to fit it within the standard, but, AFAIK, there is no way to pass a vector in g-code. So, I may be in the minority on this one.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268292098, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGpFRdXF0LQgfaxU2EWx_rky_7d3WPiks5rKATtgaJpZM4LR0oN .

-- Courage et bonne humeur.

[tbfleming]

@arthurwolf does it support mixing raster and vector in the same file like gcode does?

[arthurwolf]

This would use the same communication channel as Gcode, everything that wouldn't be the bitmap data would be gcode, so yes, sure.

G1 X10 ( vector engrave something ) M1234 S100 ( set speed for binary raster engrave ) M12345 S123345 ( next 123345 bytes are raster data )

G1 Y10 ( vector engrave something ) **obviously** completely random gcodes here, not making a proposal :) On Tue, Dec 20, 2016 at 5:56 PM, Todd Fleming wrote: > @arthurwolf does it support mixing raster > and vector in the same file like gcode does? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , > or mute the thread > > . > -- Courage et bonne humeur.

[arthurwolf]

Writing bitmap data in JS ( or any language is very easy ), I found this after 5 seconds of googling : https://gist.github.com/vukicevic/8112515

On Tue, Dec 20, 2016 at 5:59 PM, Arthur Wolf wolf.arthur@gmail.com wrote:

This would use the same communication channel as Gcode, everything that wouldn't be the bitmap data would be gcode, so yes, sure.

G1 X10 ( vector engrave something ) M1234 S100 ( set speed for binary raster engrave ) M12345 S123345 ( next 123345 bytes are raster data )

G1 Y10 ( vector engrave something ) **obviously** completely random gcodes here, not making a proposal :) On Tue, Dec 20, 2016 at 5:56 PM, Todd Fleming wrote: > @arthurwolf does it support mixing > raster and vector in the same file like gcode does? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , > or mute the thread > > . > -- Courage et bonne humeur.

-- Courage et bonne humeur.

[giseburt]

I agree with @chamnit that we should try to stick to core GCode if possible. Only resorting to JSON or such when it's determined that there's a clear advantage or need.

I don't think binary is a good idea, unless you're willing to give up the ability to adjust the stream live, such as feed hold, adjusting laser power or feed rate, etc.

I guess we really should determine what data throughput is needed. At a given feedrate, say 1m/s, what's the rate we need to send commands? What's the maximum reasonable pixel resolution we wish to support?

Once we determine how fast we need to send commands, then we can make informed decisions on how to transmit them in a way they can make it through the pipes we're using (serial, USB, etc).

[arthurwolf]

On Tue, Dec 20, 2016 at 6:04 PM, Rob Giseburt notifications@github.com wrote:

I agree with @chamnit https://github.com/chamnit that we should try to stick to core GCode if possible. Only resorting to JSON or such when it's determined that there's a clear advantage or need.

I don't think binary is a good idea, unless you're willing to give up the ability to adjust the stream live, such as feed hold, adjusting laser power or feed rate, etc.

We can sacrifice one value out of 256 ( greyscale being from 1 to 255 instead of 0 to 255 for example, I'm talking scaling not exclusion, the bitmap generator taking care of that ) and have that byte mean "return to gcode mode until next \n". However if we have to choose between fast data transfer and the ability to adjust speeds on the fly, I'm not even sure we want to choose that ability ( though as I said if we save a byte to mean "stop" we don't have to make that choice ). Transfer speed is really something a lot of people are complaining about, Gcode really isn't adequate here, it's much much too wasteful. Even binary bitmap engraving is likely going to be a bottleneck that will leave us quite a bit bellow how fast we could otherwise go.

I guess we really should determine what data throughput is needed. At a given feedrate, say 1m/s, what's the rate we need to send commands? What's the maximum reasonable pixel resolution we wish to support?

I don't believe there is one. Engraving can go really fast.

Once we determine how fast we need to send commands, then we can make informed decisions on how to transmit them in a way they can make it through the pipes we're using (serial, USB, etc).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268298361, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGpFawdBuw3DLQxdP6yaGWBGNSqsiL1ks5rKAqBgaJpZM4LR0oN .

-- Courage et bonne humeur.

[arthurwolf]

And to be clear on why I don't think we need the ability to adjust speed or power, it is because it's likely even though in some circumstances it's not supposed to, it will probably mess up the picture's quality to adjust it mid-way ( as in : top and bottom "look" different )

On Tue, Dec 20, 2016 at 6:10 PM, Arthur Wolf wolf.arthur@gmail.com wrote:

On Tue, Dec 20, 2016 at 6:04 PM, Rob Giseburt notifications@github.com wrote:

I agree with @chamnit https://github.com/chamnit that we should try to stick to core GCode if possible. Only resorting to JSON or such when it's determined that there's a clear advantage or need.

I don't think binary is a good idea, unless you're willing to give up the ability to adjust the stream live, such as feed hold, adjusting laser power or feed rate, etc.

We can sacrifice one value out of 256 ( greyscale being from 1 to 255 instead of 0 to 255 for example, I'm talking scaling not exclusion, the bitmap generator taking care of that ) and have that byte mean "return to gcode mode until next \n". However if we have to choose between fast data transfer and the ability to adjust speeds on the fly, I'm not even sure we want to choose that ability ( though as I said if we save a byte to mean "stop" we don't have to make that choice ). Transfer speed is really something a lot of people are complaining about, Gcode really isn't adequate here, it's much much too wasteful. Even binary bitmap engraving is likely going to be a bottleneck that will leave us quite a bit bellow how fast we could otherwise go.

I guess we really should determine what data throughput is needed. At a given feedrate, say 1m/s, what's the rate we need to send commands? What's the maximum reasonable pixel resolution we wish to support?

I don't believe there is one. Engraving can go really fast.

Once we determine how fast we need to send commands, then we can make informed decisions on how to transmit them in a way they can make it through the pipes we're using (serial, USB, etc).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268298361, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGpFawdBuw3DLQxdP6yaGWBGNSqsiL1ks5rKAqBgaJpZM4LR0oN .

-- Courage et bonne humeur.

-- Courage et bonne humeur.

[chamnit]

Serial @ 115200 baud can send over 1,100 lines with 10 characters each. Or 1,100 pixels/second. That's pretty good for a "slow" serial interface. Suppose we were able to cut this in half by creating a g-code to place a laser power vector with a raster pass, this would double potential performance to 2,200 pixels/second. A modest improvement.

Let's look at a native USB interface at a 12Mbit/s or 480 Mbit/s, 10 characters per line would theoretically give 11,000+ pixel/second and 468000 pixels/second, respectively. At least an order of magnitude better. So, bandwidth for streaming a g-code program is not a problem. It's a problem with the streaming interface or controller. Improvements must be made there.

@arthurwolf : I agree that the ability to dynamically adjust speed and power (overrides) is not as important during a raster job, but it's still important for everything else. I would not exclude a particular feature from this conversation, because there may be very interesting use-cases in the future that may prove very groundbreaking.

[arthurwolf]

I think there's a confusion here between having a protocol to stream data for raster engraving as fast as possible, and a more general conversation about Gcode ( unless I don't understand what's going on ) I think we should stick to trying to find the best method to transmit data for raster engraving ( and I don't think Gcode is it )

We already have three possible transfer methods ( UART, USB/Serial, and Ethernet/Wifi, others might come later ), all with many possible different speeds in different setups. I think to make as many users happy as possible we should try to make it so no matter how fast their communication method is, they get the best engraving speed possible. Engraving jobs are long, the faster we can go the happier everybody is. And technically, transfer speeds are currently the bottleneck for many users. I believe that's what this conversation was started about.

Using the current Gcode standard, using modal Gcodes and relative mode ( X0.1S0.5\n ), you can get raster engraving files to under 10 characters per pixel. But why stick to that if we can get it down to one character per pixel ...

Professional machines can engrave at several meters per second, with high resolutions, and we have the hardware to do it to, what's missing is a way to transfer the data efficiently, and the software to send it, currently.

On Tue, Dec 20, 2016 at 6:29 PM, Sonny Jeon notifications@github.com wrote:

Serial @ 115200 baud can send over 1,100 lines with 10 characters each. Or 1,100 pixels/second. That's pretty good for a "slow" serial interface. Suppose we were able to cut this in half by creating a g-code to place a laser power vector with a raster pass, this would double potential performance to 2,200 pixels/second. A modest improvement.

Let's look at a native USB interface at a 12Mbit/s or 480 Mbit/s, 10 characters per line would theoretically give 11,000+ pixel/second and 468000 pixels/second, respectively. At least an order of magnitude better. So, bandwidth for streaming a g-code program is not a problem. It's a problem with the streaming interface or controller. Improvements must be made there.

@arthurwolf https://github.com/arthurwolf : I agree that the ability to dynamically adjust speed and power (overrides) is not as important during a raster job, but it's still important for everything else. I would not exclude a particular feature from this conversation, because there may be very interesting use-cases in the future that may prove very groundbreaking.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268304718, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGpFWBbK34yM9TkoGXIq7GYBKepWD3kks5rKBBfgaJpZM4LR0oN .

-- Courage et bonne humeur.

[chamnit]

@arthurwolf : It's both. The first post shows two examples of how to more efficiently command a raster pass. This is related to improving g-code. The other part of the conversation is whether or not the problem is the command and if it's the streaming interface and the controller.

IMHO, g-code is a great separator between CAM and the controller. The LW folks are doing an amazing job coming up with new ways to improve raster processes by eliminating excessive motions. If you upload a bitmap to the controller, this places the CAM responsibility solely on the controller. It makes the process a lot more opaque and inflexible. I think it'd be a good idea to keep this clean separation of responsibilities amongst our community.

FWIW, I'm pretty sure most professional machines still run on g-code. The fab shop at my lab has several production-level machines that support high-speed machining tool paths, include a 5-axis mill and CNC lathe. These all still run on straight g-code and often have jobs in the 100's MBs.

[arthurwolf]

Well if we are just going to be using Gcode ( as in : not change what we are doing now ), there's not so much need for discution I think : current Gcode does allow for a pretty compact format : G1 is modal so you don't need it on every line. And X can be relative, so you can just do "X0.1". That just leaves the S parameter, which is something like "S0.1" or "S127" depending on the controller. We could improve the speeds by removing the "X0.1" ( making it modal somehow ) I guess but that's not a huge improvement.

Even though we loose some of the improvements laserweb CAM offers, I still think lots of users would want the benefits of having an optimally compressed transfer.

On Tue, Dec 20, 2016 at 6:59 PM, Sonny Jeon notifications@github.com wrote:

@arthurwolf https://github.com/arthurwolf : It's both. The first post shows two examples of how to more efficiently command a raster pass. This is related to improving g-code. The other part of the conversation is whether or not the problem is the command and if it's the streaming interface and the controller.

IMHO, g-code is a great separator between CAM and the controller. The LW folks are doing an amazing job coming up with new ways to improve raster processes by eliminating excessive motions. If you upload a bitmap to the controller, this places the CAM responsibility solely on the controller. It makes the process a lot more opaque and inflexible. I think it'd be a good idea to keep this clean separation of responsibilities amongst our community.

FWIW, I'm pretty sure most professional machines still run on g-code. The fab shop at my lab has several production-level machines that support high-speed machining tool paths, include a 5-axis mill and CNC lathe. These all still run on straight g-code and often have jobs in the 100's MBs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268312349, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGpFRmQ3PBaZYv42iqvoDJR5pkzZHEIks5rKBeMgaJpZM4LR0oN .

-- Courage et bonne humeur.

[wolfmanjm]

I am not in favor of bastardizing GCode. I was think along the lines of a $ command followed by at least 64 characters of base64. For smoothie it is vital that a write be made of at least the size of the USB buffer that CDC/ACM sends, otherwise it is terribly inefficient. The buffer size depends on what type of USB it is. Smoothie uses 64 bytes USB buffers. If you write less than that then the USB buffer sent is not filled up and the USB protocol handles that, but inefficiently. If a speed adjustment is potentially needed for every pixel (as currently required) then I think a simple run length encoding is enough, the Format I was thinking is trivial so that Smoothie needs to simply unpack the data and convert it back to G codes. Smoothie cannot accept binary data over the serial lines so it must be encoded in someway, base64 seems the most efficient and easiest. How the line is encoded I have not thought about in detail yet. But a simple binary encoding for G0/1/2/3, the position is binary (X/Y), and the speed/feedrate is binary. Only the coordinates that have changed are sent (so Y is only sent once per scan line for example), speed is not sent unless it changes etc. keep it really simple. I think doing this would keep the planner queue quite full even for raster.

[tbfleming]

I'm ok with $ commands. It may not technically be gcode, but I still call them gcode since they may appear in gcode files.

[wolfmanjm]

FWIW as arthur says you can use normal gcode and with Smoothie you would get much better performance by using minimal gcode relying on modal behavior and sending multiple gcodes per line and writing then as a single buffer in the writ() command on the host. As Smoothie is usb/cdc/acm serial it requires that the data be sent in a buffer, if you write 4 bytes it will pack 4 bytes into a 64 byte buffer and send that and do checksums and resend on the 64 byte buffer that only has 4 bytes in it. It also depends on how the host driver is written, some may delay sending in case it can pack more data into the 64byte buffer, this would really slow things down. (This is the difference between a /dev/ttyACMx and /dev/ttyUSBx device driver).

[tbfleming]

@cprezzi See @wolfmanjm 's comment above.

[aldenhart]

I agree with Sonny. I think we should have adequate bandwidth with USB and other protocols to handle this with regular old Gcode. We are running host to board UART serial at rates well above 115K, so serial can do it, too. Let's test this in real-world situations before we feel the need for any new protocol.

If we end up needing a more compact representation for higher performance I'd prefer to wrap it in JSON, as that way you can carry as much descriptive or "hinting" metadata as the controller needs, in addition to the image object itself. I'd also like to find a very simple run length encoded protocol to encode the image, as this will parse better than uncompressed bitmaps, or highly compressed (2d compressed) formats like JPG.

Also, I support Sonny's desire to keep Gcode as standard as possible: we have been working problem of "non standard" Gcode, and have some things written up here explaining the why and how. https://github.com/synthetos/g2/wiki/g2dialect

See especially the section on Consensus Gcode, which is what we extracted as generally accepted practice form the NIST GCode, Fanuc, Haas, LinuxCNC, Tormach and other Gcode specs. We have started to use this as a baseline for anything new, and are really trying to hold the line on adding anything new to Gcode itself: https://github.com/synthetos/g2/wiki/g2dialect-Consensus-Gcode

[chamnit]

After thinking about it a little more, a raster pass is a lot of like a canned cycle. A specialized set of repetitive motions. If we treat a raster command like one, it wouldn't be a gross bastardization of the g-code standard. Canned cycles have some weird error rules, but I don't think they are standardized in anyway. We would just have to follow the overall rules set for canned cycles, like how they are modal and can be cleared with a G80 or another motion mode.

There isn't much to work with in terms of available parameter words, but it might be enough to significantly cut down on the number of lines sent. There are: P, L, R, Q, and S can still be used. If those 5 are just used, that cuts down the number of lines by 80%. Technically, IJK words could also be used, since they are only used for arcs.

The sequence can be invoked by a X0.1 to indicate incremental 0.1mm motions in X. (not sure how it would operate in absolute mode). A sequence of parameter words (S, L, P, R, Q) tells the cycle how many increments to move with a laser power level. If the sequence is out of order, it's an error. If the sequence is short, just move those increments passed.

I'd prefer this to a '$' command, because there are some existing rules to account for modal states with this approach.

[arthurwolf]

If we are going to stick with uncompressed gcode, I completely agree with the idea that this is like canned cycles. It really is very very similar. And even if we are going to compress things, but can't do binary, a compressed ( $ + base64 ) version of canned cycles gcode is pretty much the best thing we can do.

On Tue, Dec 20, 2016 at 8:28 PM, Sonny Jeon notifications@github.com wrote:

After thinking about it a little more, a raster pass is a lot of like a canned cycle. A specialized set of repetitive motions. If we treat a raster command like one, it wouldn't be a gross bastardization of the g-code standard. Canned cycles have some weird error rules, but I don't think they are standardized in anyway. We would just have to follow the overall rules set for canned cycles, like how they are modal and can be cleared with a G80 or another motion mode.

There isn't much to work with in terms of available parameter words, but it might be enough to significantly cut down on the number of lines sent. There are: P, L, R, Q, and S can still be used. If those 5 are just used, that cuts down the number of lines by 80%. Technically, IJK words could also be used, since they are only used for arcs.

The sequence can be invoked by a X0.1 to indicate incremental 0.1mm motions in X. (not sure how it would operate in absolute mode). A sequence of parameter words (S, L, P, R, Q) tells the cycle how many increments to move with a laser power level. If the sequence is out of order, it's an error. If the sequence is short, just move those increments passed.

I'd prefer this to a '$' command, because there are some existing rules to account for modal states with this approach.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268334676, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGpFYa5B5LmY1HeNGqbQfP8vbgvK2hRks5rKCxJgaJpZM4LR0oN .

-- Courage et bonne humeur.

[chamnit]

It would also be possible to create a set of canned cycles that modulate laser power based on mathematical curves.

For example, the R,P,Q and/or I,J,K words could be coefficients of a polynomial, while L is how many increments along this polynomial you want to burn. You could do some interesting things this way and still allow for some hand coding by users.

It would be relatively difficult for CAM programs to generate g-code with these, but maybe not. But it would help relieve some of the data bandwidth requirements of specifying a unique laser power per pixel. Instead, specify the function over multiple pixels.

[tbfleming]

@chamnit that's similar to lossy compression approaches. Polynomials poorly fit a lot of functions, so (D)FFT is more common.

[chamnit]

@tbfleming : You're completely right about poor poly-fits. I'm not familiar with what the D in front of FFT means. Derivative? Either way, it's an idea that can be extended to use any parametrized math function to control laser power over a motion. The main limitation would be to keep the processor load on the controller low.

I've always used existing FFT libraries when I needed them. So, I don't know their computational cost. Polynomials are fairly low-overhead in comparison. Just a lot of floating point multiplications.

EDIT: Looks like you mean discrete FFTs.

[tbfleming]

The D (Discrete) was a brain fart; FFT is always discrete. It's also expensive if you're not using a processor with single-cycle multiply-and-add with a special non-linear indirect-addressing mode (i.e. DSPs).

[wolfmanjm]

@tbfleming FYI Sxxx on a line by itself is not supported by smoothie. it must be on a G0/1/2/3 line

G1 X0.15 Y0.15 S0.0000
S0.5529
X0.30

[wolfmanjm]

I did some timing over USB serial on a 100MHz smoothie. The smoothie just read incoming data and kept a time of how long it took to read all the characters, the sender was a slightly modified version of grbls simple-stream.py. the results of just sending a sample file of gcode lines as per @tbfleming example was...

received 580671 bytes, 118125.171875 bytes/sec // plain one gcode per line
received 547092 bytes, 160890.203125 bytes/sec // combine lines to send 64 bytes minimum
received 542997 bytes, 171098.609375 bytes/sec // combine lines to send 128 bytes minimum

So sending the short lines to smoothie is slower than sending at least 64byte buffers, and sending 128 is even better. However you can't simply combine the supplied example as in order to send multiple gcodes in one line you need to have an explicit G0 or G1 so..

G1 X0.15 Y0.15 S0.0000
X0.30
X0.60 S0.5673
X0.90 S0.5098
X1.20 S0.4654
X1.50 S0.3804

would need to become...

G1X0.15Y0.15S0.0000G1X0.30G1X0.60 S0.5673G1X0.90 S0.5098G1X1.20 S0.4654G1X1.50 S0.3804

Thus increasing the number of characters per gcode line which is counter productive.

So I still think we can send more gcodes per line with a simple compression scheme, and reducing the bytes even further by using binary instead of ascii numbers, then using base64 to keep it as ascii.

I would totally oppose using regular gcodes to do this or creating more bastardized gcodes.

IMO this needs to be kept simple and make the processing of the line in the firmware as simple as possible as well. A simple method to me would be just taking generated gcode, post processing it to strip out unnecessary codes, assigning a binary stream to the gcode and sending it. on the Firmware side it would be as simple as expanding the line back to gcode and processing as normal.

[tbfleming]

@lautr3k :

@tbfleming FYI Sxxx on a line by itself is not supported by smoothie. it must be on a G0/1/2/3 line

G1 X0.15 Y0.15 S0.0000
S0.5529
X0.30

[cprezzi]

@wolfmanjm That sounds very promissing. I could concatenate gcode lines until I get up to 128 bytes (or a timeout) and just strype out all \n, then write to the port. I will try that (with none optimized gcode).

[cprezzi]

I would realy like to see a specific Gcode for raster data. Something in a form: G1X100 (X0.1Bbinary data of a line) whith the second x value for pixel with. This could also work for diagonal lines like G1X100Y100 (X0.1Y0.1Bbinary_data) or probably curves G3....

[aldenhart]

g2core (tinyg) addresses this problem with an extensible JSON structure using the Gcode active comment capability. Active comments are nothing new - (MSG ....) is one example, and LinuxCNC defines others. https://github.com/synthetos/g2/wiki/JSON-Active-Comments

g2core adds a JSON active comment ({...}). This is to keep the Gcode as standard as possible while providing extensibility. So the resulting Gcode line could look something like this:

G1 F1000 S1000 ({"img":"encoded_image", "enc":"image encoding used", "hints":"any polynomials controlling power modulation, image processing hinting, etc"})

Also, If you dig back most of these problems about image representation and processing have been solved in the PostScript/PDF era, and are just waiting to be ported into this space. PDF added hinting so that the image processing could be tuned to a specific printer's capabilities. That way they achieved "one file, prints anywhere" capability. We should try to do the same thing with Gcode. Using the JSON structure supports overloading - beyond whatever baseline usage conventions are arrived at for this or some other function.

[ghost]

Just tagging in @stefanix too - as I said, best if we all work the problem together

On Dec 21, 2016 2:30 PM, "Alden Hart" notifications@github.com wrote:

g2core (tinyg) addresses this problem with an extensible JSON structure using the Gcode active comment capability. Active comments are nothing new

• (MSG ....) is one example, and LinuxCNC defines others. https://github.com/synthetos/g2/wiki/JSON-Active-Comments

g2core adds a JSON active comment ({...}). This is to keep the Gcode as standard as possible while providing extensibility. So the resulting Gcode line would look something like this:

G1 F1000 S1000 ({...whatever JSON objects are needed to describe the raster, any polynomials controlling power modulation, image processing hinting, etc....})

Also, If you dig back most of these problems about image representation and processing have been solved in the PostScript/PDF era, and are just waiting to be ported into this space. PDF added hinting so that the image processing could be tuned to a specific printer's capabilities. That way they achieved "one file, prints anywhere" capability. We should try to do the same thing with Gcode. Using the JSON structure supports overloading - beyond whatever baseline usage conventions are arrived at for this or some other function.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268511595, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr25y47i7eDYWw3nnZs8SgI1sZ3CEPks5rKRvlgaJpZM4LR0oN .

[cprezzi]

Also taggin in @martinxyz as he wrote the pulseraster branch of the lasersaur in our fablab.

[ghost]

Ahh didn't know you two knew each other! Small world!

[cprezzi]

Yes, small world. Very small for people programming laser stuff ;)

[cprezzi]

@wolfmanjm : If I combine multiple gcodes in one line, do I get only one ok for the whole line or one ok per Gx? Or something else?

[chamnit]

My preference is not over-complicate things. In my view, adding a canned cycle is technically not bastardizing the g-code standard. They are designed to address problems like this. So, a laser raster canned cycle command can be designed to simply setup the raster line and its increments.

Earlier we talked about using a '$' line to pass raster data. While in the canned cycle modal state, each '$' data line would be treated as an input for the canned cycle to execute. It would be either HEX or Base64. I don't think binary is a good idea, because it's generally more difficult to deal with. You could support different data types and sizes by adding a code character after the '$'. For example, 8-bit values in HEX could be prefixed by $*, while 8-bit Base^$ is $&. Or whatever looks nice.

If a '$' binary line is sent while not in a laser raster modal state, it'll return an error.

[wolfmanjm]

@cprezzi By default you get one ok per line sent, so if you had 10 G codes on one line you would get one ok. However this is configurable, but you will get better performance if you only get one ok per line sent.

[cprezzi]

Thanks @wolfmanjm.

About raster data: Don't forget that a raster line could be much longer than the input buffer of the firmware, so we need to be able to split lines in smaller parts.

[wolfmanjm]

@cprezzi first this is NOT a firmware buffer, it is the size of the USB protocol buffer in the host and transmitted over the USB cable. if you were to write 1 character it would still send 64 bytes. When combining multiple lines they need to be whole gcode command lines, you cannot break a gcode into two lines. This is the size you need to write(...) from the host, multiple writes will only send smaller blocks. The size of a line must not exceed 64 bytes, so there must be a \n at least every 64 bytes, however you can send multiple lines concatenated (with the \n) in one buffer of 128 bytes or more if you want to make sure the USB buffers are always full. NOte that if a line does exceed 64 bytes (ie no \n in the first 64 bytes) everything upto the next \n will be discarded. (this is why we can send long lines that are comments as discarding a comment has no effect). It is also why I dislike the proposal to use "active comments" I think this is a bad idea. Smoothie discards comments very early on in the process.

[chamnit]

Are we talking about how to improve Smoothie raster performance or trying to come up with a universal way to send raster data across all platforms? It seems like this USB problem is fairly specific to Smoothie.

For raster data, the canned cycle proposal would accept binary data in multiple lines. Size wouldn't matter. It would simply execute as many valid pixels sent. If another is sent afterwards, then it execute those. Canned cycles are modal, so you only need to setup the motion once. The binary data after can be sent as many times as needed, in whatever packet size.

If a '$' prefix or wrapping in a '()' comment doesn't work, thats fine. The issue remains that there is no way in g-code to send a vector of data. And, personally, I don't think it'll improve performance that significantly by implementing a special g-code, because I still think it's a controller issue, rather than g-code. I'm just proposing something that is feasible based on what people seem to want.

[cprezzi]

@wolfmanjm I'm allready experimenting with a special implementation for smoothie, where I group consecutive gcodes togheter in one line of up to 128 bytes before sending them. Additionally I delete repeating g-modes. It showed that this way I can double the speed without stutter. More testing will follow.

Thanks @chamnit for the explanation. Does this mean each raster byte sent would initiate a move for one pixel, as long as the canned cycle is not finished? This sounds very convenient.

[cprezzi]

@openhardwarecoza If you like to play with my implementation, you can checkout "smoothie_buffer" in LW3. The only changes are in server.js.

[ghost]

No rush on my side (the big move is in progress, PC is packed already and i will probably only be there and unpacking from Monday) - to me its more important that we get a proven, scientific, well discussed, and FINAL way forward (; I was getting a little frustrated that we keep coming back round to the same problem and as we see it seems to ne heavily biased as a smoothie only problem. Thats why i stopped some other dev plans to make smoothie process bitmaps from sd (which doesnt really solve any problem since we still cant upload to sd over usb while running a job, neither via uart (and thus not via esp8266) - so for me the big goal is still to just optimise usb/serial transfer. But i called stop until its properly, and publicly discussed. Thats the best way to do large fixes

On Dec 21, 2016 11:17 PM, "Claudio Prezzi" notifications@github.com wrote:

@openhardwarecoza https://github.com/openhardwarecoza If you like to play with my implementation, you chan checkout "smoothie_buffer" in LW3. The only changes are in server.js.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268642292, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr29HFJ9rsnnQ_9dNp8y8frxDsu4maks5rKZeFgaJpZM4LR0oN .

[chamnit]

@openhardwarecoza : I showed the theoretical bandwidths of different interface types above. At 115200 serial baud rate, you can send about 1100 pixels/sec at 10 characters per line. Grbl's protocol has been reported to do 700 pixel/sec with plain g-code. So there is a good saturation and bandwidth utilization.

Native USB has theoretically 10x to 400x more bandwidth and Smoothie has a processor at least 10x faster than Grbl's 328p. There shouldn't be a problem with using g-code. So, this is why I still think this is a protocol issue, rather than g-code problem.

Creating a new g-code is more of a patch than a permanent solution. At best, you could get maybe 3x data compression with a Base64 scheme (2x with hexadecimal), if you count extra setup commands and wrapping required. The effort would be better invested in increasing interface bandwidth utilization instead.

[wolfmanjm]

In addition to seeing a raw throughput of around 100,000 bytes/sec (when sending sub optimal half full USB buffers) which is about 6,000 pixels/sec. I measured that smoothie can process about 850-1500 gcodes/sec depending on how full the planner queue is. (the more entries in the planner queue the slower it is). That would be about on average 1000 pixels/sec if each gcode moved 1 pixel. If the pixel width is 0.1mm and the feedrate was about 100 mm/sec and acceleration was around 5000 (enough for a 0.1mm segment to reach full speed) then the queue would be consumed at a rate of one gcode per ms or 1000 pixels per second. So the problem is indeed not necessarily the communications, it is the combined processing of gcodes and other processing like issuing steps etc. If the queue is consumed faster than it can be filled it will keep on stalling. This measurement does not really explain why compressing the input stream by reducing the number of characters sent does seem to help a lot, or running from sdcard is faster, as the limitation is actually processing the gcodes not feeding them into the planner. FWIW I do not think I can optimize the planner any further than it is. We do a lot more processing than GRBL does so I am not surprised that grbl may be faster in this respect. Since we added the new motion control that does acceleration per tick the calcs are more intensive for every gcode received as we need to fully pre calculate the step timing data so we can get the 100,000 steps/sec max rate.

[ghost]

@wolfmanjm interesting stats! And helps confirm my call was correct two days ago! (Processing bitmap to gcode inside smoothie on top of this would have been even slower right?)

Would anothr laser specific build option with less calculations be an option?

On Thu, Dec 22, 2016 at 1:37 AM, Jim Morris notifications@github.com wrote:

In addition to seeing a raw throughput of around 100,000 bytes/sec (when sending sub optimal half full USB buffers) which is about 6,000 pixels/sec. I measured that smoothie can process about 850-1500 gcodes/sec depending on how full the planner queue is. (the more entries in the planner queue the slower it is). That would be about on average 1000 pixels/sec if each gcode moved 1 pixel. If the pixel width is 0.1mm and the feedrate was about 100 mm/sec and acceleration was around 5000 (enough for a 0.1mm segment to reach full speed) then the queue would be consumed at a rate of one gcode per ms or 1000 pixels per second. So the problem is indeed not necessarily the communications, it is the combined processing of gcodes and other processing like issuing steps etc. If the queue is consumed faster than it can be filled it will keep on stalling. This measurement does not really explain why compressing the input stream by reducing the number of characters sent does seem to help a lot, or running from sdcard is faster, as the limitation is actually processing the gcodes not feeding them into the planner. FWIW I do not think I can optimize the planner any further than it is. We do a lot more processing than GRBL does so I am not surprised that grbl may be faster in this respect. Since we added the new motion control that does acceleration per tick the calcs are more intensive for every gcode received as we need to fully pre calculate the step timing data so we can get the 100,000 steps/sec max rate.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/LaserWeb4/issues/99#issuecomment-268674327, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr2ym7_55xnvhc4JyxNncvRktVwb-sks5rKbg0gaJpZM4LR0oN .

[cprezzi]

@wolfmanjm The concatenation of multiple gcodes per line is not working correctly. Smoothie seems to execute only one S value per line (checked with oscilloscope), so this is not usable at all (even if it has speed up the max feed alot).