DCS Speed

[repetier]

Hi, I'm the author of Repetier-Server and I got the feature request to support connection of the server through DuetServerControler. Having a 8mbit SPI connection and the ability to print over it I assume the general speed is good. So added a connector that connect through socket as described here: https://github.com/Duet3D/DuetSoftwareFramework#api

So far it works since we can handle it as normal serial connection internally. All we need is sending gcode and fetching all firmware responses so we analyse them to fill in the gui. So it seems all we need to support is SimpleCode which is working so far ok, but the speed is completely unusable. Here form log the timings on a simple test:

Send: 9:34:25.376: N113 M114
Recv: 9:34:25.498: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:25.501: N114 M114
Recv: 9:34:25.618: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:25.620: N115 M114
Recv: 9:34:25.709: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:25.711: N116 M114
Recv: 9:34:25.826: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:25.828: N117 M114
Recv: 9:34:25.948: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:25.950: N118 M114
Recv: 9:34:26.053: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.057: N119 M114
Recv: 9:34:26.168: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.171: N120 M114
Recv: 9:34:26.256: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.260: N121 M114
Recv: 9:34:26.363: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.366: N122 M114
Recv: 9:34:26.423: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.427: N123 M114
Recv: 9:34:26.556: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.560: N124 M114
Recv: 9:34:26.647: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.650: N125 M114
Recv: 9:34:26.767: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.770: N126 M114
Recv: 9:34:26.889: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.892: N127 M114
Recv: 9:34:26.986: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:26.989: N128 M114
Recv: 9:34:27.106: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:27.108: N129 M114
Recv: 9:34:27.227: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:27.230: N130 M114
Recv: 9:34:27.349: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:27.351: N131 M114
Recv: 9:34:27.470: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:27.472: N132 M114
Recv: 9:34:27.558: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:27.561: N133 M114
Recv: 9:34:27.678: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:27.680: N134 M114
Recv: 9:34:27.799: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:27.802: N135 M114
Recv: 9:34:27.930: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:27.933: N136 M114
Recv: 9:34:28.051: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:28.054: N137 M114
Recv: 9:34:28.145: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:28.148: N138 M114
Recv: 9:34:28.269: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:28.271: N139 M114
Recv: 9:34:28.389: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:28.391: N140 M114
Recv: 9:34:28.479: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
Send: 9:34:28.482: N141 M114
Recv: 9:34:28.541: X:0.000 Y:0.000 Z:0.000 E:0.000 E0:0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000

As you can see we get around 10 commands per second this way, which surely will not result in good prints. So my questions is basically are I'm using the right command or is there a faster way to communicate. We do not require to wait for firmware response if that is why it is so slow. Getting responses with a delay later would also work. Or is there another way to send quickly lots of gcode quickly. Using serial connection instead is much faster. Unfortuantely I can not use your client api, so wrote my own tiny specialized version.

[chrishamm]

The reason why your connector is awfully slow is because SimpleCode waits for a response, which may take a moment. It would be better to send codes that do not require an immediate response with the Asynchronous flag. That way DCS can buffer multiple codes and send them all at once (as it does internally when printing a file). To keep track of replies from those asynchronous codes, I suggest you create an intercepting connection in Executed mode. There you can filter for codes with the Async flag set.

[repetier]

Thanks for the quick answer. Hope you can help a bit more since I have no gasp on how to do it completely. With socat I connected manually and send {"mode": "Intercept","version":10,"interceptionMode": "Executed"} to get in the data you mentiond. In final solution I do thsi with a second socket connection I guess since the othe rone is still needed. I get something like:

{
    "connection": {
        "id": 8,
        "apiVersion": 10,
        "pluginName": null,
        "permissions": ["commandExecution", "codeInterceptionRead", "codeInterceptionReadWrite", "managePlugins", "manageUserSessions", "objectModelRead", "objectModelReadWrite", "registerHttpEndpoints", "readFilaments", "writeFilaments", "readFirmware", "writeFirmware", "readGCodes", "writeGCodes", "readMacros", "writeMacros", "readMenu", "writeMenu", "readSystem", "writeSystem", "readWeb", "writeWeb", "fileSystemAccess", "launchProcesses", "networkAccess", "superUser"],
        "isConnected": true
    },
    "sourceConnection": 8,
    "result": [{
        "time": "2021-04-29T11:20:10.6967356+01:00",
        "type": 0,
        "content": ""
    }, {
        "time": "2021-04-29T11:20:10.6969782+01:00",
        "type": 0,
        "content": "\n"
    }],
    "type": "M",
    "channel": "SBC",
    "lineNumber": 673,
    "indent": 0,
    "keyword": 0,
    "keywordArgument": null,
    "majorNumber": 105,
    "minorNumber": null,
    "flags": 2054,
    "comment": null,
    "filePosition": null,
    "length": 10,
    "parameters": [],
    "command": "Code"
}

and need to answer {"command": "Ignore"} so connection does not block your communication. sourceConnection is the id from the other connection I need to compare so I know answer is froma send I have send previously.

Where I'm stuck now is how to send a gcode asynchronously. What do I need to send here? Still SimpleCode with extra field? Or is there a different command for async send gcode.

[chrishamm]

You'd have to send a fully-populated Code object with CodeFlags.Asynchronous from your comand connection but I figure you're not keen on parsing G/M/T-codes to JSON objects. Would it be easier if I add a new connection mode like CodeStream where you could send and receive codes+replies asynchronously and line-based without having to worry about JSON at all? I figure Octoprint could reuse the same new connection mode too.

[repetier]

Guess Octoprint has the same problem yes. What we just need is a connection where we send gcodes and sooner or later receive answers. But it would not help to just copy the results, which is nice here. We also rely on the "ok" or "start" message. start is not such a big issue as I can send that on connection start, but we count commands send vs. "ok" to throttle speed. After all if user hits pause in server he expects to pause soon so it does not help if I already have send the complete gcode, if you know what I mean.

Your proposal would at least reduce a lot of traffic as I see there is quite some overhead that we do not require and it makes mangling to required protocols easier.

Talking easy json is also no problem. So how about this special connection mode. We send gcode just like with SimpleCode but it gets send asynchronously. We also get results about send gcode and responses as they got send or received along with connection id. We extract it from there and have the benefit that we also see the off server gcode being send. This is quite important so our printer state is up to date. E.g. if the duet web controller user sends M220 S120 we would otherwise not know and still show 100%. But if we also see gcodes send off server and can distinguish it from our stuff by connection id we can decide to ignore or parse gcode/answers as well.

[chrishamm]

Okay fine, I'll squeeze this into the upcoming version 3.3-RC1. It's okay to stick with SimpleCode for M220 or other UI-related codes because you know they're in effect as soon as you get a result back.

The new code stream could report "ok" (or Marlin-like responses in general) for every completed code but you need to be careful with the selected input channel (another parameter to SimpleCode or to interceptors): I suggest you send all codes over the Telnet channel which outputs Marlin-like responses by default (hence the CodeConsole app uses it too). You could use the USB input channel as well but that isn't encouraged because it might block other requests from the native USB line. Other channels like SBC (as seen in your example above) do not use Marlin emulation by default unless you send M555 P2 first.

[repetier]

Not sure what you mean with telnet channel. CodeConsole uses string output = await connection.PerformSimpleCode(input); to send commands and uses the DuetAPIClient.

We already support telnet for RRF but I'm not aware that there is also a telnet port for the DuetControlServer. If that would have the same format that would be a good intermediate solution.

For the M220 example I'm not sure you understood what I meant. I would like to see in the new mode that such a command was send from a different connection just so I can assume it changed the state. But I would need to distinguish these from own send gcodes, hence response in json with connection id so I can see oh my id so I already know or different id, so I have to examine it in addition but not expect a "ok" for it. Actually I don't need "ok" in that case as I can add my "ok" when I see I send the gcode.

[chrishamm]

Please check the second parameter of connection.PerformSimpleCode; it lets you specify the internal code channel where the G/M/T-code is supposed to be executed. This particular parameter defaults to CodeChannel.SBC but it would be better to assign it to CodeChannel.Telnet to get Marlin emulation right away. RRF has AFAIR 10 different code inputs (channels) where codes may be executed.

My idea to use SimpleCode for the UI instead of the new code stream was just an idea to make development more convenient for you, because then you don't need to worry about flow control and/or buffered codes at all.

[repetier]

Ok now I understand nothing any more. I found a enum with 11 channels for different connections or file/print. I have no idea what they do. I thought the control server is connected via SPI with the board for fast communication. When I select USB channel does it then try to connect via usb instead or in addition and communicates that way? And all I want is to not have a real usb connection in addition to talk. I want the server to take my commands and send me the responses. And if possible also tell me what other commands got send to printer as well for better state analysis. And until now I thought every client gets a connection id and your server tells us which client did what by the connection id.

[chrishamm]

RRF is capable of processing multiple G/M/T-codes on multiple code inputs at the same time. That enum lets you choose where a code is supposed to be interpreted. Examples: Codes from the web interface go to CodeChannel.HTTP, codes from the USB line go to CodeChannel.USB, and codes from the Pi typically go to CodeChannel.SBC.

Every input channel has different initial settings. For example by default Marlin-formatted output is only available on the USB and Telnet channels, so to get the "ok" responses as you usually expect execute your codes on the Telnet channel like so:

string output = await connection.PerformSimpleCode(input, CodeChannel.Telnet);

instead of:

string output = await connection.PerformSimpleCode(input);

I hope that makes things a bit clearer. The new code stream interface is basically ready now, I just need to run a few more tests next.

[repetier]

Ok, so it is not meant to say how it is connected but in which format I get the answer. Makes sense if that changes over time like adding SBC where you do not need ok/resend/checksum compared to usb. SBC is ok. Once you have a new pi server I will test if I can handle it. If possible please tell me the required json requests. I have no idea how I can get them from the C# code. Was already happy to learn that in readme you omitted the "version" field when selecting the mode. But that was quite easy since I got a error message. Now I just copy the version for first answer hoping that will always work:-)

[chrishamm]

Version 3.3rc1 is out now which brings the new code stream mode. It is available from the unstable package feed over apt.

To enter this new mode you need to send

{"mode": "CodeStream","version":11,"channel": "Telnet"}

Then you can writeln/readln codes/responses as you wish. Just note that you might need to modify your flow control a bit so that it can send up to 32 codes before expecting an ok response. That should greatly improve the throughput.

[repetier]

Thanks, I have now tested it and it works fine without errors. The results on my first speed tests are quite interesting. In ping pong mode it is as slow as with SimpleCode, so the shortest roundtrip time for some reason seems to be around 100ms. This does not really feel correct for me without knowing how your system internally works. But do you maybe have a sleep in the response parser or something else that adds latency when it gets no response. Before I continue here some results with bigger send buffers so we send commands in parallel also we currently have no command count limit which I will add later. But in general bigger buffer means more parallel commands.

I have not listed ping pong because it times out due to low speed. You see the maximum commands per second is reached with buffer sizes of around 2000 byte. Then we can transfer between 800-900 lines per second which is fast enough, but also approximately 64 buffers in parallel, so on pause it takes a very long time which is not good.

As comparison here the timings using same setup but connected with usb instead over dcs:

As you see it even reaches speeds of over 1000 lines/s with less buffer. So the firmware is not really the bottleneck and I also think spi should not be the problem. Even 1500 lines/s is less than 60kb of data per second. If you want to play around with the server the beta with dcs integrated is available for armhf here: https://download1.repetier.com/files/server/debian-armhf/Repetier-Server-1.0.5-Linux.deb

Another side question regarding dcs connection - I omitted checksums as I read that should be done. But do I need to care about resends or will they not happen in telnet mode since dcs handles these and does not pass through resend messages? So far I had none but that could just be good luck.

[benecito]

Hi both of you. I am building chocolate 3D printers based on duet and Repetier server. In the beginning I used a duet 2 (wifi) and an usb connection, but this meant no (easy) access to the duet for updates or alike. So I switched to an Ethernet connection which made it possible to access DWC and the file system over the pi running Repetier Server. As the duet 3 is simply overkill for my printers I was quite happy with the idea of the 3 mini and got my first one last week with high hopes having both systems running on the same pi. I started to experiment and this is pretty much exactly what I was hoping for. I updated my prototype today and ran a test print. So far everything looks good, but like mentioned above, the "delay" if I for example pause the print is quite high. Can I expect something coming up to solve this issue?

[chrishamm]

I suppose a new M-code to cancel codes that are already in the buffer could be useful for this scenario, but I am afraid this change won't make it into v3.3. I'll discuss this with @dc42 and implement it as part of v3.4.

[dc42]

Better would be for Repetier server to hook into the pause/cancel mechanism built into DSF and RRF, that cancels most movement commands already in the movement queue and tells DSF which command it should restart at. But that would require larger changes to Repetier server.

[repetier]

Ok I made some progress on the issue. DCS is in deed throtteling communication. In it's config it has

"SpiPollDelay": 25,

when I reduce it to 1 I get these speeds for different buffers:

The first 2 lines with 1000 and 500 are with 25 spi delay. You see an immediate increase in throughput. Need to play and find our more about the other polling delays especially the socket delay if that can improve further.

I'm also improving my handling in that I can limit buffered command in addition to buffer size so we get no big backlog on short commands. Should help reducing latency as well.

I also wanted to monitor other communication especially from duet webserver, so I have a second channel. On linux I can simulate the problem easily with socat:

socat - UNIX-CONNECT:/var/run/dsf/dcs.sock {"version":11,"id":12}{"mode": "Intercept","version":11,"interceptionMode": "Executed"}

Now it listens for all data and if telnet connections send data I receive them, send {"command": "Ignore"} also I'm not interested in the telnet and telnet has no connectionid set (is always 0). But as soon as I send e.g. "M114" in the web interface the connection closes instead of receiving the data.

Is this a bug or am I doing something wrong here? At the moment this problem disconnects the printer on every command so I need to disable it for now to have it working.

[repetier]

Ok, I have now the version with limiting commands ready and did some tests. Test setup:

• Only Duet Server and Repetier-Server running
• Pi 3
• Buffer is set high in Repetier-Server so always parallel commands are important
• Duet3 6H Mainboard

Test 1

SpiPollDelay = 1, vary max. parallel commands

• 8: 371 cmds/s
• 16: 620 cmds/s
• 24: 770 cmds/s
• 32: 840 cmds/s
• 48: 960 cmds/s

Test 2

16 parallel commands, vary SpiPollDelay

• 1ms: 620 cmds/s
• 5ms: 580 cmds/s
• 10ms: 470 cmds/s
• 25ms: 240 cmds/s
• 50ms: 126 cmds/s
• 100ms: 70 cmds/s

What is needed depends on the model, but I know some real serial connection were just as low as 330 lines/s and printed well. So anything above 500 should be no problem at all.

If you ask why SpiPollDelay I think it is quite simple. Board is fast and when a bunch of data is processed there is no response to read and you go into delay. Delay means no new communication even if new data is there, so it adds artificial delay form time to time even if execution directly would be possible. So here more frequent checking reduces round trip time massively. For my part I will stick to 1 as it gives best performance and allows reducing max. parallel commands to a nicer level. Maybe you think about reducing it for the default as well now you have seen what difference it makes.

Side note why I want the intercept:

• See other commands being send so I can detect state changes in our analyser
• Also see e.g. M118 responses - currently it is not possible to trigger our scripts when a build in code sends M118 since we do not receive it. Actually I do not even receive it when I send it in my own console. On the other side duet web server does see it, so it must be send some where.

Better would be for Repetier server to hook into the pause/cancel mechanism built into DSF and RRF, that cancels most movement commands already in the movement queue and tells DSF which command it should restart at. But that would require larger changes to Repetier server.

Not sure how it works, but in Repetier-Server a command that is send is more or less removed. We back it up for resend requests of course, but the lines from gcode are skipped. So any pause in server means just not sending more commands.

[dc42]

@chrishamm, can we make spiPollDelay adapt dynamically? i.e. use a smaller delay when there are commands waiting to be sent and RRF is able to receive them; but even so limit the rate at which DCS asks for object model updates.

[chrishamm]

@dc42 Yes, that's part of my planned performance improvements for v3.4.

[benecito]

All right, I ran a few prints during the weekend and they turned out good. When pausing it takes a while but there are worse things. But still looking forward to v3.4 which hopefully might solve this problem.

But a real problem right now is that the duet does not talk to the server (or the server does not listen) If I run out of chocolate / filament the duet can't tell the server to pause / cancel the print. Will there be a solution? The way I understand it, Repetier Server would like to listen to all commends but does not get any right now?

[repetier]

@chrishamm Did you miss this questions? I fear it is quite important for me and it seems to be bug, also maybe I just do something wrong.

I also wanted to monitor other communication especially from duet webserver, so I have a second channel. On linux I can simulate the problem easily with socat:

socat - UNIX-CONNECT:/var/run/dsf/dcs.sock {"version":11,"id":12}{"mode": "Intercept","version":11,"interceptionMode": "Executed"}

Now it listens for all data and if telnet connections send data I receive them, send {"command": "Ignore"} also I'm not interested in the telnet and telnet has no connection id set (is always 0). But as soon as I send e.g. "M114" in the duet web interface, the connection closes instead of receiving the data.

Is this a bug or am I doing something wrong here? At the moment this problem disconnects the printer on every command so I need to disable it for now to have it working.

[chrishamm]

General messages are reported over the object model so you'd need a subscribe connection to listen for messages as well. Every new message is reported as an item in the messages[] array.

If it helps, I can report general messages from the Duet over the CodeStream interface as well. Likewise I think a new code to clear cached codes would help to speed up pause requests but again that will have to wait until v3.4.

I do not see why you would need a separate interceptor if you already maintain a CodeStream connection.

Thanks for pointing out that the connection ID isn't set by the code stream, I'll fix this in the next build.

[repetier]

If it helps, I can report general messages from the Duet over the CodeStream interface as well. Likewise I think a new code to clear cached codes would help to speed up pause requests but again that will have to wait until v3.4.

Yes, I think that you really should do that. When I connect to real usb I get the M118 responses for example. CodeStream should at least behave the same since it is DWC solution to replace it for hosts without any big changes. And since I haven't found a description of the object model and want to avoid having a third socket that would be great. Reverse engeneering from C# library is not easy with all transformations etc. Actually I tried but have big problems understanding it and as C++ user I can not use the sources.

But the problem that any send command from web interface closes all interceptor socket connections is still unanswered:-) Even with the improved codestream fix I'd like to see the other commands being send so I can adjust eg. speed multiplier if someone changes it over duet interface.

[chrishamm]

I'll improve the docs shortly - sorry if some things are still unclear.

I cannot confirm your bug report though. To reproduce it, I started /opt/dsf/bin/CodeStream, entered ~360 G4 S1 calls, and sent M115 from DWC with /opt/dsf/bin/CodeLogger open. It does not disconnect, I can see intercepted codes, and I do get immediate responses over DWC. So perhaps there is an error in the way you're using the interceptor - yet I still don't understand why you need it in the first place. Perhaps open the log of DCS when it happens and check for potential error messages via journalctl -u duetcontrolserver -ef.

If you wish to open a second channel where codes are interpreted almost immediately, make a new CodeStream connection with the USB channel like

{"mode": "CodeStream","version":11,"channel": "USB"}

That will give you Marlin emulation again but I wouldn't send regular print instructions over it, because the same channel is used by the native USB port as well.

PS: If you increase the log level to trace in /opt/dsf/conf/config.json, you can actually see the JSON requests and responses in the DCS log.

[repetier]

Here I had you CodeLogger running. It worked fine as long a sonly server send M105 over stream (CodeStream) connection. At the time I entered M114 in your web interface at the top of the image the logger stopped. That is exactly what I also get with socat or in server when I listen to it.

As you see journal has no errors.

[chrishamm]

If you launch DCS somewhere else the log will remain clear - it doesn't look like that's a standard installation. It does work on my setup, even with M114:

[Pre] Telnet: G4 S1
[Post] Telnet: G4 S1
[Pre] HTTP: M114
[Post] HTTP: M114
[Executed] HTTP: M114 => X:0.000 Y:0.000 Z:0.000 E:0.000 E0:-0.0 Count 0 0 0 Machine 0.000 0.000 0.000 Bed comp 0.000
[Executed] Telnet: G4 S1 =>
ok
[Pre] Telnet: G4 S1
[Post] Telnet: G4 S1
[Executed] Telnet: G4 S1 =>
ok

There should be a note in the DCS log saying why the connection of the interceptor is terminated.

[repetier]

In syslog I get this error message:

May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://192.168.1.38:3000/machine/code text/plain;charset=UTF-8 4
May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Cors.Infrastructure.CorsService[5] CORS policy execution failed.
May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Cors.Infrastructure.CorsService[6] Request origin http://192.168.1.38:3000 does not have permission to access the resource.
May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'DuetWebServer.Controllers.MachineController.DoCode (DuetWebServer)'
May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3] Route matched with {action = "DoCode", controller = "Machine"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] DoCode() on controller DuetWebServer.Controllers.MachineController (DuetWebServer).
May 19 10:17:43 RepetierServer DuetWebServer[27534]: DuetWebServer.Controllers.MachineController[0] [DoCode] Executing code 'M114'
May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Mvc.Infrastructure.ContentResultExecutor[1] Executing ContentResult with HTTP Response ContentType of text/plain; charset=utf-8
May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2] Executed action DuetWebServer.Controllers.MachineController.DoCode (DuetWebServer) in 96.4813ms
May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'DuetWebServer.Controllers.MachineController.DoCode (DuetWebServer)'
May 19 10:17:43 RepetierServer DuetWebServer[27534]: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://192.168.1.38:3000/machine/code text/plain;charset=UTF-8 4 - 200 85 text/plain;+charset=utf-8 97.2959ms

And you are right. This is running on our image with your packages installed there (unstable of course). I have added for our next image to activate DCS/DWS in a simple config file so users can easily use the combined preconfigured set of both products.

[repetier]

Ok, have now tested your original sd card (Came with DCS 3.1). Switched to unstable, run sudo apt update sudo apt upgrade

Here, same thing happens. CodeLogger exists when I send a command in DWS. No other changes made to your sd card. So it does not seem to depend on having it installed my self.

If this does not crash on your pi - have you made any mods to the image since 3.1 that do not depend on the deb packages?

[chrishamm]

Sorry for the late response. We've just released 3.3-RC3 which addresses potential problems affecting code interceptors.

[repetier]

Ok have tested new RC3 a bit. Good news is that interceptor is now working also for me stable. I found one problem that can cause prints to stall though - interceptors need to respond every request which is ok in general. In my base I had a bug that did not send it every time. But also when network is stale or you did not yet detect it was closed you would wait for the answer. I understand that this is only happening on problems, but maybe there should be a timeout in case a plugin or whatever is also using it has a problem. Timeout should maybe 5s so also slow services would not create an unexpected close, which is also not good. I leave that up to you - we have fixed the problem on our side so for us it is working.

The other issue is M118 - I tried USB and Telnet for CodeStream mode and none contained the M118 response you wanted to add. Did you forget/not make it or is there some flag I have to set? I saw that also the interceptor responses do not contain the message so it is currently only the status interceptor and I would like to prevent the 3rd connection to reduce load on cpu. I guess when I subscribe

{
    "mode": "Subscribe",
    "subscriptionMode": "Patch"
}

I would get them somehow. Please let me know if I do something wrong, you plan to add it for RC4 or if I need to watch subscribe as well.

[repetier]

I have done some speed tests now with interceptor. The result is quite disappointing. Here my test:

Running on iMac with socket over SSH connected,

No interceptor, 5ms SPI poll: 8xx lines/s, DCS Cpu load: 100% Interceptor original,5ms SPI poll, 390 lines/s, DCS Cpu load: 210% Interceptor bypass handling,5ms SPI poll, 450 lines/s, DCS Cpu load: 220% Above with DuetWebServer stopped: 405-450 lines/s, DCS Cpu load: 250% DWS has no real impact on speed, guess that it is not using interceptors. No interceptor, but CodeLogger running as /opt/dsf/bin/CodeLogger > /dev/null 160 lines/s, DCS Load 160%

Having an interceptor in addition more than halfs the performance, also it is working as it should. I see good increase in cpu usage on server as well as DCS due to the extra round trips. There is a very simple solution to more or less gain full performance in this case. Many CPU time is used to construct and destruct json and serialization. If you could extend the register call:

{
    "mode": "Intercept",
    "interceptionMode": "Executed"
}

Into

{
    "mode": "Intercept",
    "interceptionMode": "Executed",
    "excludeSourceId": "23",
    "autoIgnore": true
}

then you could just skip for the commands/responses on that source - no network roundtrip and no json conversions. In my case it is the CodeStream source - I completely ignore the content if I see it since I already know what I have done my self. And if I'm the one printing I'm creating the important load and am just interested in messages from other sources.

The other trick would be a flag "autoIgnore" - you would still need to send the data package, but no need to wait for response. In fact you should not expect a response here. So you are just pushing the data and no wait for answer to continue. That way your cpu will rise but no unneccessary delays. Useful for all watchers that just watch like your CodeLogger example and our server.

I have updated the 1.0.5 armhf beta with interceptor switch for testing or choosing between detecting all and speed.

[benecito]

@chrishamm @dc42 can I expect some solution in the next days to send things from the duet to repetier server? the most important thing I‘d need is a way to pause\stop the print (on the servers side) if I for example run out of filament. So far I used messages for this, but as of now RS does still not receive any and thus the print does not stop if out of filament.

If there is nothing coming up soon I will have to ship my first printers with an additional USB connection, which I‘d like to avoid. I‘d like to ideally ship them with RRF 3.3 and RS 1.0.5 both in stable without the extra USB cable.

[chrishamm]

@benecito The code base for v3.3 is frozen except for bug fixes. We'll enhance the new streaming mode (and possibly interceptor connections) further in v3.4 but it may take a while before the next beta is released.

[benecito]

@benecito The code base for v3.3 is frozen except for bug fixes. We'll enhance the new streaming mode (and possibly interceptor connections) further in v3.4 but it may take a while before the next beta is released.

Is there any "workaround" I can use in the meantime?

[chrishamm]

Perhaps you could use DSF for printing/filament triggers and Repetier for controlling the print job via M32/M25/M24. Since I am not very familiar with Reptier I cannot say if this is possible though. You could symlink DSF's G-code directory to Repetier's G-code directory as well (remove and symlink /opt/dsf/sd/gcodes to whatever G-code directory is used by Repetier).

[repetier]

I have now optimized the Repetier-Server code to use faster json parsing library and many micro optimizations. I also watch the state changes which gratefully do not add speed penalty. So I can now see M118 responses. On a pi 3 I can send around 630 commands/s with 2ms poll time and 16 commands in parallel. With interception it now only drops to 430 lines/s which is still sufficient to publish it in next release with 3.3 final as dependency.

Having said that, I of course hope you come back to the optimizations I wrote above. Server + DCS take up to 250% cpu time due to all the unrequired overhead when we get close to that limit. And more interceptors form other sources will add extra penalty of course. With your protocol version I guess it is easy to see when you have added it from the number. So for version 11 I make the "slow" solution and on updated version 12 I might then switch/add extra parameter to reduce the load.

[chrishamm]

DSF v3.4-b2 and later are now much faster than previous versions and can achieve significantly higher throughput.

As part of v3.4-b3 I've added a new IPC command to clear already queued codes as well, that's something you may want to try out to speed up pauses from Repetier Server too (see InvalidateChannel). This new command completes when all the pending G/M/T-codes have been flushed so you can estimate the last valid file position from the code replies of your CodeStream connection rather than from the Executed interceptor.

[repetier]

What exactly does InvalidateChannel do and how can I activate it?I mean which json request tells it to add a channel to inactive list. Can I tell the interceptor with this to ignore command from my gcode send channel while parsing all other channels? Comment /// Invalidate all pending codes and files on a given channel (including buffered codes from DSF in RepRapFirmware)

is not saying anything. I would expect from this it deletes pending gcodes which does sound not like that we would want. What we need is that we do not get informed about additions in a channel so we can eliminate one round trip with heavy load in case server is running.

[chrishamm]

I am closing this issue. If you have any further questions, please get in touch with me directly.