machinekit / machinekit-hal

Universal framework for machine control based on Hardware Abstraction Layer principle
https://www.machinekit.io
Other
109 stars 63 forks source link

Mesa 8i20 hm2 driver broken? #235

Open ShadeTechnik opened 5 years ago

ShadeTechnik commented 5 years ago

I've been working on a custom daughtercard for the DE10-Nano and mksocfpga. The one thing that has alluded me was Smart serial as I was testing it with an 8i20 that I had yanked from an existing machine running LinuxCNC. My config would discover the 8i20 and all of it's properties, then immediately throw communication faults and disconnect. So I was working with SS firmware which was probably unverified in mksocfpga, a SS hardware implementation that could not be verified due to the above and only an 8i20 that I know for a fact works fine.

I picked up a 7i66-8 today to mess with. I plugged it into my board and it discovers, can read and write to I/O and shows 0 SS fault-counts. So it works perfectly fine. I have to draw the conclusion that the 8i20 driver is broken in mk-hal. I took a quick peak and it appears to be rather old compared to LinuxCNC, but I'm no programmer so that's about the limit of what I can see. Possible to verify and correct 8i20 driver issue in MK-Hal?

the-snowwhite commented 5 years ago

This issue is related:

Can you kindly place a link to that 8i20 driver you looked at ?

ShadeTechnik commented 5 years ago

Calling it the 8i20 driver is probably a misnomer rather than saying it's the SS driver, and the way it reacts to the 8i20.

Believe you've seen this before, the SS driver in MK differs quite a bit from LCNC 2.8.

https://github.com/machinekit/machinekit/blob/6c1b9e39f37f70698133b647c791a92c923ab80d/src/hal/drivers/mesa-hostmot2/sserial.c#L498

https://github.com/LinuxCNC/linuxcnc/blob/4fdbed0965dd972456f1cdd12091aa5ff3dffbfc/src/hal/drivers/mesa-hostmot2/sserial.c#L350

my 8i20 sw revision is 139, AFAIK there is no user flashable firmware for a microcontroller card. As I mentioned on the group, the 8i20 and 7i64 do not autodiscover like other SS remote cards, which is probably the reason the SS driver references them specifically.

the-snowwhite commented 5 years ago

I did at some point in the other thread create a lcnc updated sserial.c deb package direct download link here, can you test this (again) and provide your test report here ?

ShadeTechnik commented 5 years ago

I tested that version again and got the same result. I tested the 7i66 with that mk-hal version installed immediately beforehand and the 7i66 did work, the 8i20 discovered and failed same as before.

the-snowwhite commented 5 years ago

Well that is as close as I can get you to running the actual current lcnc sserial.c driver on Mk. As it is possible the 8i20 got/is damaged is it possible for you to re-test it on the "old", or any different setup ?

ShadeTechnik commented 5 years ago

If the 8i20 didn’t work my milking machine wouldn’t have a spindle. I previously took it out of testing since dragging the DE10 setup down to the mill while trying to figure out what was going on with SS.

Now I have a 2nd nano and board setup so I just unplug the 7i76e from the 8i20, plug the nano board in and try it.

My 8i20 works fine, I just milled something with it in linuxcnc. The main difference here is how the 8i20 does smart serial compared to other SS boards.

I’ll see if I can get some info about sw version changes between hm2 and 8i20 software. Can anyone tell me what versions the current mksocfpga firmware and hm2 software were derived from?

Sent from my iPhone

On Sep 27, 2019, at 7:41 AM, Michael Brown notifications@github.com wrote:

Well that is as close I can get you to running the actual lcnc sserial.c driver on Mk. As it is possible the 8i20 got/is damaged is it possible for you to re-test it on the "old", or any different setup ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

ShadeTechnik commented 5 years ago

One question I guess worth asking is whether anyone knows a case where someone is using an 8i20 over SS in MK. It’s a rather popular drive with the LCNC croud but have yet to hear of one used with MK

Sent from my iPhone

On Sep 27, 2019, at 12:07 PM, Justin White jwhite@shadetechnik.com wrote:

If the 8i20 didn’t work my milking machine wouldn’t have a spindle. I previously took it out of testing since dragging the DE10 setup down to the mill while trying to figure out what was going on with SS.

Now I have a 2nd nano and board setup so I just unplug the 7i76e from the 8i20, plug the nano board in and try it.

My 8i20 works fine, I just milled something with it in linuxcnc. The main difference here is how the 8i20 does smart serial compared to other SS boards.

I’ll see if I can get some info about sw version changes between hm2 and 8i20 software. Can anyone tell me what versions the current mksocfpga firmware and hm2 software were derived from?

Sent from my iPhone

On Sep 27, 2019, at 7:41 AM, Michael Brown notifications@github.com wrote:

Well that is as close I can get you to running the actual lcnc sserial.c driver on Mk. As it is possible the 8i20 got/is damaged is it possible for you to re-test it on the "old", or any different setup ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

the-snowwhite commented 5 years ago

I't might be easier the other way around: To locate the exact mesa hostmot2 source files for the working Linuxcnc setup you have and then compare them with whats in the Mksocfpga repo.


The source files are online on the mesa product web pages. What you need to find out is the exact bitfile revision you are running on your Lcnc hardware setup (which mesa board is it ?), and then find a way to mach this revision number with the source files it is compiled from. I do not know how or by whom the Lcnc mesa bitfiles are created and distributed.


The mksocfpga was born with the freshest mesa hm2 source files I could identify at that time.... (the distribution model was(and probably still is) all files in 1 folder...) There was also looked into creating a versioned central repo for the mesa hostmot2 hdl sources, however this effort didn't come to fruit...

ShadeTechnik commented 5 years ago

I don't think it's mksocfpga, I think it is the SS driver in mk-hal. There's not really any reason to think the firmware built from mksocfpga is at fault considering an auto discovery card works. I can't be certain on that but knowing what I know now about the 8i20 requiring explicit driver instructions that is what I would guess.

I have 2 machines that run LinuxCNC 2.8pre1. One runs a Mesa 7i96 (host Ethernet card) and the other runs a 7i76e (host ethernet card) plus the 8i20 (SS remote servo drive). Neither of these machines use custom firmware of any sort, all cards were purchased directly from Mesa within the last 2 years. I bought the cards that did what I needed them to do stock so the Firmware is the stock Mesa firmware, no special requests, no custom flash.

The 7i76e+8i20 just works. As explained on GG, I also plugged the 8i20 into the machine running the 7i96, and it just worked there too. I don't know what the stock firmware is exactly but it's source is certainly contained here http://www.mesanet.com/software/. I suppose I could ask PCW which firmware ships with each card if necessary.

The best thing to do I think is to install MK on a clean hard disk and run it on one of my x86 LCNC machines and see if the 8i20 will work. In that case MKSOCFPGA is out of the equation and it comes down to mk-hal/mk-cnc. The only question is, is MK properly supporting Mesa eth cards? They are all newer to the scene and I never hear of anyone running MK with Mesa hardware really. The 7i96 is most likely not supported in MK because it required special hm2 drivers in LinuxCNC that at the time only existed in 2.8pre1. The 7i76e is probably the most popular card Mesa sells period so if that does not work in MK it probably should.

So if you can tell me whether the 7i76e host card is working in MK, and exactly what kind of MK install you would want to see on x86 (preempt-rt)I can test that. I'm not trying to get this working for sake of my specific setup, I'm just raising the issue that the 8i20 likely does not work properly with off the shelf Mesa hardware in MK.

On Fri, Sep 27, 2019 at 5:02 PM Michael Brown notifications@github.com wrote:

I't might be easier the other way around: To locate the exact mesa hostmot2 source files for the working Linuxcnc setup you have and then compare them with whats in the Mksocfpga repo. The source files are online on the mesa product web pages. What you need to find out is the exact bitfile revision you are running on your Lcnc hardware setup (which mesa board is it ?), and then find a way to mach this revision number with the source files it is compiled from. I do not know how or by whom the Lcnc mesa bitfiles are created and distributed.

The mksocfpga was born with the freshest mesa hm2 source files I could identify at that time.... (the distribution model was(and probably still is) all files in 1 folder...) There was also looked into creating a versioned central repo for the mesa hostmot2 hdl sources, however this effort didn't come to fruit...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/machinekit/machinekit-hal/issues/235?email_source=notifications&email_token=ANIOO6R3GZLORPZSA43V6N3QLZYF7A5CNFSM4I2SQJS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD72CWCA#issuecomment-536095496, or mute the thread https://github.com/notifications/unsubscribe-auth/ANIOO6UEPFHANBSVR3OUTOLQLZYF7ANCNFSM4I2SQJSQ .

-- Justin White Shade Technik

the-snowwhite commented 5 years ago

I just looked at the new mesa_2019 sources from [here: 7i96.zip] (http://www.mesanet.com/software/parallel/7i96.zip) There are a considerable amount of additions and disruptive register changes, up front I can say the lcnc hm2 and machinekit hm2 are highly incompatible atm.


The current Lcnc SSerial implementation iin the mesa HDL sources is currently more different than I expected no wonder just porting the lcnc sserial didn't work...... I'll expect this to reflect in the lcnc hostmot2.h file...also


I'll continue my analysis tomorrow and put up a new issue regarding my findings.

the-snowwhite commented 5 years ago

mesa_2019.tar.gz Here the new mesa sources sorted into a mksocfpga file structure, for easy diff/compare.

the-snowwhite commented 5 years ago

@ShadeTechnik I'ts not very straight forward to port between lcnc and machinekit as machinekit uses pins instead of parameters, but here is an updated deb that should have a more full lcnc mk sserial experience.. machinekit-hal-rt-preempt_0.2.1569672670.git58ad363bf-1~stretch_armhf.tar.gz for testing

ShadeTechnik commented 5 years ago

Gave it a try this morning with the new MK-hal deb and it's still a no-go for the 8i20. I had to remove the 8i20 from the machine I typically use it for for other reasons and being that I have different SS card on hand (7i66) it gives me a bit more opportunity to notice things.

The 8i20, I plug it into another LinuxCNC+7i96 project machine I have that has no special consideration for the 8i20, nothing in hal other than CONFIG="sserial_port_0=00xxxx", and the card will show a status heartbeat and sserial pins will show a fault count of 0. This is just plugging the card into the SS port with jumpers set to supply logic power from SS port, no HV power, no enable.

Plugging the same 8i20 into the Nano, same conditions. I get only a very brief Status light then a fault light, no heartbeat. SS pins as follows. hm2_5i25.0.sserial.port-0.port_state = 10 (3 on working SS setups) hm2_5i25.0.sserial.port-0.port_state2 = 0 hm2_5i25.0.sserial.port-0.port_state3 = 0 hm2_5i25.0.sserial.port-0.fault-count = 207 (should be 0, shuts down @ 200)

The nano with a 7i66 shows a port_state of 3 with a fault count of 0 and works perfectly fine. Same cable, same nano, same daughtercard. Michael, I have a suggestion I'll post on GG

the-snowwhite commented 5 years ago

Just a stray thought that popped up when I noticed that the 7i96 is connected to the PC via ethernet:

The ethernet provides galvanic isolation as far as the pc/controller goes, so could this be some kind of ground loop problem between in the power rails and the rs4xx interface ?

ShadeTechnik commented 5 years ago

So is the DE10 nano. I VNC into the nano over ethernet plugged into my network switch, I don't bother plugging a monitor into it usually.

the 7i76e and 7i96 communicate with their host PC over ethernet. The DE10 has been tested plugged into a monitor with no ethernet and not plugged into a monitor with ethernet attached to my switch, same results. RS422 is much more immune to this type of thing because of the differential signalling, otherwise it is functionally the same as rs232.

I don't have any sort of problem with the nano and a 7i66 and I've tried this in alot of different scenarios so I don't think that's the case.

the-snowwhite commented 5 years ago

From the thoughts perspective: In this setup the DE10 Nano is actually the pc so the ground loop would be between the DE10 Nano and the 8i20...

Another thing to explore would be to connect the 7I96 to the DE10 Nano instead of the PC... (does mk have a 7i96 driver, or is it easy to locate and port ?)


Sorry I can't do more from here but It's very hard for me to do remote debugging (debug without having the actual HW on hand). The expensive part in that is actually not the parts but the freight between you and me (in Denmark...) BTW vat is not an issue as I'm Danish vat registered. (as long as the sender remembers to place this number up front in my address, else it gets very expensive for me to receive packets from abroad)

ShadeTechnik commented 5 years ago

Getting the DE10 setup with the 7i96 would honestly be kind of a PIA, and I'm not sure what it would prove exactly.

What I could possibly do is get another board setup with MK and see if the 7i96 will work with it, then if it will communicate with an 8i20 over SS. That would narrow it down to MK-hal having a bad SS/8i20 driver. I recently built setup a small x86 PC to replace the PC that I originally had connected to the 7i96, I installed the current LinuxCNC master 2.7 something and even it did not work with the 7i96, I had to go back to 2.8pre1. So I doubt given the current Mesa support situation if the 7i96 will work at all

the-snowwhite commented 5 years ago

Yes currently I'm certain It won't work unless someone updates the hm2_eth.c file. Well I've flagged for some assistance, lets see who steps in.....with some hand work...


right now I need some closure on other stuff waiting for reactions....

ShadeTechnik commented 5 years ago

Try swapping out the sserial.h header file with the LCNC version? I see differences but they don't look like compatibility issues.

https://github.com/LinuxCNC/linuxcnc/blob/master/src/hal/drivers/mesa-hostmot2/sserial.h#L227

the-snowwhite commented 5 years ago

I have just pushed the branch with the lcnc sserial experiments.

Try swapping out the sserial.h header file with the LCNC version?

I think I did that

We even tried some more imports if you look at the latest commit you tested

RicoKirstein commented 4 years ago

@the-snowwhite Hi, did you get this fixed? The last two days we were experimenting with a DE0 nano Soc and some custom boards. Since we didn't have enough pins and some stmf1s around we were thinking about just doing the handwheel on smartserial. We did get it working exactly once. And then after changing some code never again in the newest machinekit version. After finding this thread we reverted our sd card back to an old version I think debian-9.2-console-armhf-2017-10-29, since I knew that Mesa smartSerial cards definitely worked on that version as I had used it before. And it worked instantly. The FPGA firmware is identical between both builds.

Cheers, Rico

ShadeTechnik commented 4 years ago

Smart serial on com with Mesa daughter boards that have automatic identification do work as of the last I tested. What did not work was specifically the 8i20 and another board (7i4x maybe?). These two boards. Were designed before smart serial could do auto id so there are explicit IDs in the drivers.

I dont know what work has been done in the past year, just pointing out that most smart serial cards did work in some capacity.

On Tue, Oct 20, 2020, 2:49 PM RicoKirstein notifications@github.com wrote:

@the-snowwhite https://github.com/the-snowwhite Hi, did you get this fixed? The last two days we were experimenting with a DE0 nano Soc and some custom boards. Since we didn't have enough pins and some stmf1s around we were thinking about just doing the handwheel on smartserial. We did get it working exactly once. And then after changing some code never again in the newest machinekit version. After finding this thread we reverted our sd card back to an old version I think debian-9.2-console-armhf-2017-10-29, since I knew that Mesa smartSerial cards definitely worked on that version as I had used it before. And it worked instantly. The FPGA firmware is identical between both builds.

Cheers, Rico

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/machinekit/machinekit-hal/issues/235#issuecomment-713067335, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANIOO6QGMMLVKVNATIXUMJDSLXLTPANCNFSM4I2SQJSQ .

the-snowwhite commented 4 years ago

@RicoKirstein The FPGA firmware is identical between both builds Yeah there hasn't been that much evolution in the mksocfpga mesa fpga stuff, AFAICR. On the other (soft) hand Machinekit is now riding on machinekit-hal and cnc part is (99%)mainstream linuxcnc (emcapplication). Do you remember which repo(s) you used for the newest machinekit version that you got to work ?

RicoKirstein commented 4 years ago

I believe the version that worked was preinstalled on the de0 sd card image in the home folder. But i will retest and check which version it was on Friday.

RicoKirstein commented 4 years ago

Sorry, it took me a while to validate it. The problem was indeed not with the newer version of Machinekit but a combination of a too long cable and somehow the device id changing from device.0.1 to device.0.0 in the newer version.

the-snowwhite commented 4 years ago

@RicoKirstein Great you got it to work However since this is an ongoing issue: It would be great if you could provide the detailed steps you went thru to get this to work. As a reference for others