OpenMMC for Sayma_AMC no RTM power

[jbqubit]

From https://github.com/m-labs/sinara/issues/517. There's no power applied to RTM (all LEDs are dark). Power was working fine prior to updating MMC firmware. I built my own firmware from master.

[jbqubit]

Using lateset from @wizath on benchtop and in crate I see rolling reboot.

[wizath]

What FlashMagic settings are you using? This (again) looks like something happens after enabling RTM Hardware (it only initializes 3V3 MP nad LEDS). Can you try if AMC alone gets initialized?

Also, could you post MCH logs? I wonder if there was some kind of error related to power.

I'm using these

[jbqubit]

AMC+RTM installed in second AMC slot. Here's what I see from MCH console after cycling power on crate.

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.03.14 10:23:59 =~=~=~=~=~=~=~=~=~=~=~=
 CU1 FRU 40 added
LSHM(0): CU1 front properties min 0 max 15 norm 3
LSHM(0): CU1 front now operational at level 11 (73%)
LSHM(0): CU1 rear properties min 0 max 15 norm 3
LSHM(0): CU1 rear now operational at level 11 (73%)
ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=12 SET_FRU_LED_S
TATE_REQ
AMC2(6): WARNING: entering Communication lost state !
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
Activation: modules are ready
Activation: all modules ready, Allowance Period (20 sec) stopped - continue with
 module startup !
Activation: starting AMC2
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=8 GET_DE
VICE_ID_REQ
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
WARNING: usb: TX buffer(0) pending, dtdt=0x00008080, EPSR=0x000a0004, EPCR1=0x00
880000
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=15 GET_D
EVICE_ID_REQ
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=32 GET_D
EVICE_ID_REQ
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=43 GET_D
EVICE_ID_REQ
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=57 GET_D
EVICE_ID_REQ

[jbqubit]

@wizath... scratch that last MCH log... I was using older version of .hex from a week ago. Will repeat with your 3/13 .hex shortly.

[jbqubit]

Recently I've been using FlashMagic with Interface "None (ISP)" which seems to work.

Just now I've flashed openMMC_fm_working.hex using FlashMagic configuration as in your screenshot. RTM installed in second AMC slot. Here's what I see from MCH console after inserting AMC into crate. Rolling reboot again.

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.03.14 10:36:48 =~=~=~=~=~=~=~=~=~=~=~=
: REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=15 READ_FRU_DATA_REQ

IpmiFru_ReadRecords(6): ERR - failed to communicate with EMMC
AMC2(6): WARNING: entering Communication lost state !
WARNING: usb: TX buffer(0) pending, dtdt=0x00008080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2, result -11
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2, result -11
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2, result -11
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=15 GET_DEVICE_ID_REQ
AMC2(6): INFO - AMC removed in state M7
AMC2(6): stop management task 'AM74' for i2c 0x74
LSHM(0): CM sensor 254 LUN 2 <unknown> hotswap M0->M1
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=51 SET_FRU_LED_STATE_REQ
AMC2(6): WARNING: entering Communication lost state !
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=61 GET_DEVICE_ID_REQ
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=4 GET_DEVICE_ID_REQ
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=21 GET_DEVICE_ID_REQ
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=32 GET_DEVICE_ID_REQ
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,1)ipmiMsgSender(6): REQ(I2C=0x74) failed on bus 2 - no ACK
R(6,2,2)ipmi_SendFru(6): timeout - no response for REQ: 0x20->0x74, Seq=46 GET_DEVICE_ID_REQ

[wizath]

Could you check with AMC alone?

[jbqubit]

With AMC alone no rolling reboot of MCH. But AMC power LEDs still dark. As soon as I additionally plug in the RTM rolling reboot of MMC resumes.

I advocate that we stop debugging this delicate MCH-IPMI-MMC dance for the moment. Is it possible to generate MMC source where MCH is 100% ignored. Unconditional power supplies on, peripherals configured.

[wizath]

Ok, so i'll focus for a while to update EXAR configuration

[jbqubit]

Before shifting to work on EXAR please generate source that unconditionally powers on and configures everything for Sayma AMC+RTM. Even better if this is a single (few) source solution. OpenMMC may be a good long term solution but with struggles to configure HMC830 and simply power AMC+RTM in crate the OpenMMC is a distraction.

[gkasprow]

I will post you the AMC board that has the OpenMMC working fine with our crate. Just to make sure there is no HW issue on your board related with MCH-MMC communication.

[gkasprow]

@jbqubit I've just tested. It works well in the crate and on the bench with same firmware loaded. Tomorrow will ship it.

[jbqubit]

Great! Please send me tracking number by email.

[jbqubit]

(@gkasprow Moving parts of our email discussion related to MMC to github.)

I received replacement AMC from @gkasprow. Board powers on fine on bench top when connected to RTM. The Blue front panel LED is not blinking. @gkasprow says this is proper behavior of board in uTCA crate -- MMC firmware on this AMC is now more standards compliant.

@gkasprow @wizath For this AMC + RTM in uTCA rack. I see fast flashing of blue, red, green lights on front panel. No power LEDS are illuminated on RTM or AMC. ​

[jbqubit]

The no power in uTCA crate situation is discussed in #475. So no need to discuss that here.

@gkasprow Please post copy of MMC firmware .hex that was shipped on this replacement board and close issue.

[hartytp]

I gather this is fixed in the latest MMC firmware and has been tested by @gkasprow (but correct me if I'm wrong about that).

[jbqubit]

The MMC firmware that effects this is being tracked in https://github.com/sinara-hw/sinara/issues/475.