OpenMMC for Sayma_AMC no RTM power

[jbqubit]

From https://github.com/m-labs/sinara/issues/517. There's no power applied to RTM (all LEDs are dark). Power was working fine prior to updating MMC firmware. I built my own firmware from master.

[gkasprow]

@jbqubit Did you try it in the MTCA crate?

[jbqubit]

No, the PCBs are not in the crate. This PCB pair worked fine prior to updating the MMC firmware.

[wizath]

You have to add #define BENCH_TEST in order to run MMC without crate. It's not tested yet, but should work fine. You should see something like BENCH Test active on the UART log if the mode is activated.

[jbqubit]

@wizath Which file should I edit? Please work closely with @filipswit to carefully document these things.

[wizath]

You can edit main.c and add on top #define BENCH_TEST 1, this should work fine.

[henrique-silva]

@wizath Adding -DBENCH_TEST=true on the CMake call will have the same effect. The CMake call will print in purple "Bench test mode is active!", but I haven't added any info com the UART log yet. This flag also bypasses a "software lock" that we implemented here to follow the uTCA standard, which states that the firmware should only boot after the ENABLE pin is deasserted .

[jbqubit]

I'm put my recently updated AMC into a uTCA crate along with an RTM. No power is applied to RTM.

-DBENCH_TEST=true I will try this now.

[jbqubit]

-DBENCH_TEST=true

Does not work. Only P12V0 led illuminates on RTM.

[wizath]

I'll get sayma tommorow and check that

[jbqubit]

Where can I get an older .hex that work? Would like to continue debugging this afternoon...

[gkasprow]

Try this. I didn't test it. lpc1776_ethernet_I2C.zip

[jbqubit]

lpc1776_ethernet_I2C.zip

Better but 1V2 and 2V0 on RTM are still off.

[gkasprow]

Are you sure the transistor that turns them on is on its place ?

[jbqubit]

The transistor was in the process of falling off on this RTM. I've fully removed the transistors. Using the .hex in your .zip the RTM powers on while sitting on bench top. I can boot AMC + RTM using gateware with no SAWG.

When sitting for several minutes on bench top the current starts at 2.8 A, then steps to 3.5A and often ends at 3.9A.

When AMC+RTM is inserted into uTCA crate, RTM is not powered.

[wizath]

I've fixed the issue with RTM power, since at the initialization MMC tried to disable RTM power and in result blocked whole I2C interface. Everything should work fine now along with UART commands (#521).

[jbqubit]

https://github.com/m-labs/mmc-firmware/commit/c18a7ecbb1b175b476c143f041592fe7b4083e4c I've compiled and reflashed. Now MMC does rolling reboot. No power is applied to AMC or RTM when inserted into the uTCA crate. Works fine on benchtop.

[wizath]

Did you remove BENCH_TEST directive for uTCA crate? I'll test it in crate in a moment.

[jbqubit]

I set -DBENCH_TEST=false. See attached for the .hex I'm using. The build script I'm using is here.

mmchex.zip

[wizath]

Try to remove whole BENCH_TEST declaration. As I can see, there are #ifdef declarations, so it does not matter if it's true or false.

./modules/main.c:#ifdef BENCH_TEST ./modules/main.c: printf("BENCH Test active\n"); ./modules/sensors/hotswap.c:#ifdef BENCH_TEST ./modules/sensors/hotswap.c:#ifdef BENCH_TEST ./modules/rtm.c:#ifdef BENCH_TEST ./port/board/afc-timing/payload.c:#ifndef BENCH_TEST ./port/board/afc-bpm/v3_1/payload.c:#ifndef BENCH_TEST ./port/board/afc-bpm/v3_0/payload.c:#ifndef BENCH_TEST

[jbqubit]

No. That doesn't help.

Please test on your hardware using my build script. That way we are on the same page.

[wizath]

I've replicated issue - apparently MMC enables hardware, but is not getting PWR level change message from MCH. Working on it atm.

[jbqubit]

@wizath What did you find on this? Is there a hack you can implement that just makes cards work in rack?

[wizath]

Quick way - comment the #ifdefs in rtm.c

//#ifdef BENCH_TEST
rtm_power_level = 0x01;
rtm_pwr_lvl_change = 0x00;
//#endif

That way, the RTM will start without getting message from MCH. It works on Vadatech MCH (I don't have NAT at this moment)

[jbqubit]

In the crate the MMC applies power to neither AMC nor RTM. I don't see how commenting out BENCH_TEST will impact the AMC no-power problem.

[gkasprow]

According to the MTCA specification, the power is enabled after MMC gets confirmation from MCH that it is capable delivering it (it asks for certain power capability). So if you don't plug it to the crate, the MCH won't communicate and give permission to MMC to turn the power on. So to operate outside of the crate we can do two things

• flash the firmware that has negotiation disabled
• implement the mechanism that will enable payload power when MMC does not get ENABLE signal, after some timeout. I have foreseen that problem and there is a detector and the signal routed to the MMC called P12V0_OK that can be used together with ENABLE signal to detect off-crate operation.

[jbqubit]

Powering boards on benchtop works fine. Powering boards /inside the crate/ does not work.

[wizath]

This is strange, AMC is working fine for me on both MCH regardless of RTM status.

1. Is the RED led lit when you put sayma into crate?
2. Could you open serial monitor and toggle hotswap to see if there is any activity on console?

[jbqubit]

On front of AMC red, blue and green lights are flashing in phase at about 3 Hz. On the serial terminal I see boot chatter repeated again and again. I suspect AMC is doing rolling reboot. I see same behavior in two different crates.

[jbqubit]

Green indicator on MCH for corresponding AMC slot is flashing green.

[wizath]

Could you show me this boot chatter?

[jbqubit]

Here's an excerpt.

[wizath]

It looks like after enabling RTM system reboots. Can you try without RTM? I've configured the firmware to halt on any error to ease debugging so this bootloop seems strange to me.

[jbqubit]

With RTM disconnected I see the following behavior.

• rolling display of "AMC Hotswap handle pressed!"
• front panel: blue led solid, green led flashing, red led off

[jbqubit]

With RTM installed it looks like MMC reset happens in middle of printing "RTM Enable Hardwar"

[wizath]

Anything happens if you toggle hotswap handle? AFAIK this rolling AMC Hotswap handle pressed could be an issue with IPMI commands, since the state is changed only if event is successfully sent to MCH hotswap_send_event( hotswap_amc_sensor, new_state_amc ) == ipmb_error_success

[gkasprow]

Doesn't the MMC detect over-current event at that time?

[jbqubit]

Anything happens if you toggle hotswap handle?

With RTM removed... The rolling message changes between "AMC Hotswap handle pressed!" and "AMC Hotswap handle released!".

[wizath]

Can you change line 72 in ipmi.c (https://github.com/m-labs/mmc-firmware/blob/master/modules/ipmi.c) to #if 1 and see if there are any IPMI messages received from the MCH?

[jbqubit]

I've changed line 72 and reflashed. No IPMI messages. Rolling reboot of MMC observed.

[wizath]

mmc.tar.gz

Could you try these? Maybe there is something wrong with FlashMagic flashing, I'll try @ monday.

[jbqubit]

I've converted your .bin to .hex and flashed. I see same rolling reboot.

[gkasprow]

@wizath could you embed the build date to the message the MMC prints after the start? This would clearly indicate that the last version was loaded

[jbqubit]

What's the next step on this? One approach would be to produce a simplified .hex that unconditionally turns on everything on AMC and RTM boards.

[gkasprow]

I brought today to the lab the NAT MCH so finally @wizath can recreate the problem

[wizath]

I've checked programming with FlashMagic under Windows. openMMC_fm_working.zip

Please flash this image.

[jbqubit]

@wizath Is this intended to be used in crate or on bench top?

[wizath]

Crate

13.03.2018 6:55 PM "Joe Britton" notifications@github.com napisał(a):

@wizath https://github.com/wizath Is this intended to be used in crate or on bench top?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/m-labs/sinara/issues/520#issuecomment-372759517, or mute the thread https://github.com/notifications/unsubscribe-auth/ALD4xHqJSLnoXU2b4pjOOBi5VbKNtEazks5teAgVgaJpZM4SbaBm .

[gkasprow]

However I have an idea to make the MMC detect if it operates in the crate or on the bench. In the second case all the power supplies will be brutally on. This will simplify the things a lot.

[jbqubit]

@gkasprow Agreed. Please make that the default for out of crate operation.

[gkasprow]

@jbqubit did you manage to load the bit file @wizath posted above?