sinara-hw / sinara

Sayma AMC/RTM issue tracker
Other
42 stars 7 forks source link

NAT-PM-AC600D current monitor #371

Closed jbqubit closed 6 years ago

jbqubit commented 6 years ago

I have a NAT-PM-AC600D power supply in the uTCA.4 rack. I can communicate with it via IPMI from MCH. AMC1 and AMC2 are populated with Sayma_AMC (not RTM). However, the supply reports 0 A for those slots. Based on the manual I expect there to be current info.

image

Here's what the MCH reports for the power supply sensors.

 Info  50 ********************
---------------------------------------
FRU Info for device 50:
---------------------------------------
Common Header    : 0x01 0x00 0x00 0x01 0x09 0x11 0x00 0xe4 
---------------------------------------
Internal Use Area : -
---------------------------------------
Chassis Info Area : -
---------------------------------------
Board Info Area          : at offs=8, len=64
Manufacturer(11)         : N.A.T. GmbH
Board Name(13)           : NAT-PM-AC600D
Serial Number(04)        : 0106
Part Number(04)          : 0c11
FRU file ID(01)          : 8
Customer Info-0 (14)     : FW V1.08/HV1.4
---------------------------------------
Product Info Area        : at offs=72, len=64
Manufacturer(11)         : N.A.T. GmbH
Product Name(13)         : NAT-PM-AC600D
Product Number(04)       : 0c11
Part Version(04)         : V1.3
Product Serial Number(04): 0106
Asset Tag(00)            :  -
FRU file ID(00)          :  -
Customer Info-0 (14)     : FW V1.08/HV1.4
---------------------------------------
Multi Record Area  : at offs=136

Record(0): Type ID=0xc0, PICMG Record ID=0x27, offset=0x000, len=12
Power Module Capability Record:
    Maximum Current Output: 50.0 A
---------------------------------------
Sensor Information for FRU 50 / PM1
==================================================================
  #   SDRType  Sensor Entity Inst  Value   State    Name
------------------------------------------------------------------
 30   MDevLoc          0x0a  0x61                    NAT-PM-AC600D
  1   Full     Temp    0x0a  0x61  27 C       ok     T_CPU
  2   Full     Temp    0x0a  0x61  46 C       ok     T_XFrm
  3   Full     Temp    0x0a  0x61  38 C       ok     T-PSB
  4   Full     Temp    0x0a  0x61  51 C       ok     T-PFC1
  5   Full     Temp    0x0a  0x61  46 C       ok     T-REC
  6   Full     Voltage 0x0a  0x61  120 V      ok     VINAC
  7   Full     Voltage 0x0a  0x61  382 V      ok     VINDC
  8   Full     Voltage 0x0a  0x61  12.3 V     ok     12V
  9   Full     Voltage 0x0a  0x61  3.4 V      ok     3.3V
 10   Full     Current 0x0a  0x61  1.00 A     ok     I_Sum
 11   Compact  Current 0x0a  0x61  0.65 A     ok     I_CH01
 12   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH02
 13   Compact  Current 0x0a  0x61  0.55 A     ok     I_CH03
 14   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH04
 15   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH05
 16   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH06
 17   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH07
 18   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH08
 19   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH09
 20   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH10
 21   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH11
ipmiMsgSender(5): REQ(I2C=0x72) failed on bus 1 - no ACK
R(5,1,1) 22   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH12
 23   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH13
 24   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH14
 25   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH15
 26   Compact  Current 0x0a  0x61  0.00 A     ok     I_CH16
 27   Compact  0xf2    0x0a  0x61  0x01              HotSwap
 28   Event       -    0x0a  0x00                    PwrChan
 29   Event       -    0x0a  0x00                    PM_Stat
 30   Compact  0xf0    0x1e  0x61  0x10              HS 050 PM1
------------------------------------------------------------------

@gkasprow The power supply should still report even if Sayma_AMC don't (yet) support IPMI, right?

vdirksen commented 6 years ago

Hi Joe,

if the Sayma_AMCs have no MMC code in the moment, you can enable power on your own responsibility with the following command assigning maximum 4 Amperes to AMC1 and AMC2:

amcpwr on = 5, 40, 0 amcpwr on = 6, 40, 0

Syntax: amc_pwr_on = fru_id, current_max, t_delay

Reference: NAT-MCH User manual chapter 10.4 Unmanaged AMC Module Configuration

The following two commands will give you information of any overcurrent, the assigned maximum power, and the current

show_pm show_sensorinfo 50

Let me know, if it was ok, that I respond via email or if I have to add my answer to github.

Have a great weekend.

Best regards Vollrath

jbqubit commented 6 years ago

Using the web interface at 192.168.1.80 I downloaded a copy of the Startup Configuration file from the MHC. I added the following two lines.

amc_pwr_on = 5, 40, 0
amc_pwr_on = 6, 40, 0

Uploaded the file back to the MHC and rebooted. Subsequently I see the following at the serial terminal.

show_pm image

show_sensorinfo 50

image

AMC1 and AMC2 are stuffed with Sayma_AMC. I can read the currents for these boards as sensors number 15 and 16. Good.

sbourdeauducq commented 6 years ago

Uploaded the file back to the MHC and rebooted. Subsequently I see the following at the serial terminal.

show_pm

@jbqubit That did not work. As you can see POn is not y for the AMC slots, and the FPGA is not powered (https://github.com/m-labs/sinara/issues/475).

jbqubit commented 6 years ago

I've put my annotated MCH configuration script on git here.

My Sayma_AMC and Sayma_RTM are in AMC slot 2. The green LED on power supply for AMC 2 is blinking green. Using NAT web interface I download 'System Information' text file. Here's what I see related to AMC power.

************* PM Information *************************
---------------------------------------------------------------
PM1: - unknown
PM2: - online, primary(fru 51)    : budget 80.0 A (alloc 20.2 A avail 59.8 A)
PM3: - unknown
PM4: - unknown
|-------------------------------------------------------------|
|chan  FRU FruId  primPM  secPM  PS1  POn  ENA   MP  PP  Amps |
|-------------------------------------------------------------|
  1   MCH1     3   2       -      y    y    y    y   y   1.0   
  2   MCH2     4   2       -      -    -    -    -   -         
  3    CU1    40   2       -      y    -    y    y   y   7.6   
  4    CU2    41   2       -      y    -    y    y   y   7.6   
  5   AMC1     5   2       -      -    -    -    -   -         
  6   AMC2     6   2       -      y    -    y    y   y   4.0   
  7   AMC3     7   2       -      -    -    -    -   -         
  8   AMC4     8   2       -      -    -    -    -   -         
  9   AMC5     9   2       -      -    -    -    -   -         
 10   AMC6    10   2       -      -    -    -    -   -         
 11   AMC7    11   2       -      -    -    -    -   -         
 12   AMC8    12   2       -      -    -    -    -   -         
 13   AMC9    13   2       -      -    -    -    -   -         
 14  AMC10    14   2       -      -    -    -    -   -         
 15  AMC11    15   2       -      -    -    -    -   -         
 16  AMC12    16   2       -      -    -    -    -   -         
|-------------------------------------------------------------|
Sensor Information for FRU 51 / PM2
==================================================================
  #   SDRType  Sensor Entity Inst  Value   State    Name
------------------------------------------------------------------
 31   MDevLoc          0x0a  0x62                    NAT-PM-AC1000
  1   Full     Temp    0x0a  0x62  27 C       ok     T_CPU
  2   Full     Temp    0x0a  0x62  42 C       ok     T_XFrm1
  3   Full     Temp    0x0a  0x62  45 C       ok     T_XFrm2
  4   Full     Temp    0x0a  0x62  41 C       ok     T-PSB
  5   Full     Temp    0x0a  0x62  38 C       ok     T-PFC1
  6   Full     Temp    0x0a  0x62  36 C       ok     T-REC
  7   Full     Voltage 0x0a  0x62  136 V      ok     VINAC
  8   Full     Voltage 0x0a  0x62  382 V      ok     VINDC
  9   Full     Voltage 0x0a  0x62  12.3 V     ok     12V
 10   Full     Voltage 0x0a  0x62  3.4 V      ok     3.3V
 11   Full     Current 0x0a  0x62  4.50 A     ok     I_Sum
 12   Compact  Current 0x0a  0x62  0.45 A     ok     I_CH01
 13   Compact  Current 0x0a  0x62  0.00 A     ok     I_CH02
 14   Compact  Current 0x0a  0x62  1.95 A     ok     I_CH03
 15   Compact  Current 0x0a  0x62  1.85 A     ok     I_CH04
 16   Compact  Current 0x0a  0x62  0.00 A     ok     I_CH05
 17   Compact  Current 0x0a  0x62  0.00 A     ok     I_CH06

The mapping between current sensor on the power supply (FRU 51) and AMC channel is in the NAT manual.

lines 15, 16, 17, ... are current for AMC1, AMC2, AMC3, ... line 14 is total current draw of all AMCs

@gkasprow When I run Sayma_AMC and Sayma_RTM on the desktop in my lab using stand-alone power supply I observe 3.3 A current. Above, the current is only 1.85 A suggesting the MMC hasn't configured the power supplies.

hartytp commented 6 years ago

When I run Sayma_AMC and Sayma_RTM on the desktop in my lab using stand-alone power supply I observe 3.3 A current

That seems like a lot to me. IIRC, it was only about 2A with the gateware version I'm using.

jbqubit commented 6 years ago

I also have 3 BaseMod installed on the RTM.

jbqubit commented 6 years ago

Closing since the question of how to use MCH to readout AMC current is resolved.